Skip to content

Fix scalar-quantized CAGRA graph input encoding - #3

Draft
nvzm123 wants to merge 1 commit into
zackm_cuvslucene-139from
scalar-quantized-cagra-input-fix
Draft

nvzm123 wants to merge 1 commit into
zackm_cuvslucene-139from
scalar-quantized-cagra-input-fix

Conversation

@nvzm123

@nvzm123 nvzm123 commented Sep 22, 2026

Copy link
Copy Markdown
Owner

Summary

This does not add scalar quantization: the accelerated scalar codec, service registration, and benchmark selector already exist. It fixes the unsigned byte encoding supplied to the intermediate CAGRA graph build.

  • Initialize per-dimension maxima with negative infinity so all-negative dimensions are measured correctly.
  • Emit monotonic unsigned 7-bit values in [0, 127], matching the unsigned CuVSMatrix.DataType.BYTE contract.
  • Remove the ineffective signed-to-unsigned conversion and its redundant full byte-vector copy.
  • Add focused regressions for negative-only dimensions and monotonic values across zero.
  • Regenerate the affected Lucene API reference pages.

Why

The old helper emitted signed values in [-64, 63]. Its conversion used (byte) (value & 0xff), which is a bitwise no-op after casting the result back to a Java byte. Because the cuVS byte matrix is exposed as kDLUInt, negative codes were read as 192..255 while zero remained 0, introducing a discontinuity into CAGRA distances.

The direct unsigned formula is the intended centered code plus 64:

(round(127 * normalized) - 64) + 64 == round(127 * normalized)

For Euclidean distance, that constant shift preserves the intended pairwise distances while satisfying the unsigned matrix contract. Float.MIN_VALUE was also replaced because it is the smallest positive float, not a negative lower bound.

Usage

No benchmark-harness or YAML API change is required. The existing benchmark selector is:

"algoToRun": "CAGRA_HNSW_SCALAR"

This branch is based directly on the current NVIDIA/cuvs#2476 head, 8faae5092.

Compatibility and scope

  • Lucene's persisted/search scalar quantization and on-disk format are unchanged.
  • Existing indexes are unchanged; newly built scalar indexes may have different graph topology because the intermediate CAGRA input is corrected.
  • The public helper's byte output intentionally changes from centered signed codes to unsigned 7-bit codes.
  • This PR contains no post-ingest parallelism, graph-only persistence, Lucene 10.5 work, heuristic changes, or benchmark-harness changes.
  • Benchmark quality has been validated only for Euclidean distance. Adding 64 is not inner-product or cosine invariant, so this PR does not claim quality correctness for those metrics.

Validation

Exact branch head:

  • Focused scalar, quantized-format, and multi-layer round-trip tests: 111 tests, 0 failures, 0 errors, 14 intentional skips.
  • Full cuvs-lucene module: 346 tests across 33 suites, 0 failures, 0 errors, 30 skips.
  • ./build.sh java lucene --run-java-tests completed successfully, including Panama binding generation and local artifact installation.
  • mvn -q spotless:check and git diff --check passed.
  • API-reference generation was idempotent across consecutive runs.
  • fern/build_docs.sh check validated all 284 MDX files with 0 errors. Its two warnings were the existing unauthenticated redirect check and light-mode accent contrast warning.

Historical Euclidean single-run ablations of this same three-file logical patch on a later integration base showed Deep1B-1M recall at efSearch=1500 increase from 92.4417% to 94.9120%, while Jasper-1M moved from 97.9313% to 97.9920%. These are supporting observations, not exact-head performance claims or evidence that scalar quantization should become a default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant