Skip to content

[core] Read primary-key vector results by position#8576

Merged
JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/pk-vector-05-position-read
Jul 12, 2026
Merged

[core] Read primary-key vector results by position#8576
JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/pk-vector-05-position-read

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the physical-row materialization path for primary-key vector search results. Bucket-local ANN and exact hits can now reuse IndexedSplit for distributed transport and be read from their source data files while preserving physical positions and scores.

Changes

  • Reuse the existing IndexedSplit contract instead of introducing a vector-specific split: for primary-key raw reads, its rowRanges are physical positions in exactly one data file and its optional scores follow expanded range order.
  • Document that IndexedSplit ranges use the coordinate system of their table read path: stable row IDs for Data Evolution and single-file physical positions for primary-key raw reads.
  • Add a dedicated PrimaryKeyIndexedSplitRead and provider to validate the single-file contract, expand ranges into physical positions, align optional scores, and route indexed primary-key reads.
  • Add PrimaryKeyVectorPositionReader, which filters file batches by selected positions and exposes the physical position and score through ScoreRecordIterator.
  • Keep RawFileSplitRead independent of IndexedSplit: it only provides the lower-level single-file read for an already selected position bitmap, while the dedicated indexed reader owns indexed-split semantics.
  • Intersect indexed positions with file-index and deletion-vector filtering before reading, so only live matching rows are materialized.
  • Keep scalar predicate pushdown, but disable regular TopN and limit pushdown for indexed primary-key results because applying them before position filtering could incorrectly discard ANN hits.

Testing

  • Verify existing IndexedSplit serialization remains compatible.
  • Verify physical position and score propagation for both single-batch and multi-batch file readers.
  • Verify KeyValueTableRead routes IndexedSplit through the dedicated primary-key indexed reader rather than the generic raw provider.
  • Verify indexed ranges are converted to the selected-position bitmap passed to the raw file reader.
  • Verify non-contiguous physical positions preserve score alignment.
  • Verify a regular read limit does not discard ANN hits at higher physical file positions.
  • Run the focused Core suite: 9 tests passed.
  • Run Checkstyle, Spotless, Maven enforcers, and the focused tests on JDK 8.

Notes

  • Follows [core] Add primary-key vector bucket search #8569, which adds bucket-local primary-key vector search and returns data-file plus physical-position results.
  • Generating IndexedSplit instances from bucket search, cross-bucket Top-K, snapshot scan routing, and Flink source integration remain follow-up work.

@JingsongLi JingsongLi marked this pull request as ready for review July 12, 2026 12:13

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the latest commit (17ed7d8). The dedicated primary-key indexed read/provider cleanly isolates indexed reads from regular raw-read TopN/limit mapping state, while preserving predicate and deletion-vector filtering. I also verified physical-position/score propagation, multi-batch behavior, serialization compatibility, and the regular-read-then-indexed-read cache order. The focused 22-test suite and the additional cache-order regression passed on JDK 8. Looks good to me.

@JingsongLi JingsongLi merged commit d41ec73 into apache:master Jul 12, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants