Repository navigation
[spark] Support centroid-based sharding for vector global indexes - #11249
Draft
JingsongLi wants to merge 14 commits into
Draft
JingsongLi wants to merge 14 commits into
JingsongLi wants to merge 14 commits into
Conversation
JingsongLi
marked this pull request as draft
November 18, 2026 11:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add five Spark global index types for Data Evolution append tables. Use
ivf-flat-centroid,ivf-sq-centroid,ivf-pq-centroid,ivf-rq-centroidordiskann-centroidinsys.create_global_index. Options use the same prefix, for exampleivf-pq-centroid.nlist=64. Partitions can use both ordinary and centroid types for the same vector column. Readers search each selected range with its registered index type and merge the scores for the final top-K. Set one distance metric for the column across partitions. PIP-47Spark uses the same contiguous row-ID grouping as BTree. Each such range within a partition with any non-null vectors gets its own router and native shards. Spark pins a snapshot, counts vectors per split and plans a deterministic sample with the ordinary IVF-FLAT trainer. Each map task reads the selected rows from its own splits. For each range, one task trains a router on the sampled rows in reservoir order. Spark broadcasts the router, shuffles by centroid ID and builds a local IVF or DiskANN index in each populated shard. A single commit replaces the selected partitions' files of the requested type.
The router and every associated shard carry the same
GlobalIndexMeta.rowRange. Planning, visibility, raw fallback and conflict checks use the standard range. A gap can therefore be searched from source data when the query uses detail or chunk mode. The centroid writer stores each assigned row’s offset from the start of the group range. Both ordinary and centroid readers use the standard OffsetGlobalIndexReader to translate filter bitmaps and restore table row IDs for single and batch searches. Row-ID reassignment can reuse files when the whole group range moves by a constant offset. The plugin-ownedGlobalIndexMeta.indexMetaidentifies the router generation and shard count or the shard's centroid ID and underlying local writer metadata. Upgrading test deployments from either of this PR’s earlier layouts with absolute row IDs requires dropping those indexes with the previous build and rebuilding them.The native router and deterministic sample planner are available in Apache’s published
paimon-vector-index-java:0.7.0-SNAPSHOT. Validation used0.7.0-20261119.003537-3. apache/paimon-vector-index#131Tests
Four disjoint ranges on SIFT100k
Use the first 100,000 SIFT1M vectors (128D, L2). The target partition contains four runs of 25,000 rows separated by three rows allocated to another partition. The ordinary range builder produced 19 aligned shards at 6,251 rows per shard. The previous centroid layout at
643346bdd9has one router and 16 shards. The current layout has four routers and four shards per router. Every query visits all four intervals. The ordinary and four-router layouts were rebuilt and timed after switching to range-relative row IDs. In a paired comparison with the previous four-router build, 0 of 34,500 measured query executions changed their exact top-10 hit count. Both centroid layouts have 16 native shards in total.Each cell gives ordinary range / previous partition router / current interval routers.
Run on an Apple M4 Pro, Java 17, Spark
local[4], four native threads, no refinement. Build includes distributed sampling, model training, shuffling, writes and commit. Four families used 300 calibration queries and 2,000 separate measured queries. For IVF-PQ the first measured difference was +1.285 percentage points against ordinary range (78.640% vs 77.355%), outside the matching tolerance. PQ probe settings were recalibrated on 1,200 queries from the original calibration corpus and confirmed on 3,500 previously unused queries. Each layout was rebuilt three times and compared against exact ground truth for this 100k base. Each query opened fresh readers after warming the OS page cache. P95 includes router and shard reads and merging results. Manifest planning, Spark SQL query scheduling and fetching source rows are excluded. Reads include router files and measure logical bytes returned by Paimon. Build times, round P95 and mean bytes/query are medians across the three rounds.Paired Recall@10 differences use percentage points, with 95% intervals from 5,000 bootstrap resamples of the held-out query IDs, averaging each query's three measurements first. The existing matching rule requires an absolute mean difference of at most 1 point and a lower confidence bound above −1 point.
IVF in all layouts used 16 local lists and a training sample ratio of 1. Ordinary IVF probes were 4/4/8/4 for FLAT/SQ/PQ/RQ. The current routers probed two of four centroids per range, with local IVF probes of 8/8/5/12. The previous router probed 6/6/4/4 centroids, with local IVF probes of 8/8/5/8. IVF-PQ used 32-byte codes. DiskANN used graph degree 32, build width 64, 32 PQ subquantizers and F16 raw vectors. Ordinary DiskANN searched with graph width 16. The previous router selected eight shards and the current routers selected two per range, with graph width 64 for both. Run the benchmark with
-Dbenchmark.source-ranges=4to reproduce the interleaved layout.The four-router layout opens 12 files per query. For IVF-PQ, the three layouts open 19, 5 and 12 files. The four routers reduce logical reads by 62% against ordinary range. IVF-RQ reads 3.21 MiB with four routers versus 2.31 MiB with ordinary range. With a query spanning every interval, additional routers add fan-out compared with the previous single partition router.
Joint automatic probes for unfiltered centroid IVF
For an L2 IVF-FLAT or IVF-PQ query with K from 10 through 100 and both widths automatic, a 64-center router that initially selects eight centers can instead search four shards with 16 local probes each. Paimon checks the existing headers of the four nearest shards in parallel. Each must have exactly 64 local lists and at least 16 × K indexed vectors. The regular search path handles filters, explicit widths, other layouts and metrics, other families, and underfilled local searches. Batch planning shares header reads and groups queries by both shard and local policy. Existing index files work directly. The integration uses the existing native API.
On identical pinned SIFT1M and normalized GloVe 500k indexes, 21 paired scenarios covered 8,828 query pairs, including fallback controls. Every enabled query returned the same row-ID / score fingerprint and logical bytes as explicit 4/16. The fallback controls matched the previous automatic reader on every query. In the 2,048-query independent GloVe IVF-PQ K=10 run, the mean recall difference was +0.29 percentage points (paired bootstrap 95% interval −0.22 to +0.80). Exploratory K=1 runs on GloVe had lower bounds below the −1 point threshold, so the shipped policy starts at K=10.
K=100 timing used 384 queries per cell, fresh readers, warm page cache, four native threads and randomized arm order per query. For same-process timing, the old-width control explicitly selects eight outer shards and keeps inner auto. Its per-query IDs, scores and bytes matched the pre-change real auto reader. Files/query decreased from 9 to 5 in all four cells. Byte counts include the router.
Local verification passed 40 vector tests, eight Spark Java tests and nine Spark SQL integration tests on JDK 8 / Spark 3, plus Spotless and Checkstyle. The new native-backed test exercises shared and mixed shard layouts, single and batch search, partial overrides, a row-ID filter, and K outside the tuning window on a single-thread executor.
Automatic outer probes with filters and larger top-K
Local verification passed 32 focused tests on Spark 3 / JDK 8 and all nine centroid SQL integration tests on Spark 4.1 / JDK 17. The Spark 4.0 shared test sources also compiled with the vector index test dependency.
The small-router policy is implemented in the companion native PR. Use a native library containing that change to get the new automatic floor. Earlier 0.7.0-SNAPSHOT builds probe all four or eight outer centroids for these unfiltered queries. Native probe policy
Compared two release JNI builds against the same pinned index snapshot for each pair. The baseline was
e794291. The Paimon benchmark omitscentroid.nprobein both arms. The dataset contains 100,000 SIFT vectors (L2, 128 dimensions) in four disjoint row-ID ranges. Random masks retain exactly 80%, 50% or 5% of rows per range. Ground truth is exhaustive L2 within the eligible set. Each pair used the same 80–512 held-out queries and identical shard-local settings. Inputs came from original SIFT query positions 8000–9249. Fresh readers counted all logical bytes returned by Paimon streams, including the routers, on an Apple M4 Pro with four native threads.All 32 paired scenarios met the existing recall criterion, an absolute mean difference of at most 1 percentage point and a paired 95% bootstrap lower bound above −1 point. The largest measured recall decrease was 0.111 points (IVF-FLAT, 8 centers, K=1000, 100% eligible). The lowest lower bound was -0.180 points. 10,944 paired query measurements were taken across scenarios. Scenarios can reuse query IDs. The table gives old → new.
To reproduce a query-only pair, use
VectorShardingBenchmarkwithbenchmark.centroid-based.nprobes=auto,benchmark.top-kand the samebenchmark.read-snapshoton both runs. For a filter, passbenchmark.allowed-idsand exact ground truth computed over that set. Pointpaimon.vindex.native.pathat each release JNI binary in turn.Earlier single-range SIFT1M results
Each cell below gives ordinary range → centroid for a single range. These previously reported measurements used one million vectors and the earlier option syntax. The four-range measurements above exercise the layout in this PR.
The four IVF results come from earlier runs with independently trained shard models. Build time, P95 and reads are medians over three rounds. IVF-PQ used 32-byte codes in both layouts and 3,500 held-out queries. Each other family used 2,000 held-out queries. All five comparisons met the prespecified matching rule, which requires an absolute mean recall difference of at most one percentage point and a paired 95% bootstrap lower bound above −1 point.
DiskANN was rebuilt once per layout using the unified IVF-FLAT outer sampler. Both builds used 64 shards, graph degree 32, build search width 64, 32 PQ subquantizers and F16 raw vectors. Range searched 64 shards with graph search width 16. Centroid searched 10 shards with width 128. The recall difference was −0.105 percentage points (paired 95% interval [−0.260, +0.055]). Centroid took 8.12 s longer to build and reduced query P95 from 46.75 ms to 13.26 ms.