Skip to content

[spark] Support centroid-based sharding for vector global indexes - #11249

Draft
JingsongLi wants to merge 14 commits into
apache:masterfrom
JingsongLi:codex/centroid-sharding
Draft

JingsongLi wants to merge 14 commits into
apache:masterfrom
JingsongLi:codex/centroid-sharding

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Add five Spark global index types for Data Evolution append tables. Use ivf-flat-centroid, ivf-sq-centroid, ivf-pq-centroid, ivf-rq-centroid or diskann-centroid in sys.create_global_index. Options use the same prefix, for example ivf-pq-centroid.nlist=64. Partitions can use both ordinary and centroid types for the same vector column. Readers search each selected range with its registered index type and merge the scores for the final top-K. Set one distance metric for the column across partitions. PIP-47

Spark uses the same contiguous row-ID grouping as BTree. Each such range within a partition with any non-null vectors gets its own router and native shards. Spark pins a snapshot, counts vectors per split and plans a deterministic sample with the ordinary IVF-FLAT trainer. Each map task reads the selected rows from its own splits. For each range, one task trains a router on the sampled rows in reservoir order. Spark broadcasts the router, shuffles by centroid ID and builds a local IVF or DiskANN index in each populated shard. A single commit replaces the selected partitions' files of the requested type.

The router and every associated shard carry the same GlobalIndexMeta.rowRange. Planning, visibility, raw fallback and conflict checks use the standard range. A gap can therefore be searched from source data when the query uses detail or chunk mode. The centroid writer stores each assigned row’s offset from the start of the group range. Both ordinary and centroid readers use the standard OffsetGlobalIndexReader to translate filter bitmaps and restore table row IDs for single and batch searches. Row-ID reassignment can reuse files when the whole group range moves by a constant offset. The plugin-owned GlobalIndexMeta.indexMeta identifies the router generation and shard count or the shard's centroid ID and underlying local writer metadata. Upgrading test deployments from either of this PR’s earlier layouts with absolute row IDs requires dropping those indexes with the previous build and rebuilding them.

The native router and deterministic sample planner are available in Apache’s published paimon-vector-index-java:0.7.0-SNAPSHOT. Validation used 0.7.0-20261119.003537-3. apache/paimon-vector-index#131

Tests

  • 503 focused tests passed on JDK 8, Spark 3 and Scala 2.12, including nine Spark SQL integration tests. They cover BTree and centroid range agreement, searches inside an unindexed gap, range-relative shard IDs, single and batch filtering at a nonzero range start, queries after row-ID reassignment without rebuilding the index files, deletes, snapshot conflicts, atomic replacement, mixed index types, multi-partition builds, all-null ranges and the shared scalar filtering path for vector and full-text searches. Spotless and Checkstyle passed on the five affected modules.
  • The earlier single-range training parity test selected the same 65,536 sample rows on SIFT1M for all five families. The serialized router matched streaming IVF-FLAT training, including with DiskANN seed 93 and a 4 MiB local build budget.

Four disjoint ranges on SIFT100k

Use the first 100,000 SIFT1M vectors (128D, L2). The target partition contains four runs of 25,000 rows separated by three rows allocated to another partition. The ordinary range builder produced 19 aligned shards at 6,251 rows per shard. The previous centroid layout at 643346bdd9 has one router and 16 shards. The current layout has four routers and four shards per router. Every query visits all four intervals. The ordinary and four-router layouts were rebuilt and timed after switching to range-relative row IDs. In a paired comparison with the previous four-router build, 0 of 34,500 measured query executions changed their exact top-10 hit count. Both centroid layouts have 16 native shards in total.

Each cell gives ordinary range / previous partition router / current interval routers.

Index Recall@10 (%) Build (s) Query P95 (ms) Files/query Logical MiB/query
IVF-FLAT 98.790 / 98.425 / 99.295 0.18 / 0.54 / 0.74 1.99 / 1.66 / 1.62 19 / 7 / 12 13.13 / 10.13 / 12.82
IVF-SQ 97.835 / 97.530 / 98.400 0.16 / 0.53 / 0.72 1.70 / 1.09 / 1.11 19 / 7 / 12 3.73 / 2.70 / 3.42
IVF-PQ 78.111 / 78.463 / 78.323 6.45 / 7.08 / 6.82 3.91 / 1.64 / 1.99 19 / 5 / 12 4.35 / 0.86 / 1.67
IVF-RQ 90.715 / 90.705 / 91.680 0.23 / 0.58 / 0.95 1.67 / 0.79 / 1.46 19 / 5 / 12 2.31 / 1.24 / 3.21
DiskANN 99.450 / 99.785 / 99.785 9.08 / 9.73 / 10.21 8.41 / 5.26 / 5.06 19 / 9 / 12 10.15 / 6.87 / 6.79

Run on an Apple M4 Pro, Java 17, Spark local[4], four native threads, no refinement. Build includes distributed sampling, model training, shuffling, writes and commit. Four families used 300 calibration queries and 2,000 separate measured queries. For IVF-PQ the first measured difference was +1.285 percentage points against ordinary range (78.640% vs 77.355%), outside the matching tolerance. PQ probe settings were recalibrated on 1,200 queries from the original calibration corpus and confirmed on 3,500 previously unused queries. Each layout was rebuilt three times and compared against exact ground truth for this 100k base. Each query opened fresh readers after warming the OS page cache. P95 includes router and shard reads and merging results. Manifest planning, Spark SQL query scheduling and fetching source rows are excluded. Reads include router files and measure logical bytes returned by Paimon. Build times, round P95 and mean bytes/query are medians across the three rounds.

Paired Recall@10 differences use percentage points, with 95% intervals from 5,000 bootstrap resamples of the held-out query IDs, averaging each query's three measurements first. The existing matching rule requires an absolute mean difference of at most 1 point and a lower confidence bound above −1 point.

Index Current minus ordinary range (95% CI) Match Current minus previous router (95% CI) Match
IVF-FLAT +0.505 [+0.285, +0.725] Yes +0.870 [+0.635, +1.110] Yes
IVF-SQ +0.565 [+0.305, +0.820] Yes +0.870 [+0.595, +1.135] Yes
IVF-PQ +0.211 [-0.194, +0.623] Yes -0.140 [-0.569, +0.297] Yes
IVF-RQ +0.965 [+0.555, +1.365] Yes +0.975 [+0.550, +1.385] Yes
DiskANN +0.335 [+0.200, +0.470] Yes +0.000 [-0.110, +0.105] Yes

IVF in all layouts used 16 local lists and a training sample ratio of 1. Ordinary IVF probes were 4/4/8/4 for FLAT/SQ/PQ/RQ. The current routers probed two of four centroids per range, with local IVF probes of 8/8/5/12. The previous router probed 6/6/4/4 centroids, with local IVF probes of 8/8/5/8. IVF-PQ used 32-byte codes. DiskANN used graph degree 32, build width 64, 32 PQ subquantizers and F16 raw vectors. Ordinary DiskANN searched with graph width 16. The previous router selected eight shards and the current routers selected two per range, with graph width 64 for both. Run the benchmark with -Dbenchmark.source-ranges=4 to reproduce the interleaved layout.

The four-router layout opens 12 files per query. For IVF-PQ, the three layouts open 19, 5 and 12 files. The four routers reduce logical reads by 62% against ordinary range. IVF-RQ reads 3.21 MiB with four routers versus 2.31 MiB with ordinary range. With a query spanning every interval, additional routers add fan-out compared with the previous single partition router.

Joint automatic probes for unfiltered centroid IVF

For an L2 IVF-FLAT or IVF-PQ query with K from 10 through 100 and both widths automatic, a 64-center router that initially selects eight centers can instead search four shards with 16 local probes each. Paimon checks the existing headers of the four nearest shards in parallel. Each must have exactly 64 local lists and at least 16 × K indexed vectors. The regular search path handles filters, explicit widths, other layouts and metrics, other families, and underfilled local searches. Batch planning shares header reads and groups queries by both shard and local policy. Existing index files work directly. The integration uses the existing native API.

On identical pinned SIFT1M and normalized GloVe 500k indexes, 21 paired scenarios covered 8,828 query pairs, including fallback controls. Every enabled query returned the same row-ID / score fingerprint and logical bytes as explicit 4/16. The fallback controls matched the previous automatic reader on every query. In the 2,048-query independent GloVe IVF-PQ K=10 run, the mean recall difference was +0.29 percentage points (paired bootstrap 95% interval −0.22 to +0.80). Exploratory K=1 runs on GloVe had lower bounds below the −1 point threshold, so the shipped policy starts at K=10.

K=100 timing used 384 queries per cell, fresh readers, warm page cache, four native threads and randomized arm order per query. For same-process timing, the old-width control explicitly selects eight outer shards and keeps inner auto. Its per-query IDs, scores and bytes matched the pre-change real auto reader. Files/query decreased from 9 to 5 in all four cells. Byte counts include the router.

Dataset Index Recall@100 (%) Logical MiB/query P95 (ms)
SIFT1M IVF-FLAT 82.22 → 86.06 8.66 → 8.60 2.12 → 1.84
SIFT1M IVF-PQ 74.78 → 76.66 1.93 → 1.37 4.04 → 2.46
GloVe 500k IVF-FLAT 56.20 → 59.57 3.56 → 3.54 1.44 → 1.18
GloVe 500k IVF-PQ 49.58 → 50.98 1.41 → 0.97 4.33 → 2.48

Local verification passed 40 vector tests, eight Spark Java tests and nine Spark SQL integration tests on JDK 8 / Spark 3, plus Spotless and Checkstyle. The new native-backed test exercises shared and mixed shard layouts, single and batch search, partial overrides, a row-ID filter, and K outside the tuning window on a single-thread executor.

Automatic outer probes with filters and larger top-K

Local verification passed 32 focused tests on Spark 3 / JDK 8 and all nine centroid SQL integration tests on Spark 4.1 / JDK 17. The Spark 4.0 shared test sources also compiled with the vector index test dependency.

The small-router policy is implemented in the companion native PR. Use a native library containing that change to get the new automatic floor. Earlier 0.7.0-SNAPSHOT builds probe all four or eight outer centroids for these unfiltered queries. Native probe policy

Compared two release JNI builds against the same pinned index snapshot for each pair. The baseline was e794291. The Paimon benchmark omits centroid.nprobe in both arms. The dataset contains 100,000 SIFT vectors (L2, 128 dimensions) in four disjoint row-ID ranges. Random masks retain exactly 80%, 50% or 5% of rows per range. Ground truth is exhaustive L2 within the eligible set. Each pair used the same 80–512 held-out queries and identical shard-local settings. Inputs came from original SIFT query positions 8000–9249. Fresh readers counted all logical bytes returned by Paimon streams, including the routers, on an Apple M4 Pro with four native threads.

All 32 paired scenarios met the existing recall criterion, an absolute mean difference of at most 1 percentage point and a paired 95% bootstrap lower bound above −1 point. The largest measured recall decrease was 0.111 points (IVF-FLAT, 8 centers, K=1000, 100% eligible). The lowest lower bound was -0.180 points. 10,944 paired query measurements were taken across scenarios. Scenarios can reuse query IDs. The table gives old → new.

Index Centroids per range Top-K Eligible rows Recall@K (%) Files/query Logical MiB/query
IVF-FLAT 4 100 100% 98.645 → 98.645 20 → 16 23.40 → 17.14
IVF-FLAT 4 100 80% 98.561 → 98.561 20 → 20 23.40 → 23.40
IVF-FLAT 8 100 80% 97.934 → 97.932 36 → 24 23.37 → 14.91
IVF-SQ 8 100 80% 97.781 → 97.779 36 → 24 6.74 → 4.32
IVF-PQ 8 100 80% 84.138 → 84.135 36 → 24 5.64 → 3.55
IVF-RQ 8 100 80% 95.320 → 95.318 36 → 24 6.52 → 4.12
DISKANN 8 100 80% 100.000 → 99.997 36 → 24 32.19 → 19.75
IVF-FLAT 8 10 5% 97.422 → 97.422 36 → 36 23.37 → 23.37
IVF-FLAT 8 1,000 100% 94.468 → 94.357 36 → 20 23.38 → 12.30
IVF-PQ 8 1,000 80% 80.648 → 80.645 36 → 24 5.64 → 3.55
DISKANN 8 1,000 80% 100.000 → 99.994 36 → 24 35.17 → 21.18
IVF-FLAT 4 5,000 100% 90.586 → 90.586 20 → 20 23.40 → 23.40
IVF-FLAT 8 5,000 100% 84.634 → 84.634 36 → 32 23.37 → 20.45

To reproduce a query-only pair, use VectorShardingBenchmark with benchmark.centroid-based.nprobes=auto, benchmark.top-k and the same benchmark.read-snapshot on both runs. For a filter, pass benchmark.allowed-ids and exact ground truth computed over that set. Point paimon.vindex.native.path at each release JNI binary in turn.

Earlier single-range SIFT1M results

Each cell below gives ordinary range → centroid for a single range. These previously reported measurements used one million vectors and the earlier option syntax. The four-range measurements above exercise the layout in this PR.

Index Recall@10 (%) Build (s) Query P95 (ms) Files/query MiB/query
IVF-FLAT 99.000 → 99.220 3.32 → 6.87 9.22 → 5.41 64 → 25 67.18 → 37.16
IVF-SQ 98.015 → 97.950 3.81 → 4.64 7.58 → 3.38 64 → 11 22.54 → 11.03
IVF-PQ 72.886 → 73.094 66.80 → 67.41 33.08 → 1.69 64 → 5 14.98 → 1.11
IVF-RQ 90.075 → 90.550 3.44 → 5.89 6.20 → 2.09 64 → 13 13.04 → 3.46
DiskANN 99.460 → 99.355 98.13 → 106.25 46.75 → 13.26 64 → 11 82.49 → 16.98

The four IVF results come from earlier runs with independently trained shard models. Build time, P95 and reads are medians over three rounds. IVF-PQ used 32-byte codes in both layouts and 3,500 held-out queries. Each other family used 2,000 held-out queries. All five comparisons met the prespecified matching rule, which requires an absolute mean recall difference of at most one percentage point and a paired 95% bootstrap lower bound above −1 point.

DiskANN was rebuilt once per layout using the unified IVF-FLAT outer sampler. Both builds used 64 shards, graph degree 32, build search width 64, 32 PQ subquantizers and F16 raw vectors. Range searched 64 shards with graph search width 16. Centroid searched 10 shards with width 128. The recall difference was −0.105 percentage points (paired 95% interval [−0.260, +0.055]). Centroid took 8.12 s longer to build and reduced query P95 from 46.75 ms to 13.26 ms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant