Skip to content

[python] Stream raw vector batches for local single-query search - #11254

Open
wangzhigang1999 wants to merge 4 commits into
apache:masterfrom
wangzhigang1999:zhigang/python-vector-stream-search
Open

wangzhigang1999 wants to merge 4 commits into
apache:masterfrom
wangzhigang1999:zhigang/python-vector-stream-search

Conversation

@wangzhigang1999

Copy link
Copy Markdown
Contributor

Closes #11253

Purpose

PyPaimon's local executor currently collects all raw vectors for a single query before it scores the first batch. This change passes Arrow batches directly to the existing scorer, which updates one Top-K heap throughout the scan.

With parallel reads, each split group has at most one read in flight and one batch of lookahead. The calling thread scores whichever batch is ready. If scoring fails, it waits for outstanding reads before closing the readers.

Tests

The new tests cover parallel Top-K results, bounded read-ahead, candidate filtering, and reader cleanup after errors. The regression run passed 202 tests and skipped one S3 checksum test that requires PyArrow 22+. Both versions matched the published L2 Top-10 for GIST1M queries 0, 499 and 999. Dataset

Both versions read the same unindexed GIST1M snapshot (1M rows × 960 dimensions, 12 splits). The table shows medians for query 0 over three runs after one warmup per version, comparing master d90a902d with the patch.

Executor Storage Peak scoring-process RSS (MiB), before → after Query time (s), before → after
Local (2 CPUs) Local Parquet 4,865 → 795 (-83.7%) 12.90 → 8.57 (-33.6%)
Local (2 CPUs) OSS 5,158 → 683 (-86.8%) 13.25 → 9.47 (-28.5%)
Ray (2 workers) Worker-local Parquet 268 → 268 (0.0%) 11.69 → 11.54 (-1.3%)
Ray (2 workers) OSS 282 → 279 (-1.1%) 18.35 → 18.75 (+2.2%)

Ray workers already score one split at a time from a batch reader, and their peak RSS stayed close across versions. The Ray rows report the largest scoring worker's peak and include the time to fetch the ten hits.

@wangzhigang1999
wangzhigang1999 force-pushed the zhigang/python-vector-stream-search branch from 7a64f9c to d290df0 Compare November 18, 2026 15:50
for block in batches:
# Bound the Arrow take as well as the scoring matrix for wide vectors.
blocks = batches if score_candidates is None else _slice_batches(batches, _score_block_size(query_vector))
for block in blocks:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In _parallel_raw_batches (line 902), del batch, future is used after yield batch so the just-scored RecordBatch can be freed before waiting on the next read. However, in Python a for block in blocks: loop keeps block, row_ids, and vectors bound in _score_raw_batches's frame while blocks.__next__() is executing to fetch the next batch (and similarly batch remains bound in _slice_batches on line 1137 while batches.__next__() runs).

Because block / vectors (and batch in _slice_batches) share the underlying Arrow buffers, the previous RecordBatch stays pinned in memory throughout the next batch read until the loop variable is reassigned.

Consider explicitly deleting the loop references at the end of each iteration (matching del batch, row_ids, vectors in _score_raw_batch_queries on line 868), and doing del batch at the end of the outer loop in _slice_batches:

        for block in blocks:
            row_ids = block.column(SpecialFields.ROW_ID.name).to_pylist()
            if score_candidates is not None:
                positions = [i for i, row_id in enumerate(row_ids)
                             if row_id in score_candidates]
                if not positions:
                    del block, row_ids
                    continue
                vectors = block.column(self._vector_column.name).take(positions)
                row_ids = [row_ids[i] for i in positions]
            else:
                vectors = block.column(self._vector_column.name)
            for row_id, score in zip(
                row_ids, _iter_arrow_scores(vectors, query_vector, metric)
            ):
                if score is not None:
                    if reject_nan and math.isnan(score):
                        raise ValueError("Ray vector search cannot rank NaN scores.")
                    _offer_score(top_k_heap, self._limit, row_id, score)
            del block, row_ids, vectors

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. I could reproduce the previous Arrow buffer still being held when the reader was asked for the next batch. Fixed in eafb7eb. The regression test checks the buffer lifetime with filtering off, with a matching candidate, and with no matches.

) -> "_BatchStream":
"""Open on the caller thread and register cleanup before submitting the first read."""
reader_scope = streams.enter_context(ExitStack())
reader = reader_scope.enter_context(_open_raw_batch_reader(table_read, splits))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a split group yields multiple batches (>= 2 batches across its splits/row groups), combining _read_one_batch_ahead with submitting next(batches, None) after yield batch has two side effects:

  1. First-batch latency & eager second-batch read: On the initial executor.submit(next, batches, None), _read_one_batch_ahead reads current = next(reader, None) (batch 0) and then immediately enters for following in reader: to read batch 1 before yielding batch 0. The caller cannot start scoring batch 0 until two full batches have been read by that worker, and if scoring fails on the first batch, every multi-batch worker has already read a second batch.
  2. No I/O–compute overlap within a split group: Because pending[executor.submit(next, batches, None)] = batches happens after yield batch, the worker thread for that split group sits idle while the caller scores batch (holding following in its suspended generator frame). Then, once scoring finishes, the next next(batches, None) call cannot return the already-buffered following batch until it finishes reading the subsequent batch from storage.

Would it be simpler to have batches yield one batch per next() call (closing reader_scope in finally on exhaustion) and submit executor.submit(next, batches, None) before yield batch? That way batch 0 can be scored after a single batch read, reading batch k + 1 (or hitting EOF and closing the reader after the last batch) overlaps on the worker thread while the caller scores batch k, and each split group still has at most 1 batch being scored + 1 read in flight.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this ordering. You’re right about the overlap: on split groups with multiple batches, submitting the next read before scoring brought the first batch to the scorer 6–24% sooner across the local Parquet and OSS cases. The caller gets its Top-K after the scan finishes, though, and median end-to-end times changed by less than 5% in all six cases (five measured runs each).

The deciding case for me was 32 single-batch splits on OSS. Both versions passed 20/20 runs under a 6 GiB virtual address space limit, but median peak RSS rose from 299 to 347 MiB with early submission. The current lookahead reaches EOF and closes each reader before handing its last batch to the scorer. With early submission, that close can still be waiting on a worker. Memory results varied by layout (early submission did save 98 MiB on the large local table), so I’m keeping the current ordering for this PR to preserve early close on small splits. Thanks for suggesting the comparison.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Streaming raw search has clear end-to-end value. I ran the local Python regression selection with 186 passing tests and verified the parallel ranking issue below with a committed partitioned Parquet table against both this head and master.

batches = _open_prefetched_batches(table_read, group, streams)
pending[executor.submit(next, batches, None)] = batches
while pending:
completed = wait(pending, return_when=FIRST_COMPLETED).done

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep local Top-K stable when raw vectors contain NaN

The old materializer scored splits in plan order. Scoring batches as soon as they are ready exposes the heap to non-total ordering for NaN. For limit=1, a NaN score pushed first cannot be replaced by a later finite score. I committed [1.0, 0.0] and [NaN, 0.0] to two partitions of an unindexed Data Evolution Parquet table, then ran local L2 search for [1.0, 0.0] at read.parallelism=2. Delaying only the finite split returned the NaN row on this head. Delaying only the NaN split returned the exact match. Master returned the exact match with either delay because it scored in plan order. Please define a deterministic non-finite score policy before inserting into the heap, and test both completion orders. For example, consistently rank NaN after finite scores with the same row-ID tie break.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. I’ve kept the existing NaN behavior for local and Ray single-vector searches and made local Top-K selection deterministic.

Local search ranks NaN below every numeric score and breaks ties by smaller row ID, including ties between NaNs. If fewer than K candidates have numeric scores, the remaining slots can contain NaN rows with their original scores. Top-K selection and order_by_score() use the same ordering. Ray still raises an error as soon as a worker computes a NaN score, before its local Top-K can discard that row.

I tested the same local Parquet table with two splits and forced each split to finish first in turn. Rows 1 and 2 both scored 1.0, and rows 0 and 3 scored NaN. Both completion orders returned row 1 for K=1 and [1, 2, 0, 3] for K=4, with both NaN scores preserved. I also ran the two-split test on Ray in both orders. A NaN score failed the query in both runs, and equal numeric scores selected the smaller row ID in both runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Reduce peak memory for local vector search in PyPaimon

3 participants