Skip to content

[core] Preserve deletion vector changes in snapshot diffs - #11258

Merged
JingsongLi merged 2 commits into
apache:masterfrom
zhangyue19921010:contrib/preserve-dv-changes-in-snapshot-diffs
Nov 19, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
zhangyue19921010:contrib/preserve-dv-changes-in-snapshot-diffs

Conversation

@zhangyue19921010

Copy link
Copy Markdown
Contributor

Purpose

Primary-key snapshot diffs can miss deletes recorded only in deletion vectors: the planner removes a shared data file from both sides even though its visible rows changed. They can also report an update whose value did not change.

Compare the deletion-file descriptors before pruning shared files from primary-key diffs. Files whose descriptors are unchanged remain pruned.

Fixes #11257

Tests

Six real-table cases cover audit-log deletes, repeated vector changes, changed and unchanged values, unchanged-vector file pruning, and a non-DV control. On unchanged master, three regressions fail and three controls pass; all six pass with the patch.

All 23 selected table, timestamp-incremental, snapshot-reader, streaming follow-up, and audit-log tests pass on JDK 11 with normal Maven verify, including Checkstyle, Spotless, Enforcer, and RAT.

Comment on lines +594 to +595
|| Objects.equals(
beforeVectors.get(file), afterVectors.get(file)));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep pruning files whose deletion bitmap is unchanged

BucketedDvMaintainer.writeDeletionVectorsIndex() rewrites all DVs in a bucket whenever one changes. DeletionFile.equals() includes the index path, so a bitmap copied unchanged into the new index compares unequal here. This can make a single-row delete trigger reads of every data file with a DV in the bucket, once for each snapshot.

I reproduced this on this PR with a primary-key table (k INT, v INT) using bucket=1, deletion-vectors.enabled=true, and compaction.small-file-ratio=0 to keep two separate files. Each commit waited for compaction.

  1. Write (1,10), (2,20), (3,30) in one commit and (10,100), (11,110), (12,120) in another.
  2. Delete keys 1 and 10 in one commit, then record the snapshot ID.
  3. Delete key 2 and run an incremental-between query from the recorded snapshot to the latest with incremental-between-scan-mode=diff.

The diff correctly emits DELETE (2,20), but its split contains two files on each side. I read both versions of the second file's DV and confirmed that the bitmaps are identical. The split only needs the first file on each side.

Could we keep descriptor equality as the fast path, then compare bitmap contents when the descriptors differ and their cardinalities are equal or unknown? Different known cardinalities can keep the file without a bitmap read. The two-file case would also make a useful regression test for pruning.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in dd73c4a: descriptor equality stays the fast path, different known cardinalities keep the file without a read, and otherwise the two bitmaps are read and compared, so a vector copied unchanged into the rewritten index keeps its file pruned. Added your two-file case as a regression test; it saw two files per side before this change and one now.

@JingsongLi
JingsongLi merged commit 50d7b68 into apache:master Nov 19, 2026
19 checks passed
@zhangyue19921010

Copy link
Copy Markdown
Contributor Author

Thanks for your help! @JingsongLi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Primary-key snapshot diffs miss deletion vector changes

2 participants