Repository navigation
[core] Preserve deletion vector changes in snapshot diffs - #11258
JingsongLi merged 2 commits into
Conversation
| || Objects.equals( | ||
| beforeVectors.get(file), afterVectors.get(file))); |
There was a problem hiding this comment.
[P2] Keep pruning files whose deletion bitmap is unchanged
BucketedDvMaintainer.writeDeletionVectorsIndex() rewrites all DVs in a bucket whenever one changes. DeletionFile.equals() includes the index path, so a bitmap copied unchanged into the new index compares unequal here. This can make a single-row delete trigger reads of every data file with a DV in the bucket, once for each snapshot.
I reproduced this on this PR with a primary-key table (k INT, v INT) using bucket=1, deletion-vectors.enabled=true, and compaction.small-file-ratio=0 to keep two separate files. Each commit waited for compaction.
- Write
(1,10), (2,20), (3,30)in one commit and(10,100), (11,110), (12,120)in another. - Delete keys
1and10in one commit, then record the snapshot ID. - Delete key
2and run anincremental-betweenquery from the recorded snapshot to the latest withincremental-between-scan-mode=diff.
The diff correctly emits DELETE (2,20), but its split contains two files on each side. I read both versions of the second file's DV and confirmed that the bitmaps are identical. The split only needs the first file on each side.
Could we keep descriptor equality as the fast path, then compare bitmap contents when the descriptors differ and their cardinalities are equal or unknown? Different known cardinalities can keep the file without a bitmap read. The two-file case would also make a useful regression test for pruning.
There was a problem hiding this comment.
Done in dd73c4a: descriptor equality stays the fast path, different known cardinalities keep the file without a read, and otherwise the two bitmaps are read and compared, so a vector copied unchanged into the rewritten index keeps its file pruned. Added your two-file case as a regression test; it saw two files per side before this change and one now.
|
Thanks for your help! @JingsongLi |
Purpose
Primary-key snapshot diffs can miss deletes recorded only in deletion vectors: the planner removes a shared data file from both sides even though its visible rows changed. They can also report an update whose value did not change.
Compare the deletion-file descriptors before pruning shared files from primary-key diffs. Files whose descriptors are unchanged remain pruned.
Fixes #11257
Tests
Six real-table cases cover audit-log deletes, repeated vector changes, changed and unchanged values, unchanged-vector file pruning, and a non-DV control. On unchanged master, three regressions fail and three controls pass; all six pass with the patch.
All 23 selected table, timestamp-incremental, snapshot-reader, streaming follow-up, and audit-log tests pass on JDK 11 with normal Maven
verify, including Checkstyle, Spotless, Enforcer, and RAT.