Skip to content

[core] Sort Data Evolution file groups by user-specified columns - #11262

Open
JingsongLi wants to merge 6 commits into
apache:masterfrom
JingsongLi:codex/sort-data-evolution
Open

JingsongLi wants to merge 6 commits into
apache:masterfrom
JingsongLi:codex/sort-data-evolution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Selective queries on a Data Evolution append table can open many data, BLOB and vector files when values of the filter column are scattered across file groups. This prototype adds sys.sort_data_evolution to Spark and Flink batch SQL so callers can cluster a chosen part of the table by the columns they query.

CALL sys.sort_data_evolution('default.samples', 0, 99999, 'tenant_id,event_time', 'p = 7');

Use the $row_ranges system table to choose the bounds. Each row describes the smallest independently rewritable range in a partition at a single snapshot. The file count and total bytes include ordinary, BLOB and vector files.

SELECT start_row_id, end_row_id, file_count, file_size_in_bytes, snapshot_id
FROM `samples$row_ranges`
WHERE `partition` = '{7}'
ORDER BY start_row_id;

The input row-ID bounds are inclusive. The optional final argument filters partitions. The planner checks every intersecting file in the planning snapshot and rejects a boundary that cuts a file group before any writes start. Each selected partition gets one spillable sort. The sort uses ascending keys with nulls first and uses the old row ID to break ties.

The rewrite reads the latest visible values, applies deletion vectors, and writes ordinary columns, managed BLOBs and dedicated vectors in the same new row order. Surviving rows receive new IDs. Downstream references to those IDs need updating, and global indexes in rewritten partitions need rebuilding.

Tests

  • A Flink SQL test connects three normal file groups through a spanning BLOB and a spanning vector. It feeds the returned bounds into the sort procedure and checks the rewritten rows and the untouched adjacent group. The Spark SQL test selects bounds by partition. A core test advances the table between scan planning and reading and checks the original snapshot.
  • Spark 3.5.9 and Flink 1.20.1 and 2.2.0 SQL integration tests read back scalar, BLOB and vector values by new row ID. They also check physical file replacement, preservation of data outside the selection, partial-group rejection and repeated calls.
  • The two targeted core suites passed all 100 tests, including missing input and an index built after rewrite planning. The Flink 1.20.1 run also passed all 13 tests in DataEvolutionDeleteSqlITCase.

SortDataEvolutionBenchmark compared ordinary compaction and sorting on branches from the same 32,768-row snapshot. Both layouts had eight main data files with matching row counts. The query selected 64 rows by the sort key and read scalar, BLOB and vector values. File and byte counts come from an instrumented table read. SQL timings use 20 local Spark runs per layout after warm-up.

Local Spark measurement Ordinary compaction Sorted rewrite
Distinct data files opened 24 3
Data bytes read 2,445,853 307,224
Query p50 82.1 ms 42.6 ms
Query p95 104.1 ms 49.9 ms
Maintenance elapsed time 1.08 s 2.89 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant