Repository navigation
[core] Deduplicate initial listagg tokens - #11256
zhangyue19921010 wants to merge 2 commits into
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
The distinct-listagg correctness fix has end-to-end value. I ran normal Maven verify on JDK 11. All 86 selected tests passed, including the real-table read and compaction cases and the partial-update merge tests. I found one hot-path regression in the already-normalized accumulator case.
| existingTokens.add(token); | ||
| continue; | ||
| } | ||
| if (StringUtils.isBlank(token.toString()) || !existingTokens.add(token)) { |
There was a problem hiding this comment.
[P2] Avoid rebuilding an already-normalized accumulator on repeated values
Each merge now decodes every existing token and builds a second list of tokens and delimiters, even when the accumulator is already distinct and the input is a value it contains. That rebuilt list is discarded at line 115. For a group collecting tags, every repeated tag pays this cost during primary-key reads and compaction. I compared the master and PR aggregators against the same dependencies on JDK 11, using 512 distinct category_N tokens and 6,000 updates with category_0 per round. The outputs matched. In two JVMs per variant after warm-up, allocation rose from about 50.5 to 81.9 KB per merge and the warmed rounds took roughly 1.7–1.9× as long. Could we keep the original accumulator in the common case, materialize normalized output only after finding a token that must be removed, and avoid allocating a Java string for each existing token when checking whether it is blank?
There was a problem hiding this comment.
Done in 0d4dff7: the original accumulator is now kept and only rebuilt after the first blank or duplicate token is found, and blank checks scan the UTF-8 bytes instead of decoding (falling back only for non-ASCII). On your 512-token category_0 case this is now 42.9 KB and ~21 µs per merge, vs. 92.0 KB / ~43 µs before and 49.4 KB / ~23 µs on master.
|
closing and reopen to trigger CI rerun |
Purpose
Distinct listagg retains duplicate tokens from the first input: merging
"a,a"and"b"produces"a,a,b", including after full compaction. Normalize the accumulator before appending unseen tokens, including when partial-update sequence groups reverse the inputs. Keep non-distinct and empty-delimiter behavior unchanged.Fixes #11255
Tests
PrimaryKeyListaggDistinctTestcovers aggregation and partial-update tables, same/separate commits, custom delimiters, reversed sequence groups, and full compaction, with non-distinct and empty-delimiter controls.On unchanged master
79b8ec4788, 12 regression cases fail and 6 compatibility controls pass. With this change, all 38 selected table, listagg, and partial-update tests pass on JDK 11. Normal Mavenverifyalso passes Checkstyle, Spotless, Enforcer, and RAT.