Skip to content

[core] Deduplicate initial listagg tokens - #11256

Open
zhangyue19921010 wants to merge 2 commits into
apache:masterfrom
zhangyue19921010:contrib/deduplicate-initial-listagg-tokens
Open

zhangyue19921010 wants to merge 2 commits into
apache:masterfrom
zhangyue19921010:contrib/deduplicate-initial-listagg-tokens

Conversation

@zhangyue19921010

Copy link
Copy Markdown
Contributor

Purpose

Distinct listagg retains duplicate tokens from the first input: merging "a,a" and "b" produces "a,a,b", including after full compaction. Normalize the accumulator before appending unseen tokens, including when partial-update sequence groups reverse the inputs. Keep non-distinct and empty-delimiter behavior unchanged.

Fixes #11255

Tests

PrimaryKeyListaggDistinctTest covers aggregation and partial-update tables, same/separate commits, custom delimiters, reversed sequence groups, and full compaction, with non-distinct and empty-delimiter controls.

On unchanged master 79b8ec4788, 12 regression cases fail and 6 compatibility controls pass. With this change, all 38 selected table, listagg, and partial-update tests pass on JDK 11. Normal Maven verify also passes Checkstyle, Spotless, Enforcer, and RAT.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The distinct-listagg correctness fix has end-to-end value. I ran normal Maven verify on JDK 11. All 86 selected tests passed, including the real-table read and compaction cases and the partial-update merge tests. I found one hot-path regression in the already-normalized accumulator case.

existingTokens.add(token);
continue;
}
if (StringUtils.isBlank(token.toString()) || !existingTokens.add(token)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Avoid rebuilding an already-normalized accumulator on repeated values

Each merge now decodes every existing token and builds a second list of tokens and delimiters, even when the accumulator is already distinct and the input is a value it contains. That rebuilt list is discarded at line 115. For a group collecting tags, every repeated tag pays this cost during primary-key reads and compaction. I compared the master and PR aggregators against the same dependencies on JDK 11, using 512 distinct category_N tokens and 6,000 updates with category_0 per round. The outputs matched. In two JVMs per variant after warm-up, allocation rose from about 50.5 to 81.9 KB per merge and the warmed rounds took roughly 1.7–1.9× as long. Could we keep the original accumulator in the common case, materialize normalized output only after finding a token that must be removed, and avoid allocating a Java string for each existing token when checking whether it is blank?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0d4dff7: the original accumulator is now kept and only rebuilt after the first blank or duplicate token is found, and blank checks scan the UTF-8 bytes instead of decoding (falling back only for non-ASCII). On your 512-token category_0 case this is now 42.9 KB and ~21 µs per merge, vs. 92.0 KB / ~43 µs before and 49.4 KB / ~23 µs on master.

@zhangyue19921010

Copy link
Copy Markdown
Contributor Author

closing and reopen to trigger CI rerun

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Distinct listagg retains repeated tokens from the first input

2 participants