Repository navigation
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
The typed timestamp extension is useful for epoch and timestamp fields. The head passed 119 Java tests and 167 Python tests. It conflicts with master after #11043 in the Java aggregator/factory/tests and the Python aggregator. Please preserve merge-map.overwrite-on-equal-ts for both string and typed comparisons when rebasing.
JingsongLi
left a comment
There was a problem hiding this comment.
A second interoperability check for the newly supported timestamp precision.
860ec82 to
77d3fa4
Compare
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS. DECIMAL extends the Java types supported after #11084, and the public Python nanosecond-read fix has real correctness value. The previous DOUBLE and TIMESTAMP(9)/TIMESTAMP_LTZ(9) examples now pass, and overwrite-on-equal behavior is preserved.
Reviewed all eleven files, generic row conversion, Java/Python comparison and native routing. 128 Java field-aggregator tests and 194 Python tests passed. Two additional Java table probes passed normal verify: DECIMAL precision10/25, actual writes, reopened reads, equal-timestamp policy and compaction. Additional real filesystem probes reproduce two remaining compatibility/interop problems below: three exact-baseline legacy controls succeed but current readers fail, and native/Python readers choose different string-timestamp winners. The native probe uses Python planning and omits only scheduling kwargs unsupported by the installed Rust0.4 wheel; Python read fallback is forbidden. Current public Rust main independently still uses the same UTF-8 string comparison. Project flake8 and diff whitespace checks passed.
6e5cd5b to
472f557
Compare
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
The previous legacy Python BINARY/VARBINARY/BOOLEAN compatibility and public nanosecond-read issues are fixed. 133 Java tests and 203 Python tests passed (122 Python subtests), as did eight real Parquet DECIMAL table probes across aggregation/partial-update, overwrite policy and compaction. Normal Maven checks, project flake8 and git diff --check passed.
Two Native entry paths still disagree with the new UTF-16 comparison. I independently reproduced the native-writer issue, and additionally read the exact Java-written partial-update table through Native with Python fallback forbidden. Exact-baseline and ASCII controls support the regression; details are inline. Probes use supported Python 3.11 / PyArrow 22 / Rust 0.5.0.
| # nanosecond tail after Arrow row materialization. Equal timestamps | ||
| # replace only when overwrite-on-equal-ts is enabled. | ||
| if self._string_timestamp: | ||
| if _utf16_code_unit_greater(new_ts, existing_ts): |
There was a problem hiding this comment.
[P2] Gate Native writes for the new string comparison too
The new UTF-16 comparison is applied in Python and Java, but Native writes remain eligible and reduce map entries with Rust's UTF-8 string ordering before any read fallback. On a real PK aggregation table, one public write_arrow batch contains the same PK/map key first at timestamp U+E000 (first), then U+10000 (second). The actual Rust 0.5.0 NativeBatchTableWrite persists only second; reopening via the explicitly Python reader cannot recover the UTF-16 winner first. Two single-record Python commits retain first. The exact merge-base selects second consistently, and current ASCII controls agree, so this is introduced by the comparator change rather than a tie/ordering artifact.
Please gate Native write reduction for this aggregator until its comparator matches (or require a compatible runtime), and add a persisted-table write/read regression. Reader fallback alone cannot repair a value already discarded during writing.
There was a problem hiding this comment.
Thanks for pointing this out. Fixed in the latest commit.
Native Writer is now bypassed for merge_map_with_keytime tables, and the Python fallback handles Arrow MAP entries materialized as (key, value) tuples. Added a regression test using U+E000 and U+10000 in the same write_arrow batch, verifying that the UTF-16 ordering is preserved after commit and reopen.
| if not getattr(table, 'is_primary_key_table', False): | ||
| return True | ||
| options = table.options | ||
| if options.merge_engine() != MergeEngine.AGGREGATE: |
There was a problem hiding this comment.
[P2] Cover partial-update aggregators in the Native capability check
merge_map_with_keytime can also be used by Java partial-update tables with a sequence group, so returning true for every non-aggregation engine still permits the same comparator mismatch. I wrote a real Parquet PK table through the reviewed Java writer with top-level seq 1/2, fields.seq.sequence-group=updates, and nested string timestamps U+E000 then U+10000. Java reads first. Loading those exact persisted files in PyPaimon with Native enabled returns second, with Python fallback forbidden; this gate returns true.
Please account for configured field aggregators in the partial-update path too. If the supported Native runtime cannot match these semantics and Python cannot execute the configuration, fail explicitly before opening a Native read or require a matching runtime rather than returning a different value.
There was a problem hiding this comment.
Thanks for catching this case. Fixed in the latest commit.
The Native Reader gate now also detects merge_map_with_keytime under the Partial Update merge engine, including sequence-group fields. Since the Python Reader does not support this Partial Update aggregation path, it now raises NotImplementedError before opening the Native Reader instead of silently returning an incorrect result. Added a regression test for this behavior.
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS. DECIMAL ordering has a useful persisted-table use case, and the historical Java UTF-16 string behavior is restored. The previously reported ordinary Native writer and Java-written partial-update reader cases are addressed on this head. Three remaining P2 issues are detailed inline.
Validation on this exact head: 201 Java aggregation/partial-update tests and 208 Python tests passed. Eight real Parquet DECIMAL table probes passed across compact/wide DECIMAL, aggregation/partial-update sequence groups, both equal-timestamp policies, multiple commits, compaction and reopening. A real Java-written Unicode partial-update table now fails explicitly before Native opens. Explicitly enabled Rust 0.5 batch writer/read probes preserve the UTF-16 winner with both field-specific and default aggregator options. Additional real Native probes reproduce the MERGE source mismatch and the effective-aggregator false positive. The exact merge-base control agrees across both MERGE source forms.
Normal Maven checks and git diff --check passed. Changed-file project flake8 fails with the same E123 that stops the Python jobs in CI43866912063; therefore the PR is not CI-green. The separate Spark 3 job stopped resolving a Maven plugin dependency, which I have not attributed to this diff. No production source changes were made.
| "the table uses an unsupported BLOB view") | ||
| # Keytime: Aggregate falls back to Python; Partial Update has no | ||
| # correct Python path either, so fail before Native opens. | ||
| if not _native_merge_map_keytime_timestamp_supported(read.table): |
There was a problem hiding this comment.
[P2] Apply the keytime capability check to Native MERGE table sources
This only protects NativeTableRead. create_native_merge_into still wraps an aggregation/partial-update PK source with create_native_write_table and lets Rust read it internally, bypassing this gate. With real Rust 0.5.0, I persisted two source versions for the same PK/map key: U+E000/first, then U+10000/second. The public source read now returns first, but Native MERGE INTO a DE target using that table as source writes second. The identical Native MERGE using the public read's materialized Arrow source writes first; Python MERGE matching was forbidden in both probes. On the exact merge-base both source forms write second, confirming the new semantic disagreement. Please qualify table sources with the same capability contract before Native staging and use a correct logical source route or reject unsupported partial-update sources explicitly. Cover the committed target result, since the ordinary reader fallback cannot repair this write.
|
|
||
| options_map = options.options.to_map() | ||
| pk_set = set(table.primary_keys) | ||
| sequence_fields = set(options.sequence_field()) |
There was a problem hiding this comment.
[P2] Exclude sequence-group ordering fields from effective aggregation
For partial-update, Java resolves field aggregators using allSequenceFields, including the ordering fields from fields..sequence-group. This helper only excludes the top-level sequence.field list. A valid table with PK=id, fields.seq.sequence-group=updates, fields.default-aggregate-function=merge_map_with_keytime and fields.updates.aggregate-function=last_non_null_value has no effective keytime aggregator: seq is an ordering field and updates overrides the default. The new read and mirrored write gates nevertheless classify seq as keytime and reject both public operations with NotImplementedError. Using the same real Rust 0.5 runtime and persisted files, bypassing only these newly added gates restores successful Native write/commit and the expected seq=2, updates=second read. Please include sequence-group ordering fields when resolving effective aggregators in both gates and add read/write regressions for this configuration.
| return_value=True), \ | ||
| patch( | ||
| 'pypaimon.execution.native.write.native_write.create_native_write_table' | ||
| ) as create_table: |
There was a problem hiding this comment.
[P2] Fix the indentation that blocks all Python CI test jobs
The closing parenthesis of this patch context triggers E123 with the repository's dev/cfg.ini. The changed-file flake8 command reproduces it locally, and CI43866912063 reports this exact line before any Python tests run in all six Native/Legacy Python jobs. Please align this closing bracket with its opening context and rerun the normal Python lint step so the regression tests can execute in CI.
|
@JingsongLi Thanks for the detailed reviews and the end-to-end reproductions. They have helped uncover several issues that were not covered by the existing tests. Would you be willing to actively contribute to the implementation of this PR and its Rust counterpart (apache/paimon-rust#1378), particularly around Native integration and cross-language compatibility? Some of the underlying Native/Rust execution paths are quite complex, and your experience would be extremely valuable. I'd really appreciate your help in working through these areas, including making code changes directly where appropriate. |
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
Rechecked unchanged 3d866cafc3: the existing Native MERGE source, sequence-group effective-aggregator and E123 lint findings remain. Normal Maven checks pass 198 Java aggregation/partial-update tests; 208 Python tests pass. Eight real Java Parquet DECIMAL cases pass across compact/wide decimals, both equal-timestamp policies, aggregation/partial-update, multiple commits, full compaction and reopening. The Java-written Unicode partial-update table is rejected explicitly before an unsafe Native read. Native0.5.0 MERGE and sequence-group probes still reproduce the two functional issues, and the exact-base MERGE control agrees across source forms. Project-configured changed-file flake8 still fails at native_merge_map_keytime_fallback_test.py:426, matching current CI.
One additional availability regression is reproduced below, with an exact-base control. No production sources were modified.
| PK aggregation or partial-update table. Non-PK / other engines / | ||
| other aggregators are unchanged and still clear this gate. | ||
| """ | ||
| return not _table_uses_merge_map_keytime(table) |
There was a problem hiding this comment.
[P2] Keep supported Variant projections eligible for Native reads
Rejecting every keytime table also disables Native-only operations that do not read the keytime field. I reproduced this with a PK aggregation table containing a BIGINT-keytime map and a VARIANT payload, written in two Parquet commits. A public projection of only id and variant_get(payload, '$.score', 'float') now raises Variant extraction requires a compatible native reader because Python cannot execute that fallback. It succeeds on exact merge-base ba9b1bc and, on this head, succeeds when only this new gate is bypassed, returning the expected latest score 9.5 in both controls using Rust0.5.0. The keytime map is not projected, and its numeric timestamp does not have the string-order mismatch. Please narrow the eligibility check to capabilities/types/fields requiring the fallback, or preserve an execution path for supported Native-only projections, while retaining the necessary STRING-ordering protection. Add this mixed-feature projection regression.
…time timestamp types
…lback Inline the STRING timestamp probe so NativeTableRead stays within the execution.native boundary and passes the backend isolation check. Co-authored-by: Cursor <[email protected]>
Route all merge_map_with_keytime reads through Python before Native opens (STRING UTF-16 vs Rust UTF-8; typed keytime never had a correct Native path). Drop the Python timestamp whitelist so historical BINARY/VARBINARY/BOOLEAN and other comparable types remain readable.
Add field-level Native read gating for STRING keytime aggregators. Keep Native reads available when projections and predicates do not access keytime fields. Materialize STRING keytime MERGE sources through the Python reader while preserving source eligibility and query authorization. Reject unsupported Partial Update keytime MERGE sources before backend dispatch. Keep public and Native self-merge handling consistent. Add regression tests for keytime fallback, VARIANT projections, MERGE source materialization, and query authorization. Co-authored-by: Cursor <[email protected]>
f0e5647 to
45ebabf
Compare
Purpose
This PR extends the supported timestamp field types in
merge_map_with_keytime.The following types are supported:
Character timestamp fields retain Java
String.compareToordering semantics. The Python implementation is also aligned with the Java behavior.Tests
git diff --check