Skip to content

[spark] Reject table constraints in column-list REPLACE TABLE on Spark 4.1+ - #11267

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:replace-table-constraints
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:replace-table-constraints

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

Since Spark 4.1, Spark's own REPLACE TABLE planning rejects table constraints when the catalog does not declare SUPPORT_TABLE_CONSTRAINT. Paimon plans a column-list REPLACE TABLE with its own PaimonReplaceTableStrategy, which skipped that check. Paimon's catalogs do not declare the capability, so CREATE TABLE t (id INT, CONSTRAINT positive CHECK (id > 0)) fails, but the same statement as CREATE OR REPLACE TABLE replaced an existing t and dropped the constraint.

The strategy now calls Spark's ResolveTableConstraints.validateCatalogForTableConstraint before planning, through a new SparkShim.validateTableConstraints. The method does nothing by default, since Spark 3.x and 4.0 have no TableSpec.constraints. paimon-spark4-common overrides it, which covers the 4.1 and 4.2 connectors. RTAS is left alone: Spark's own RTAS path has no such check, and since 4.1.2 the parser rejects constraints with AS SELECT.

This came up in the review of #11246, which adds a Spark 4.3 connector on the same strategy.

Tests

DDLTestBase (run on Spark 4.1) and Spark42CompatibilityTest each get a case where CREATE OR REPLACE TABLE with a CHECK constraint must fail with UNSUPPORTED_FEATURE.TABLE_OPERATION and leave the old row in place.

@LuciferYang

Copy link
Copy Markdown
Contributor Author

cc @JingsongLi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant