-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
- I searched in the issues and found nothing similar.
Paimon version
1.20.0
Compute Engine
Flink 1.20.0
Minimal reproduce step
- Set up Flink CDC pipeline with MySQL source and Paimon sink
- Source table has primary key: client_id
- Configure Paimon sink with partition key: client_id
- Start the pipeline
- Pipeline fails when creating table with error:
"Primary key constraint [client_id] should not be same with partition fields [client_id]"
What doesn't meet your expectations?
PaimonMetadataApplier.applyCreateTable() (lines 199-203) automatically adds
partition keys to primary keys without checking if this violates Paimon's own
constraint.
Expected: Should allow partitioning by primary key, or at minimum not modify
user's schema in a way that causes validation failure.
Actual: Code forces partition keys into primary keys, then Paimon's
TableSchema.trimmedPrimaryKeys() rejects it because trimmed primary keys
would be empty.
This makes it impossible to partition by the same field as primary key in
CDC scenarios.
Anything else?
Root cause code in PaimonMetadataApplier.java lines 199-203:
for (String partitionColumn : partitionKeys) {
if (!primaryKeys.contains(partitionColumn)) {
primaryKeys.add(partitionColumn);
}
}
This logic incorrectly assumes partition keys must always be in primary keys.
Issues:
- Breaks when partition key equals primary key (current bug)
- Prevents append-only tables (no primary key) from having partitions
- Modifies user's explicit schema definition without consent
Suggested fix:
- Remove automatic modification, OR
- Only apply for bucketed tables with composite primary keys, OR
- Skip this logic when primaryKeys is empty (append-only tables)
Append-only tables SHOULD support partitioning without requiring primary keys.
Workaround: Currently must use different fields for partition and primary key, or avoid partitioning in append-only tables.
Are you willing to submit a PR?
- I'm willing to submit a PR!