Skip to content

feat: support rowkind.field write semantics#507

Open
Pandas886 wants to merge 1 commit into
apache:mainfrom
Pandas886:feat/rowkind-field
Open

feat: support rowkind.field write semantics#507
Pandas886 wants to merge 1 commit into
apache:mainfrom
Pandas886:feat/rowkind-field

Conversation

@Pandas886

Copy link
Copy Markdown

Purpose

Linked issue: close #502

I ran into the current fail-loud path when trying to write CDC-style rows with a user op column (rowkind.field). Java already supports this, and the Rust option is parsed but writing still rejects it. This PR wires up the write path for deduplicate PK tables.

Brief change log

  • Add short-string RowKind parsing (+I / -U / +U / -D, case-insensitive)
  • Introduce RowKindGenerator to map a STRING column onto internal _VALUE_KIND
  • Schema checks: PK table, deduplicate only, field exists and is unbounded STRING, reject bad combos
  • TableWrite enrich step: convert row kinds, optionally drop retracts via ignore-delete / ignore-update-before
  • Reject batches that already contain _VALUE_KIND when rowkind.field is set (fail loud, no silent override)
  • Integration tests for the main happy paths + illegal tokens / conflict cases

I deliberately left other merge engines alone — they still get a clear validation error if someone sets rowkind.field there.

Tests

Locally:

cargo test -p paimon --test rowkind_field_test
cargo test -p paimon row_kind
cargo fmt --all -- --check
cargo clippy -p paimon --all-targets --features fulltext,vortex -- -D warnings

All green on my machine.

API and Format

  • Public API: short-string helpers on RowKind, plus the existing rowkind.field option now actually works on write
  • Storage format: unchanged (still writes normal PK files with _VALUE_KIND)

Documentation

No user-facing docs update in this PR. Happy to add a short note to the write docs if reviewers want it.

Notes

This is independent of the incremental-read PR stack I also opened. Review can land in either order.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support rowkind.field for CDC-style writes on deduplicate PK tables

1 participant