Skip to content

feat: support existing changelog incremental reads#509

Open
Pandas886 wants to merge 2 commits into
apache:mainfrom
Pandas886:feat/batch-incremental-changelog
Open

feat: support existing changelog incremental reads#509
Pandas886 wants to merge 2 commits into
apache:mainfrom
Pandas886:feat/batch-incremental-changelog

Conversation

@Pandas886

@Pandas886 Pandas886 commented Jul 11, 2026

Copy link
Copy Markdown

Purpose

Linked issue: close #504

Depends on: #508 (Delta/Auto). Please review that one first — this branch is stacked on top of it, so the diff against main currently includes both commits.

This adds reading of already-written changelog files (changelog_manifest_list) plus an AuditLogTable view that exposes rowkind as the first column.

Important: this does not generate lookup/compact changelogs. If a snapshot has no changelog list, we just skip it / return empty for that snapshot rather than inventing files.

Brief change log

  • Implement Changelog planning over (start, end]
  • TableScan::plan_snapshot_changelog reusing the existing split pipeline
  • AuditLogTable wrapper: rowkind first, optional _SEQUENCE_NUMBER
  • TableRead::to_audit_log_arrow for Delta (all +I) and Changelog (from _VALUE_KIND)
  • Auto → Changelog when producer is not none
  • Tests with changelog-producer=input fixtures and multi-snapshot ranges

Diff mode is still unsupported here on purpose.

Tests

cargo test -p paimon --test incremental_batch_scan_test
cargo test -p paimon --test audit_log_table_test
cargo fmt --all -- --check
cargo clippy -p paimon --all-targets --features fulltext,vortex -- -D warnings

API and Format

  • New AuditLogTable + Changelog path on the incremental API
  • No storage format change; only reads existing changelog manifests/files

Documentation

No standalone docs in this PR (SQL docs come with the DataFusion follow-up).

Review tip

If you're looking at the full branch vs main, the interesting commit for this PR is the top one (feat: support existing changelog incremental reads). The parent is #508.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch incremental scan: read existing changelog manifests + AuditLogTable

1 participant