Skip to content

[python] Introduce 'nested_update' aggregator function#8577

Open
PyRSA wants to merge 5 commits into
apache:masterfrom
PyRSA:feature/pypaimon-nested-update-impl
Open

[python] Introduce 'nested_update' aggregator function#8577
PyRSA wants to merge 5 commits into
apache:masterfrom
PyRSA:feature/pypaimon-nested-update-impl

Conversation

@PyRSA

@PyRSA PyRSA commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

Introduce the nested_update field aggregator for pypaimon.

This change adds support for the nested_update aggregation function in the Python reader, aligning its behavior with the Java implementation. It supports nested-key based updates, optional sequence-field based conflict resolution, configurable nested-key null handling, and count limiting.

Dependency

This PR depends on #8561 and reuses several shared utility methods introduced there. Please review and merge #8561 first.

Tests

  • python -m pytest pypaimon/tests/test_field_aggregators.py::FieldNestedUpdateAggTest -q
  • git diff --check

@JingsongLi

Copy link
Copy Markdown
Contributor
  • RowType.get_field_index() returns DataField.id, but the ProjectedRow of nested key/sequence uses it as an array index.
  • The actual nested field ID is a global ID and does not guarantee continuity from 0.
  • Reproducing locally using field IDs 10, 20: Both generate mapping=[10], then directly throw an IndexError: Position 10 is out of bounds for row arity 2.
  • The existing tests use 0, 1, 2, which obscures the issue. The ordinal field should be returned, and non-sequential ID tests should be added.

@PyRSA

PyRSA commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author
  • RowType.get_field_index() returns DataField.id, but the ProjectedRow of nested key/sequence uses it as an array index.
  • The actual nested field ID is a global ID and does not guarantee continuity from 0.
  • Reproducing locally using field IDs 10, 20: Both generate mapping=[10], then directly throw an IndexError: Position 10 is out of bounds for row arity 2.
  • The existing tests use 0, 1, 2, which obscures the issue. The ordinal field should be returned, and non-sequential ID tests should be added.

Good catch, thanks! I've fixed RowType.get_field_index() to return the ordinal field position instead of DataField.id, and added a regression test using non-sequential field IDs to cover this case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants