Skip to content

Extend expression planners for Snowflake compatibility - #47

Merged
osipovartem merged 30 commits into
embucket-sync-df55.0.0from
expr-planner-aggregate-schema
Sep 3, 2026
Merged

Extend expression planners for Snowflake compatibility#47
osipovartem merged 30 commits into
embucket-sync-df55.0.0from
expr-planner-aggregate-schema

Conversation

@osipovartem

@osipovartem osipovartem commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add backward-compatible schema-aware hooks for aggregate and scalar expression planners
  • support qualified wildcard arguments and wildcard options such as alias.* EXCLUDE (...)
  • add opt-in Snowflake-compatible decimal literal normalization
  • fall back to Float64 when a fractional/scientific literal exceeds the Decimal256 range
  • preserve wildcard join keys, INSERT SELECT source schemas, volatile expressions, and zero-column sort rows
  • update h2 to 0.4.16 for RUSTSEC-2026-0258

This PR synchronizes the complete DF55 integration stack used by Rustice into the canonical embucket-sync-df55.0.0 branch. It contains the changes previously reviewed in #48 through #55.

Compatibility

The planner hooks and decimal normalization are opt-in, so default DataFusion parsing and expression planning remain unchanged. The remaining changes fix concrete planner/execution correctness cases exercised by Rustice.

With decimal parsing and trailing-zero normalization enabled:

  • 10.00 plans as DECIMAL(2, 0)
  • 10.10 plans as DECIMAL(3, 1)
  • 0.00100 plans as DECIMAL(4, 3)
  • 1.2300e2 plans as DECIMAL(3, 0)
  • scientific/fractional values outside Decimal256 range fall back to Float64; oversized integer literals remain errors

Validation

  • DataFusion SQL planner unit and integration tests for the new hooks
  • qualified wildcard and USING join tests
  • decimal literal and out-of-range fallback tests
  • focused options.slt run after refreshing the three fallback expectations
  • cargo fmt --all -- --check
  • CI matrix rerun by this update

@osipovartem osipovartem changed the title Expose input schema to aggregate expression planners Extend expression planners for Snowflake compatibility Sep 1, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation core common sqllogictest labels Sep 2, 2026
@Vedin

Vedin commented Sep 2, 2026

Copy link
Copy Markdown

ack

Preserve rows when sorting zero-column batches
Apply PIVOT aliases before schema validation
@osipovartem

Copy link
Copy Markdown
Collaborator Author

All required checks are green, including the full Rust and extended-test matrices. The focused decimal division, AVG rounding, trailing-zero normalization, and out-of-range literal tests also pass locally.

@Vedin Could you submit the acknowledgement as an approving GitHub review? Branch protection still reports REVIEW_REQUIRED, so the PR cannot be merged normally yet.

@osipovartem
osipovartem merged commit 5d91b55 into embucket-sync-df55.0.0 Sep 3, 2026
71 checks passed
@osipovartem
osipovartem deleted the expr-planner-aggregate-schema branch September 3, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants