feat: support case-insensitive column matching on reads (read-time parameter)#496
Conversation
JunRuiLee
force-pushed
the
feat/case-insensitive-upstream
branch
from
e671a08 to
9a2131c
Compare
JunRuiLee
changed the title
JunRuiLee
force-pushed
the
feat/case-insensitive-upstream
branch
2 times, most recently
from
df7d35c to
7ae85fc
Compare
JunRuiLee
changed the title
Add ReadBuilder::with_case_sensitive(bool) (default true) and PredicateBuilder::new_with_case_sensitive(fields, bool), mirroring Java's RowType.getFieldIndex(name, caseSensitive). When disabled, projection and predicate column names resolve by ASCII case-folding, error on ambiguity, and the canonical schema name is stored in the predicate leaf. Projection resolution is deferred to scan/read build time so with_projection and with_case_sensitive are order-independent; with_projection does best-effort early validation, rejecting only columns that cannot match under any case sensitivity. Default stays case-sensitive.
Derive case sensitivity from the session (!enable_ident_normalization) for scan and filter translation, and carry it into the physical scan so execute() resolves column names the same way planning did. classify_filter_pushdown (no Session available) classifies case-sensitively and refuses Exact for ASCII-case-colliding schemas so a needed residual filter is never dropped.
…itive Thread the read-time flag into projection and dict_to_predicate; add the type stub and document that it must precede with_filter for predicates.
Add paimon_read_builder_with_case_sensitive and a case_sensitive parameter on the paimon_predicate_* constructors (threaded into integer coercion and PredicateBuilder). Projection names are validated for obvious typos early and resolved lazily. Covers Doris and the Go binding.
JunRuiLee
force-pushed
the
feat/case-insensitive-upstream
branch
from
7ae85fc to
242ce95
Compare
…QL reads The read-time case-insensitive feature broke two consumers: - The C predicate constructors gained a `case_sensitive` parameter in place, but the Go binding's libffi call interfaces still pass the old argument counts, leaving `case_sensitive` undefined at the ABI boundary and corrupting every Go predicate call. Restore the existing `paimon_predicate_*` signatures (case-sensitive) and add `paimon_predicate_*_with_case_sensitive` variants for opt-in callers. Compile-time signature guards pin the existing ABI. - The DataFusion path derived case sensitivity from `enable_ident_normalization`, but DataFusion resolves columns against the provider schema before `scan` runs, so a case mismatch fails at planning and the scan-level folding is never reached. Always match case-sensitively on the SQL path and document why; case-insensitive matching stays a direct-ReadBuilder (core/C/Python) feature, covered by a new end-to-end negative test.
…sitive doc DataFusion classify_filter_pushdown always resolves case-sensitively, so an unrelated ASCII case-folding collision (e.g. Name/name) no longer downgrades an otherwise Exact filter on a different column. Remove the has_ascii_case_collision guard and retarget its test to assert Exact holds for a precise column match in a case-colliding schema. Clarify that paimon_read_builder_with_case_sensitive controls projection matching only; predicate case sensitivity is chosen by the paimon_predicate_* constructor variant, independent of this setting.
| F: Fn(&Predicate) -> bool, | ||
| { | ||
| let translator = FilterTranslator::new(fields); | ||
| // `FilterTranslator` still supports case-insensitive column resolution for |
There was a problem hiding this comment.
FilterTranslator is private to the DataFusion integration, so direct ReadBuilder callers never reach this case-insensitive branch. Every non-test call site passes true, leaving the false path test-only.
There was a problem hiding this comment.
True that every current call site passes true — but the case_sensitive parameter is intentional, not dead code. It is the explicit carrier of the "SQL path is always case-sensitive" contract: scan and the variant-extraction planner both pass true on purpose, and the end-to-end negative test (SELECT name vs a Name field fails at planning) pins that. The false path keeps FilterTranslator/classify consistent with the rest of the case-insensitivity machinery and is exercised by unit tests. I would rather keep the contract visible in the signature than hardcode true internally and lose that. Happy to revisit if you feel strongly.
There was a problem hiding this comment.
Understood. Since this is intentional and the production path is explicitly pinned to true, I don't feel strongly about it. Resolving.
resolve_projected_fields and validate_projection_possible scanned the full schema once per projected name (O(fields x projections)), regressing the default case-sensitive path from the previous hash-index lookup. Build the name index once: an exact-name map for the case-sensitive path and a folded-name map (folding collisions -> None) for the case-insensitive path, preserving ambiguity detection. Drop the now-unused find_projection_field. Also correct the C paimon_read_builder_with_projection doc: an obvious typo (no field matches under any case) is rejected by this call; only case-dependent resolution is deferred to paimon_read_builder_new_read.
QuakeWang
left a comment
QuakeWang
left a comment
There was a problem hiding this comment.
LGTM. Left a couple of non-blocking nits.
| ) -> DFResult<Vec<TableProviderFilterPushDown>> { | ||
| let fields = self.table.schema().fields(); | ||
| // SQL reads resolve columns case-sensitively (see `scan`), so classify | ||
| // pushdown the same way. `classify_filter_pushdown` still caps at |
There was a problem hiding this comment.
nit: This comment is stale after e205628c; classify_filter_pushdown no longer caps case-colliding schemas at Inexact.
There was a problem hiding this comment.
Fixed — dropped the stale sentence; the comment now just says SQL reads classify case-sensitively. (30e24ee)
| /// Set whether column-name matching (projection and predicate column | ||
| /// resolution) is case-sensitive. Defaults to `true` (exact match). When set | ||
| /// to `false`, names are matched by ASCII case-folding and an ambiguous | ||
| /// (case-colliding) request errors. Mirrors Java's |
There was a problem hiding this comment.
nit: Current Java Paimon RowType only exposes getFieldIndex(String fieldName); there is no getFieldIndex(name, caseSensitive) overload, so this reference is inaccurate.
There was a problem hiding this comment.
Good catch — Java RowType only has getFieldIndex(String). Removed the inaccurate overload reference; the doc now describes the per-read case sensitivity engines like Spark drive from spark.sql.caseSensitive, which is the actual analogue. (30e24ee)
supports_filters_pushdown no longer mentions the removed Inexact cap for case-colliding schemas (dropped in e205628). with_case_sensitive no longer cites a RowType.getFieldIndex(name, caseSensitive) overload that Java Paimon does not expose; describe the per-read case sensitivity engines drive instead.
3 participants
Purpose
Linked issue: close #495
Spark + Java Paimon support case-insensitive column matching on reads. In Java this lives at the engine / catalog layer — Spark's
spark.sql.caseSensitive(a session setting) drives Catalyst to resolve projection/filter columns case-insensitively and normalize them to the schema names before pushdown; Paimon core exposes it as a per-call parameter (RowType.getFieldIndex(name, caseSensitive)). It is not a table property.paimon-rusthad no equivalent — column resolution was always exact-case — so callers reading Paimon through the Rust core / C bindings / Python couldn't offer the behavior users get from Spark.This PR adds case-insensitive column matching as a read-time parameter (mirroring Java's per-call
caseSensitive), not a table option. Default stays case-sensitive, so existing behavior is unchanged; a reader opts in per read. When enabled, column names resolve by ASCII case-folding and an ambiguous reference (schema fields colliding under folding) is reported, mirroring Spark'sAMBIGUOUSbehavior.Scope of the opt-in
Case-insensitive matching is offered on the direct ReadBuilder API paths: core, the C binding, and Python. It is not offered through DataFusion / SQL: DataFusion resolves projection/filter columns against the provider schema during logical planning, before
TableProvider::scanruns, andenable_ident_normalizationonly lowercases unquoted identifiers — it does not make schema resolution case-insensitive. A genuine case mismatch (SELECT nameagainst aNamefield) therefore fails at planning and never reaches Paimon's resolution. So the DataFusion path always matches case-sensitively; a new end-to-end negative test pins this.Brief change log
ReadBuilder::with_case_sensitive(bool)(defaulttrue) andPredicateBuilder::new_with_case_sensitive(fields, bool)(the existingnewdelegates withtrue). When disabled, column resolution folds case, errors on ambiguity, and stores the canonical schema name in the predicate leaf so downstream name-based lookups (e.g. index pruning) keep matching. Projection duplicate detection folds case so a case-colliding projection is reported as a duplicate.with_projectionandwith_case_sensitivemay be called in any order.with_projectiondoes best-effort early validation, rejecting only columns that cannot match under any case sensitivity (an obvious typo); case-dependent outcomes surface fromnew_read.supports_filters_pushdownclassifies case-sensitively and still refuses to reportExactfor a schema with ASCII-case-folding collisions, as a conservative guard so a needed residual filter is never dropped.PyReadBuilder.with_case_sensitive(bool)threads the flag into projection and predicate conversion (type stub added).paimon_read_builder_with_case_sensitive(rb, bool)(new symbol), plus additivepaimon_predicate_*_with_case_sensitivevariants of each predicate constructor. The existingpaimon_predicate_*symbols keep their original signatures and case-sensitive behavior, so the current C ABI and its consumers (including the in-repo Go binding, which prepares fixed libffi call interfaces per symbol) are unaffected. Compile-time signature guards pin the existing ABI. (Go's own opt-in case-insensitive API is left to a follow-up.)ASCII-only folding throughout (
eq_ignore_ascii_case); no Unicode case folding.Scope: read path, top-level columns only. Nested/dotted paths, the
_ROW_IDsystem column (kept an exact reserved name), the write path, and the DataFusion/SQL path are out of scope.Tests
cargo test -p paimon --lib— projection/predicate case-insensitive match, default-sensitive rejection, ambiguity error, canonical-name storage, duplicate detection, order-independence (projection ↔ case-sensitive either order), and deferral of case-dependent errors tonew_read.cargo test -p paimon-datafusion—filter_pushdowntranslation/classify guard, and an end-to-end SQL negative test asserting thatSELECT nameagainst aNamefield fails at planning (documenting why the SQL path stays case-sensitive).PYO3_PYTHON=python3.12).API and Format
ReadBuilder::with_case_sensitive,PredicateBuilder::new_with_case_sensitive; Cpaimon_read_builder_with_case_sensitiveand additivepaimon_predicate_*_with_case_sensitivevariants. ExistingPredicateBuilder::newand the existingpaimon_predicate_*C symbols are unchanged (case-sensitive), preserving ABI compatibility.Documentation
Behavior described above and in the linked issue; the switch is a read-time API parameter, controlled by the reader/engine (not stored on the table), and is order-independent with projection. Case-insensitive matching is available via the direct ReadBuilder API (core / C / Python); the DataFusion/SQL path stays case-sensitive.