Skip to content

[SPARK-59185][SQL] Derive a StartsWith prefix filter from leading-literal LIKE patterns - #58484

Open
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:like-prefix-startswith-pushdown
Open

[SPARK-59185][SQL] Derive a StartsWith prefix filter from leading-literal LIKE patterns#58484
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:like-prefix-startswith-pushdown

Conversation

@david-mollitor-db

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

LikeSimplification rewrites simple LIKE patterns into cheaper predicates ('A%' ->
StartsWith, '%B' -> EndsWith, 'A%B' -> length guard + StartsWith + EndsWith,
'%B%' -> Contains, exact string -> EqualTo). Multi-wildcard patterns that have a
leading literal but match none of those shapes -- e.g. 'A%B%', 'AB%CD%EF', 'A_B%' --
fall through unchanged and remain a full regex Like, so the data source receives no
predicate and the per-row regex runs on every row.

This PR makes LikeSimplification additionally derive the leading literal A as the
necessary condition StartsWith(col, A), keeping the original LIKE as the exact residual:

col LIKE 'A%B%'  ==>  StartsWith(col, A) && (col LIKE 'A%B%')

StartsWith is placed first so the cheap check short-circuits the regex. The derivation is
restricted to binary-equality collations (StringType.supportsBinaryEquality, i.e.
UTF8_BINARY) and a TreeNodeTag on the residual Like keeps the rule idempotent under the
fixed-point optimizer batch. Only StartsWith is derived from the leading literal; the
LikeAll/LikeAny paths are unchanged.

Why are the changes needed?

  • Pushdown / pruning. On UTF8_BINARY, StartsWith translates to
    sources.StringStartsWith, which prunes Parquet row groups via min/max statistics. Readers
    cannot prune on the raw Like, so today a leading-literal multi-wildcard LIKE reads every
    row group.
  • Cheaper per-row evaluation. The StartsWith short-circuits the more expensive regex on
    rows that fail the prefix.
  • Results are unchanged. StartsWith(A) is implied by LIKE 'A%...', and the exact LIKE
    is retained as the residual, so the conjunction accepts exactly the same rows. This mirrors
    how PostgreSQL, SQL Server and SQLite turn a leading-literal LIKE into a sargable prefix
    predicate plus a residual recheck.

The derivation is gated on binary equality for correctness as well as benefit: under a
collation-aware collation (e.g. UTF8_LCASE) the Like regex match (Java regex case flags)
and StartsWith (CollationSupport) can disagree, so StartsWith(A) would not be a sound
necessary condition; and StringStartsWith is pushed down only for UTF8_BINARY (non-binary
is wrapped as CollatedStringStartsWith, which readers ignore). EndsWith/Contains are not
derived from trailing/inner literals because Parquet's StringEndsWith/StringContains have
canDrop = false (no pruning).

Does this PR introduce any user-facing change?

No. Query results are identical; this is a performance improvement (added pushdown and a
short-circuit conjunct on an otherwise unsimplified LIKE).

How was this patch tested?

  • New unit tests in LikeSimplificationSuite covering: derivation for 'a%b%', a multi-char
    prefix with multiple wildcards, '_' patterns, no derivation when there is no leading literal,
    no derivation when the pattern contains the escape char, no derivation for a non-binary
    (UTF8_LCASE) collation, and idempotency.
  • A new end-to-end test in ParquetFilterSuite verifying that LIKE 'ab%cd%' / 'ab%cd%ef' /
    'a_b%' push a StringStartsWith and prune Parquet row groups.
  • build/sbt 'catalyst/testOnly *LikeSimplificationSuite' and
    build/sbt 'sql/testOnly *ParquetV1FilterSuite -- -z "leading-literal"' pass; scalastyle clean.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8

…eral LIKE patterns

`LikeSimplification` rewrites simple `LIKE` patterns into cheaper predicates
(`'A%'` -> `StartsWith`, `'%B'` -> `EndsWith`, `'A%B'` -> length guard + `StartsWith` +
`EndsWith`, `'%B%'` -> `Contains`, exact -> `EqualTo`). Multi-wildcard patterns with a
leading literal that match none of those shapes -- e.g. `'A%B%'`, `'AB%CD%EF'`, `'A_B%'`
-- fall through unchanged and stay a full regex `Like`, so the data source receives no
predicate and the per-row regex runs on every row.

This derives the leading literal `A` as the necessary condition `StartsWith(col, A)` and
keeps the original `LIKE` as the exact residual:

    col LIKE 'A%B%'  ==>  StartsWith(col, A) && (col LIKE 'A%B%')

`StartsWith` is placed first so the cheap check short-circuits the regex, and it is a
predicate the existing pushdown path understands: on UTF8_BINARY it translates to
`StringStartsWith`, which prunes Parquet row groups via min/max. Results are unchanged --
`StartsWith(A)` is implied by `LIKE 'A%...'` and the exact `LIKE` is retained as the residual.

The derivation is restricted to binary-equality collations (`supportsBinaryEquality`):
under a collation-aware collation the `Like` regex match (Java regex case flags) and
`StartsWith` (`CollationSupport`) can disagree, so `StartsWith(A)` would not be a sound
necessary condition; and `StringStartsWith` only pushes down for UTF8_BINARY. Only
`StartsWith` is derived from the leading literal -- Parquet's `StringEndsWith` and
`StringContains` do not prune -- and the `LikeAll`/`LikeAny` paths are unchanged. A
`TreeNodeTag` on the residual `Like` keeps the rule idempotent under the fixed-point batch.

Generated-by: Claude Opus 4.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant