[SPARK-59176][SQL][4.2] Fix a storage-partitioned join that fails when one side reduced onto no key

[peter-toth]

What changes were proposed in this pull request?

EnsureRequirements leaves a side out of the comparison of the two sides' reduced key types when that side has no partition key and its expressions no longer describe the keys it would have had. The types then come from a side that does answer for them.

KeyedPartitioning.keyDataTypes reports the types the partition key rows were built with. With no key row to read, it falls back to the partition expressions' own types. That is still the right answer while the expressions describe the keys, and a join that reduced both sides' keys leaves expressions that do not (TransformExpression.reducedWith, SPARK-59121). Only then is the fallback a type no key of that partitioning would hold, and only then must a caller keep it out of a comparison against a real answer.

An empty side that nothing reduced stays in the comparison, which is what keeps the comparison doing its other job. Where one side has a reducer, it holds the connector's Reducer.resultType() against the paired transform, and that needs no key row.

The keyDataTypes scaladoc states the rule the fix follows, in place of the paragraph that described the failure and pointed here.

Why are the changes needed?

A storage-partitioned join whose two legs each reduced both of their sides onto one key space is co-partitioned, and joins without a shuffle. If a leg ends up with no partition key at all, the query fails instead. v2BucketingPartitionFilterEnabled produces such a leg whenever its two sides hold disjoint keys, i.e. whenever that leg is empty.

SELECT coalesce(l.ts, r.ts) FROM
  (SELECT d.ts FROM days1 d JOIN years1 y ON y.ts = d.ts) l
  JOIN
  (SELECT y.ts FROM days2 d JOIN years2 y ON y.ts = d.ts) r
  ON l.ts = r.ts

With days2 and years2 holding disjoint years, and days and years reducing onto a common LongType year key:

[STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES] Storage-partition join partition
transforms produced incompatible reduced types, left reducers: [] returned: ["BIGINT"],
right reducers: [] returned: ["INT"]. SQLSTATE: 42K09

Both reducer lists are empty, which is the sign that there was nothing left to reduce and nothing to compare. INT is the years transform's own result type, not a type any key row holds.

Does this PR introduce any user-facing change?

Yes. The query above returns its result instead of failing. Only unreleased versions are affected: the failure is reachable through SPARK-59121, and before that the same shape failed on a ClassCastException from applying the reduce a second time.

How was this patch tested?

Two new KeyGroupedPartitioningSuite tests.

The first covers the shape above in both join orders, since the side to leave out can be either one, and with both an inner and a full outer join. The inner join intersects the two key sets to nothing and so has nothing to sort, while the full outer join keeps the other side's keys and sorts them by the reported types, which is what makes those types matter. Each part of the fix fails this test on its own when disabled.

The second covers an empty side that is not marked, to pin that it stays in the comparison. A one-side days -> years reduce whose years side is emptied by an upstream inner join under the partition filter, against a reducer returning DateType where the target transform is IntegerType, still raises STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

Backport to branch-4.2

A cherry-pick of #58486's two commits, squashed, with one line tailored: the config is named V2_BUCKETING_ALLOW_JOIN_KEYS_SUBSET_OF_PARTITION_KEYS on this branch, not V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS. Nothing else differs from the branch-4.3 backport.

Measured on the branch tip: SPARK-59176: a leg reduced onto no key at all still joins fails there with STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES, the same shape as on master, so the branch is affected. The precondition is present: SPARK-59121 reached branch-4.2 as #58482.

172 tests green across KeyGroupedPartitioningSuite, GroupPartitionsExecSuite and EnsureRequirementsSuite, plus 12 in ShuffleSpecSuite. dev/lint-scala clean.

[dongjoon-hyun]

+1, LGTM (Pending CIs).

[uros-b]

+1, thank you @peter-toth and @dongjoon-hyun!

[dongjoon-hyun]

Merge Summary:

• merged into branch-4.2 9dff80a

Posted by merge_spark_pr.py