[SPARK-59176][SQL] Fix a storage-partitioned join that fails when one side reduced onto no key by peter-toth · Pull Request #58486 · apache/spark

peter-toth · 2026-09-02T20:24:13Z

What changes were proposed in this pull request?

EnsureRequirements leaves a side out of the comparison of the two sides' reduced key types when that side has no partition key and its expressions no longer describe the keys it would have had. The types then come from a side that does answer for them.

KeyedPartitioning.keyDataTypes reports the types the partition key rows were built with. With no key row to read, it falls back to the partition expressions' own types. That is still the right answer while the expressions describe the keys, and a join that reduced both sides' keys leaves expressions that do not (TransformExpression.reducedWith, SPARK-59121). Only then is the fallback a type no key of that partitioning would hold, and only then must a caller keep it out of a comparison against a real answer.

An empty side that nothing reduced stays in the comparison, which is what keeps the comparison doing its other job. Where one side has a reducer, it holds the connector's Reducer.resultType() against the paired transform, and that needs no key row.

The keyDataTypes scaladoc states the rule the fix follows, in place of the paragraph that described the failure and pointed here.

Why are the changes needed?

A storage-partitioned join whose two legs each reduced both of their sides onto one key space is co-partitioned, and joins without a shuffle. If a leg ends up with no partition key at all, the query fails instead. v2BucketingPartitionFilterEnabled produces such a leg whenever its two sides hold disjoint keys, i.e. whenever that leg is empty.

SELECT coalesce(l.ts, r.ts) FROM
  (SELECT d.ts FROM days1 d JOIN years1 y ON y.ts = d.ts) l
  JOIN
  (SELECT y.ts FROM days2 d JOIN years2 y ON y.ts = d.ts) r
  ON l.ts = r.ts

With days2 and years2 holding disjoint years, and days and years reducing onto a common LongType year key:

[STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES] Storage-partition join partition
transforms produced incompatible reduced types, left reducers: [] returned: ["BIGINT"],
right reducers: [] returned: ["INT"]. SQLSTATE: 42K09

Both reducer lists are empty, which is the sign that there was nothing left to reduce and nothing to compare. INT is the years transform's own result type, not a type any key row holds.

Does this PR introduce any user-facing change?

Yes. The query above returns its result instead of failing. Only unreleased versions are affected: the failure is reachable through SPARK-59121, and before that the same shape failed on a ClassCastException from applying the reduce a second time.

How was this patch tested?

Two new KeyGroupedPartitioningSuite tests.

The first covers the shape above in both join orders, since the side to leave out can be either one, and with both an inner and a full outer join. The inner join intersects the two key sets to nothing and so has nothing to sort, while the full outer join keeps the other side's keys and sorts them by the reported types, which is what makes those types matter. Each part of the fix fails this test on its own when disabled.

The second covers an empty side that is not marked, to pin that it stays in the comparison. A one-side days -> years reduce whose years side is emptied by an upstream inner join under the partition filter, against a reducer returning DateType where the target transform is IntegerType, still raises STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

… side reduced onto no key ### What changes were proposed in this pull request? `EnsureRequirements` compares the two sides' reduced key types only when both sides have partition keys, and takes the types from the side that has them. `KeyedPartitioning.keyDataTypes` reports the types the partition key rows were built with. With no key row to read, it falls back to the partition expressions' own types. Those types describe the keys only while the expressions do, and a join that reduced both sides' keys leaves expressions that do not (`TransformExpression.reducedWith`, SPARK-59121). The fallback is then not an answer about the keys, and holding it against the other side's real answer fails the query. The `keyDataTypes` scaladoc states the rule the fix follows, in place of the paragraph that described the failure and pointed here. One check is given up along with the untruthful comparison. Where a side has a reducer, the comparison also holds the connector's `Reducer.resultType()` against the paired transform, which needs no key row. That diagnostic is now skipped whenever a filter empties that side. Only a connector that already breaks `r(f1(x)) = f2(x)` on types is affected, and it loses a message it used to get. In one shape it loses more than the message: with `v2BucketingShuffleEnabled` on, a third child shuffled onto the merged keys evaluates the target transform per row, so the mismatch surfaces as a `ClassCastException` instead. The alternative is to skip only an already-reduced pair, which would keep the check for a side that is empty but not marked. That was not taken. The fallback is untruthful for a `KeyedShuffleSpec.createPartitioning` result as well, so a narrower guard trades a lost message for a query that fails while it is correct, which is the worse of the two and is the bug being fixed here. ### Why are the changes needed? A storage-partitioned join whose two legs each reduced both of their sides onto one key space is co-partitioned, and joins without a shuffle. If a leg ends up with no partition key at all, the query fails instead. `v2BucketingPartitionFilterEnabled` produces such a leg whenever its two sides hold disjoint keys, i.e. whenever that leg is empty. SELECT coalesce(l.ts, r.ts) FROM (SELECT d.ts FROM days1 d JOIN years1 y ON y.ts = d.ts) l JOIN (SELECT y.ts FROM days2 d JOIN years2 y ON y.ts = d.ts) r ON l.ts = r.ts With `days2` and `years2` holding disjoint years, and `days` and `years` reducing onto a common `LongType` year key: [STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES] Storage-partition join partition transforms produced incompatible reduced types, left reducers: [] returned: ["BIGINT"], right reducers: [] returned: ["INT"]. SQLSTATE: 42K09 Both reducer lists are empty, which is the sign that there was nothing left to reduce and nothing to compare. `INT` is the `years` transform's own result type, not a type any key row holds. ### Does this PR introduce _any_ user-facing change? Yes. The query above returns its result instead of failing. Only unreleased versions are affected: the failure is reachable through SPARK-59121, and before that the same shape failed on a `ClassCastException` from applying the reduce a second time. ### How was this patch tested? A new `KeyGroupedPartitioningSuite` test covers the shape in both join orders, since the side to leave out can be either one, and with both an inner and a full outer join. The inner join intersects the two key sets to nothing and never sorts them, so the full outer join is what makes the reported types matter. Each of the three parts of the fix fails the test on its own when disabled. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5)

peter-toth · 2026-09-02T20:33:17Z

cc @dongjoon-hyun

szehon-ho

makes sense to me!

dongjoon-hyun

Reviewed with the empty-side guard as the focus. One substantive question on how wide the skip is, plus a few nits.

dongjoon-hyun · 2026-09-03T03:22:37Z

+        // against the paired transform, and that check is given up here. An empty side has no
+        // row to misread, so a connector that breaks the contract loses a message rather than
+        // correctness.
+        if (leftReducedKeys.nonEmpty && rightReducedKeys.nonEmpty &&


The guard skips the comparison for any side with no keys, but the keyDataTypes fallback is only untruthful for a marked partitioning (!expressionsDescribeKeys). For an unmarked empty side the fallback is the type its keys would have had, so the old comparison there was a valid Reducer.resultType() check, and this drops it.

Shape: a one-side reduce days(ts) -> years(ts) where the years side is a GroupPartitionsExec emptied by an upstream inner join under the partition filter (unmarked, no keys), and a connector whose reducer returns LongType against an IntegerType target. Before: STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES. After: the check is skipped, and with LEFT OUTER the days side reports an unmarked years(ts) (IntegerType) over LongType key rows with expressionsDescribeKeys == true, so canCreatePartitioning accepts it. With v2BucketingShuffleEnabled, ShuffleExchangeExec re-wraps those rows with expressionDataTypes; InterpretedHashFunction hashes on the runtime value, so the stored Long keys and the looked-up Int keys hash differently and KeyGroupedPartitioner.getPartition falls back to nonNegativeMod. That is misrouted rows rather than a lost message, so the comment above ("loses a message rather than correctness", "no row to misread") understates what the PR description already concedes.

Would keys.isEmpty && !partitioning.expressionsDescribeKeys work instead? It still fixes the test here (both legs are marked) and keeps the check. The createPartitioning counter-case in the description needs a 0-partition spec winning bestSpec plus a struct key differing only in field names, which seems far rarer than a contract-breaking reducer and was equally broken before. If the wide skip stays, a SPARK-59187 cross-reference here would mark it as interim.

You are right, and I measured your shape. Written as a test, it returns empty with the guard as it was, where it used to raise STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES.

The guard is now your condition, keys.nonEmpty || expressionsDescribeKeys per side. An empty side that nothing reduced reports the types its keys would have held, so it answers for them and stays in the comparison. The createPartitioning case I used to argue for the wider skip is, as you say, both rarer and equally broken before this PR, so it does not pay for a lost reducer check.

I did not verify the misrouting chain itself, since the fix is the same whichever way that shape ends. What I did verify is that the check disappears, which is enough to reject the wider guard. Neither the comment nor the description claims anything is given up now.

Fixed in 979f420

dongjoon-hyun · 2026-09-03T03:22:37Z

@@ -612,17 +612,18 @@ case class KeyedPartitioning(
   *
   * The two cases can meet, and then the fallback is not truthful. A marked partitioning can end up


Two sentences outside this hunk now hold only when both sides have keys: lines 606-608 ("A one-side reduce keeps them equal, because ... EnsureRequirements refuses a reducer whose result type disagrees with it") and the reducersBothWays doc at 1696-1698 ("A connector that violates the contract ... fails the reduced-types check in EnsureRequirements"). canCreatePartitioning and ShuffleExchangeExec rely on that invariant through expressionsDescribeKeys. Either narrow the guard so they stay true, or qualify both.

With the narrower guard both sentences hold again. A one-side reduce with an empty unmarked target is still compared, so keyDataTypes' line about EnsureRequirements refusing a disagreeing reducer is true, and so is the reducersBothWays line. Both stay as they are.

The paragraph you commented on now says which side is left out, since that is the part that changed.

Fixed in 979f420

dongjoon-hyun · 2026-09-03T03:22:37Z

    }
  }

+  test("SPARK-59176: a leg reduced onto no key at all still joins") {


This covers the both-sides-marked shape only. The comment in EnsureRequirements says the skip is wider than that, but nothing pins the one-side reduce with an empty unmarked target. For example, the SPARK-56046 tables with purchases(years(time)) joined to a third years(time) table holding disjoint years under the partition filter, then items(days(arrive_time)) on top. That threw before this PR and returns empty now; a test asserting whichever is intended would keep the guard from drifting.

Added, close to your shape: items(days(arrive_time)) over a years(time) leg that an upstream inner join empties under the partition filter, with UnboundDaysFunctionWithToYearsReducerWithDateResult as the contract-breaking reducer. It asserts the error, so the intended answer is that it still throws.

That test is what measured the wider guard as wrong.

Fixed in 979f420

dongjoon-hyun · 2026-09-03T03:22:37Z

            rightReducers = rightReducers,
            rightReducedDataTypes = rightReducedDataTypes)
        }
+        val reducedDataTypes =


nit: the guard and this selection encode one rule in two statements that have to stay in sync. A behaviour-identical single form:

val reducedDataTypes = if (leftReducedKeys.isEmpty) { rightReducedDataTypes } else if (rightReducedKeys.isEmpty || leftReducedDataTypes == rightReducedDataTypes) { leftReducedDataTypes } else { throw QueryExecutionErrors.storagePartitionJoinIncompatibleReducedTypesError(...) }

Taken. With the narrower condition it reads:

val reducedDataTypes = if (!leftTypesDescribeKeys) { rightReducedDataTypes } else if (!rightTypesDescribeKeys || leftReducedDataTypes == rightReducedDataTypes) { leftReducedDataTypes } else { throw QueryExecutionErrors.storagePartitionJoinIncompatibleReducedTypesError(...) }

Fixed in 979f420

dongjoon-hyun · 2026-09-03T03:22:37Z

        SQLConf.V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS.key -> "true",
        SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true") {
-        checkAnswer(sql(reducedTsLegJoin), Seq(Row(Timestamp.valueOf("2021-01-03 00:00:00"))))
+        checkAnswer(sql(reducedTsLegJoin()), Seq(Row(Timestamp.valueOf("2021-01-03 00:00:00"))))


nit: this is bothTimestamps(1). Splitting into ts2020/ts2021 (bothTimestamps = Seq(ts2020, ts2021)) and using Seq(ts2021) here keeps it symmetric with row2021 on line 990.

Taken, bothTimestamps is Seq(ts2020, ts2021) now and this site uses Seq(ts2021).

Fixed in 979f420

dongjoon-hyun · 2026-09-03T03:22:37Z

+        SQLConf.V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS.key -> "true",
+        SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true") {
+        // Both orders, since the side that has no key is the one to leave out of the comparison.
+        // And both join types, since the inner join intersects the two key sets to nothing and


nit: mergeAndDedupPartitions sorts on every join type; the inner join sorts an empty sequence rather than never sorting. "has nothing to sort" would be accurate (the PR description says the same).

Right, it sorts an empty sequence. Reworded to "has nothing to sort".

Fixed in 979f420

dongjoon-hyun · 2026-09-03T03:22:37Z

@@ -612,17 +612,18 @@ case class KeyedPartitioning(
   *
   * The two cases can meet, and then the fallback is not truthful. A marked partitioning can end up
   * with no key, for instance when `v2BucketingPartitionFilterEnabled` intersects two sides that


nit: "which no key of that partitioning would have held" restates "not truthful" from the same sentence; the "no key row, no fact" reasoning and the caller rule that follow are what carry the paragraph.

Dropped.

Fixed in 979f420

Narrows the guard from "the side has no key" to "the side has no key and its expressions no longer describe them", i.e. `keys.nonEmpty || expressionsDescribeKeys`. An empty side that nothing reduced still reports the types its keys would have held, so it answers for them and stays in the comparison. That is what keeps the comparison checking a reducer's result type against the paired transform, which the wider guard dropped. Measured: with the wider guard a one-side `days` -> `years` reduce whose `years` side is emptied by an upstream inner join under the partition filter accepted a reducer returning `DateType` against an `IntegerType` target, where it used to raise `STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES`. A new test covers that shape and asserts the error. Folds the guard and the type selection into one expression, so the two cannot drift apart. The `keyDataTypes` scaladoc now says which side is left out. Its sentence about a one-side reduce, and the `reducersBothWays` sentence about a contract-violating reducer failing the reduced-types check, hold again with the narrower guard, so both stay as they are. Test fixtures: `bothTimestamps` is now `Seq(ts2020, ts2021)`, so a single expected row is `ts2021` rather than an index into the pair. One comment said the inner join never sorts, where it sorts an empty sequence.

uros-b · 2026-09-03T14:00:41Z

Thank you @peter-toth and @szehon-ho @dongjoon-hyun!

dongjoon-hyun

+1, LGTM. Thank you, @peter-toth and all.

peter-toth · 2026-09-03T16:23:46Z

Merge Summary:

merged into master 3c70da5
merged into branch-4.x 94d24b2

Posted by merge_spark_pr.py

peter-toth · 2026-09-03T16:25:43Z

Thank you @szehon-ho, @dongjoon-hyun, @uros-b for the review.

I will open backport PRs separately.

… side reduced onto no key ### What changes were proposed in this pull request? `EnsureRequirements` leaves a side out of the comparison of the two sides' reduced key types when that side has no partition key and its expressions no longer describe the keys it would have had. The types then come from a side that does answer for them. `KeyedPartitioning.keyDataTypes` reports the types the partition key rows were built with. With no key row to read, it falls back to the partition expressions' own types. That is still the right answer while the expressions describe the keys, and a join that reduced both sides' keys leaves expressions that do not (`TransformExpression.reducedWith`, SPARK-59121). Only then is the fallback a type no key of that partitioning would hold, and only then must a caller keep it out of a comparison against a real answer. An empty side that nothing reduced stays in the comparison, which is what keeps the comparison doing its other job. Where one side has a reducer, it holds the connector's `Reducer.resultType()` against the paired transform, and that needs no key row. The `keyDataTypes` scaladoc states the rule the fix follows, in place of the paragraph that described the failure and pointed here. ### Why are the changes needed? A storage-partitioned join whose two legs each reduced both of their sides onto one key space is co-partitioned, and joins without a shuffle. If a leg ends up with no partition key at all, the query fails instead. `v2BucketingPartitionFilterEnabled` produces such a leg whenever its two sides hold disjoint keys, i.e. whenever that leg is empty. SELECT coalesce(l.ts, r.ts) FROM (SELECT d.ts FROM days1 d JOIN years1 y ON y.ts = d.ts) l JOIN (SELECT y.ts FROM days2 d JOIN years2 y ON y.ts = d.ts) r ON l.ts = r.ts With `days2` and `years2` holding disjoint years, and `days` and `years` reducing onto a common `LongType` year key: [STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES] Storage-partition join partition transforms produced incompatible reduced types, left reducers: [] returned: ["BIGINT"], right reducers: [] returned: ["INT"]. SQLSTATE: 42K09 Both reducer lists are empty, which is the sign that there was nothing left to reduce and nothing to compare. `INT` is the `years` transform's own result type, not a type any key row holds. ### Does this PR introduce _any_ user-facing change? Yes. The query above returns its result instead of failing. Only unreleased versions are affected: the failure is reachable through SPARK-59121, and before that the same shape failed on a `ClassCastException` from applying the reduce a second time. ### How was this patch tested? Two new `KeyGroupedPartitioningSuite` tests. The first covers the shape above in both join orders, since the side to leave out can be either one, and with both an inner and a full outer join. The inner join intersects the two key sets to nothing and so has nothing to sort, while the full outer join keeps the other side's keys and sorts them by the reported types, which is what makes those types matter. Each part of the fix fails this test on its own when disabled. The second covers an empty side that is not marked, to pin that it stays in the comparison. A one-side `days` -> `years` reduce whose `years` side is emptied by an upstream inner join under the partition filter, against a reducer returning `DateType` where the target transform is `IntegerType`, still raises `STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) Closes #58486 from peter-toth/SPARK-59176-reduced-key-types-no-key. Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Peter Toth <peter.toth@gmail.com>

…n one side reduced onto no key ### What changes were proposed in this pull request? `EnsureRequirements` leaves a side out of the comparison of the two sides' reduced key types when that side has no partition key and its expressions no longer describe the keys it would have had. The types then come from a side that does answer for them. `KeyedPartitioning.keyDataTypes` reports the types the partition key rows were built with. With no key row to read, it falls back to the partition expressions' own types. That is still the right answer while the expressions describe the keys, and a join that reduced both sides' keys leaves expressions that do not (`TransformExpression.reducedWith`, SPARK-59121). Only then is the fallback a type no key of that partitioning would hold, and only then must a caller keep it out of a comparison against a real answer. An empty side that nothing reduced stays in the comparison, which is what keeps the comparison doing its other job. Where one side has a reducer, it holds the connector's `Reducer.resultType()` against the paired transform, and that needs no key row. The `keyDataTypes` scaladoc states the rule the fix follows, in place of the paragraph that described the failure and pointed here. ### Why are the changes needed? A storage-partitioned join whose two legs each reduced both of their sides onto one key space is co-partitioned, and joins without a shuffle. If a leg ends up with no partition key at all, the query fails instead. `v2BucketingPartitionFilterEnabled` produces such a leg whenever its two sides hold disjoint keys, i.e. whenever that leg is empty. SELECT coalesce(l.ts, r.ts) FROM (SELECT d.ts FROM days1 d JOIN years1 y ON y.ts = d.ts) l JOIN (SELECT y.ts FROM days2 d JOIN years2 y ON y.ts = d.ts) r ON l.ts = r.ts With `days2` and `years2` holding disjoint years, and `days` and `years` reducing onto a common `LongType` year key: [STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES] Storage-partition join partition transforms produced incompatible reduced types, left reducers: [] returned: ["BIGINT"], right reducers: [] returned: ["INT"]. SQLSTATE: 42K09 Both reducer lists are empty, which is the sign that there was nothing left to reduce and nothing to compare. `INT` is the `years` transform's own result type, not a type any key row holds. ### Does this PR introduce _any_ user-facing change? Yes. The query above returns its result instead of failing. Only unreleased versions are affected: the failure is reachable through SPARK-59121, and before that the same shape failed on a `ClassCastException` from applying the reduce a second time. ### How was this patch tested? Two new `KeyGroupedPartitioningSuite` tests. The first covers the shape above in both join orders, since the side to leave out can be either one, and with both an inner and a full outer join. The inner join intersects the two key sets to nothing and so has nothing to sort, while the full outer join keeps the other side's keys and sorts them by the reported types, which is what makes those types matter. Each part of the fix fails this test on its own when disabled. The second covers an empty side that is not marked, to pin that it stays in the comparison. A one-side `days` -> `years` reduce whose `years` side is emptied by an upstream inner join under the partition filter, against a reducer returning `DateType` where the target transform is `IntegerType`, still raises `STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) #### Backport to branch-4.3 A clean cherry-pick of #58486's two commits, squashed. Nothing was tailored. Measured on the branch tip: `SPARK-59176: a leg reduced onto no key at all still joins` fails there with `STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES`, the same shape as on master, so the branch is affected. The precondition is present: SPARK-59121 reached `branch-4.3` as #58481. 198 tests green across `KeyGroupedPartitioningSuite`, `GroupPartitionsExecSuite` and `EnsureRequirementsSuite`, plus 17 in `ShuffleSpecSuite`. `dev/lint-scala` clean. Closes #58499 from peter-toth/SPARK-59176-reduced-key-types-no-key-4.3. Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…n one side reduced onto no key ### What changes were proposed in this pull request? `EnsureRequirements` leaves a side out of the comparison of the two sides' reduced key types when that side has no partition key and its expressions no longer describe the keys it would have had. The types then come from a side that does answer for them. `KeyedPartitioning.keyDataTypes` reports the types the partition key rows were built with. With no key row to read, it falls back to the partition expressions' own types. That is still the right answer while the expressions describe the keys, and a join that reduced both sides' keys leaves expressions that do not (`TransformExpression.reducedWith`, SPARK-59121). Only then is the fallback a type no key of that partitioning would hold, and only then must a caller keep it out of a comparison against a real answer. An empty side that nothing reduced stays in the comparison, which is what keeps the comparison doing its other job. Where one side has a reducer, it holds the connector's `Reducer.resultType()` against the paired transform, and that needs no key row. The `keyDataTypes` scaladoc states the rule the fix follows, in place of the paragraph that described the failure and pointed here. ### Why are the changes needed? A storage-partitioned join whose two legs each reduced both of their sides onto one key space is co-partitioned, and joins without a shuffle. If a leg ends up with no partition key at all, the query fails instead. `v2BucketingPartitionFilterEnabled` produces such a leg whenever its two sides hold disjoint keys, i.e. whenever that leg is empty. SELECT coalesce(l.ts, r.ts) FROM (SELECT d.ts FROM days1 d JOIN years1 y ON y.ts = d.ts) l JOIN (SELECT y.ts FROM days2 d JOIN years2 y ON y.ts = d.ts) r ON l.ts = r.ts With `days2` and `years2` holding disjoint years, and `days` and `years` reducing onto a common `LongType` year key: [STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES] Storage-partition join partition transforms produced incompatible reduced types, left reducers: [] returned: ["BIGINT"], right reducers: [] returned: ["INT"]. SQLSTATE: 42K09 Both reducer lists are empty, which is the sign that there was nothing left to reduce and nothing to compare. `INT` is the `years` transform's own result type, not a type any key row holds. ### Does this PR introduce _any_ user-facing change? Yes. The query above returns its result instead of failing. Only unreleased versions are affected: the failure is reachable through SPARK-59121, and before that the same shape failed on a `ClassCastException` from applying the reduce a second time. ### How was this patch tested? Two new `KeyGroupedPartitioningSuite` tests. The first covers the shape above in both join orders, since the side to leave out can be either one, and with both an inner and a full outer join. The inner join intersects the two key sets to nothing and so has nothing to sort, while the full outer join keeps the other side's keys and sorts them by the reported types, which is what makes those types matter. Each part of the fix fails this test on its own when disabled. The second covers an empty side that is not marked, to pin that it stays in the comparison. A one-side `days` -> `years` reduce whose `years` side is emptied by an upstream inner join under the partition filter, against a reducer returning `DateType` where the target transform is `IntegerType`, still raises `STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) #### Backport to branch-4.2 A cherry-pick of #58486's two commits, squashed, with **one line tailored**: the config is named `V2_BUCKETING_ALLOW_JOIN_KEYS_SUBSET_OF_PARTITION_KEYS` on this branch, not `V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS`. Nothing else differs from the `branch-4.3` backport. Measured on the branch tip: `SPARK-59176: a leg reduced onto no key at all still joins` fails there with `STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES`, the same shape as on master, so the branch is affected. The precondition is present: SPARK-59121 reached `branch-4.2` as #58482. 172 tests green across `KeyGroupedPartitioningSuite`, `GroupPartitionsExecSuite` and `EnsureRequirementsSuite`, plus 12 in `ShuffleSpecSuite`. `dev/lint-scala` clean. Closes #58500 from peter-toth/SPARK-59176-reduced-key-types-no-key-4.2. Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

szehon-ho approved these changes Sep 2, 2026

View reviewed changes

dongjoon-hyun reviewed Sep 3, 2026

View reviewed changes

uros-b approved these changes Sep 3, 2026

View reviewed changes

dongjoon-hyun approved these changes Sep 3, 2026

View reviewed changes

peter-toth closed this in 3c70da5 Sep 3, 2026

This was referenced Sep 3, 2026

[SPARK-59176][SQL][4.3] Fix a storage-partitioned join that fails when one side reduced onto no key #58499

Closed

[SPARK-59176][SQL][4.2] Fix a storage-partitioned join that fails when one side reduced onto no key #58500

Closed

		@@ -612,17 +612,18 @@ case class KeyedPartitioning(
		*
		* The two cases can meet, and then the fallback is not truthful. A marked partitioning can end up

Uh oh!

Conversation

peter-toth commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

peter-toth commented Sep 2, 2026

Uh oh!

szehon-ho left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uros-b commented Sep 3, 2026

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peter-toth commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peter-toth commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

peter-toth commented Sep 2, 2026 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

peter-toth commented Sep 3, 2026 •

edited

Loading