diff --git a/.claude/skills/audit-comet-expression/SKILL.md b/.claude/skills/audit-comet-expression/SKILL.md index e864d23fa88..f0435c0593f 100644 --- a/.claude/skills/audit-comet-expression/SKILL.md +++ b/.claude/skills/audit-comet-expression/SKILL.md @@ -1,6 +1,6 @@ --- name: audit-comet-expression -description: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests. +description: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests. argument-hint: --- @@ -29,7 +29,7 @@ Audit the Comet implementation of the `$ARGUMENTS` expression for correctness an This audit covers: -1. Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1 +1. Spark implementation across versions 3.5.8, 4.0.1, and 4.1.1 2. Comet Scala serde implementation 3. Comet Rust / DataFusion implementation 4. Existing test coverage (Comet SQL Tests and Comet Scala Tests) @@ -43,7 +43,7 @@ Clone specific Spark version tags (use shallow clones to avoid polluting the wor ```bash set -eu -o pipefail -for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do +for tag in v3.5.8 v4.0.1 v4.1.1; do dir="/tmp/spark-${tag}" if [ ! -d "$dir" ]; then git clone --depth 1 --branch "$tag" https://github.com/apache/spark.git "$dir" @@ -56,7 +56,7 @@ done Search the Catalyst SQL expressions source: ```bash -for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do +for tag in v3.5.8 v4.0.1 v4.1.1; do dir="/tmp/spark-${tag}" echo "=== $tag ===" find "$dir/sql/catalyst/src/main/scala" -name "*.scala" | \ @@ -67,7 +67,7 @@ done If the expression is not found in catalyst, also check core: ```bash -for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do +for tag in v3.5.8 v4.0.1 v4.1.1; do dir="/tmp/spark-${tag}" echo "=== $tag ===" find "$dir/sql" -name "*.scala" | \ @@ -90,7 +90,6 @@ For each Spark version, read the expression file and note: Produce a concise diff summary of what changed between: -- 3.4.3 → 3.5.8 - 3.5.8 → 4.0.1 - 4.0.1 → 4.1.1 @@ -107,7 +106,7 @@ Pay attention to: ## Step 2: Locate the Spark Tests ```bash -for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do +for tag in v3.5.8 v4.0.1 v4.1.1; do dir="/tmp/spark-${tag}" echo "=== $tag ===" find "$dir/sql" -name "*.scala" -path "*/test/*" | \ @@ -732,7 +731,7 @@ used for the expression in `docs/source/user-guide/latest/expressions.md`. other pages. - Add (or update) a `## ` section, keeping sections alphabetically ordered. - Under that heading, add one bullet per Spark version checked, each including: - - Spark version (e.g. 3.4.3, 3.5.8, 4.0.1, 4.1.1) + - Spark version (e.g. 3.5.8, 4.0.1, 4.1.1) - Today's date - A brief note for any version-specific finding (behavioral difference, known incompatibility); omit the note if nothing notable. @@ -744,7 +743,7 @@ used for the expression in `docs/source/user-guide/latest/expressions.md`. Present the audit as: 1. **Expression Summary** - Brief description of what `$ARGUMENTS` does, its input/output types, and null behavior -2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1 +2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.5.8, 4.0.1, and 4.1.1 3. **Comet Implementation Notes** - Summary of how Comet implements this expression and any concerns 4. **Coverage Gap Analysis** - The gap table from Step 5, plus implementation gaps 5. **Recommendations** - Prioritized list from Step 6 diff --git a/.claude/skills/implement-comet-expression/SKILL.md b/.claude/skills/implement-comet-expression/SKILL.md index 529293f3784..7aa0fa70a00 100644 --- a/.claude/skills/implement-comet-expression/SKILL.md +++ b/.claude/skills/implement-comet-expression/SKILL.md @@ -88,7 +88,7 @@ make ### 4. Run the audit skill -Once the initial implementation passes its smoke test, run the `audit-comet-expression` skill on `$ARGUMENTS`. It compares the implementation and tests against Spark 3.4.3, 3.5.8, and 4.0.1 and produces a prioritized list of gaps. +Once the initial implementation passes its smoke test, run the `audit-comet-expression` skill on `$ARGUMENTS`. It compares the implementation and tests against Spark 3.5.8, 4.0.1, and 4.1.1 and produces a prioritized list of gaps. ### 5. Implement audit-recommended tests and iterate diff --git a/.claude/skills/wire-datafusion-function/SKILL.md b/.claude/skills/wire-datafusion-function/SKILL.md index 02bc493ceb0..ac425aaab46 100644 --- a/.claude/skills/wire-datafusion-function/SKILL.md +++ b/.claude/skills/wire-datafusion-function/SKILL.md @@ -139,7 +139,7 @@ make cd native && cargo clippy --all-targets --workspace -- -D warnings ``` -Then run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.4.3 / 3.5.8 / 4.0.1 and surface coverage gaps; iterate on tests. +Then run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.5.8 / 4.0.1 / 4.1.1 and surface coverage gaps; iterate on tests. The user generally runs tests themselves. If asked for a smoke test: diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 8a8791e7118..674c99d3f75 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -38,7 +38,7 @@ is a `workflow_call` reusable invoked from the umbrella. (PR+push) (PR+push) (PR+push) | | | v v v - spark_3_4 / spark_4_1 iceberg_1_8 / 1_9 + spark_4_1 iceberg_1_8 / 1_9 (push or PR + label) (push only) reusable workflows invoked via `uses:`: @@ -60,7 +60,6 @@ is a `workflow_call` reusable invoked from the umbrella. | `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` | | `spark_3_5` | PR or push, paths matched | Spark 3.5 sources | | `spark_4_0` | PR or push, paths matched | Spark 4.0 sources | -| `spark_3_4` | push, **or** PR with `run-spark-3.4-tests` label | Spark 3.4 sources | | `spark_4_1` | push, **or** PR with `run-spark-4.1-tests` label | Spark 4.1 sources | | `iceberg_1_10` | PR or push, paths matched | Iceberg sources | | `iceberg_1_8` | push only | Iceberg sources | @@ -86,14 +85,14 @@ umbrella doesn't watch, or operate independently of the rest of CI: ## Reusable workflows (called by `ci.yml`) -| File | Called from `ci.yml` job(s) | -| --------------------------------- | -------------------------------------------------- | -| `pr_build_linux.yml` | `pr_build_linux` | -| `pr_build_macos.yml` | `pr_build_macos` | -| `pr_benchmark_check.yml` | `pr_benchmark_check` | -| `docs.yaml` | `docs` | -| `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1` | -| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` | +| File | Called from `ci.yml` job(s) | +| --------------------------------- | -------------------------------------------- | +| `pr_build_linux.yml` | `pr_build_linux` | +| `pr_build_macos.yml` | `pr_build_macos` | +| `pr_benchmark_check.yml` | `pr_benchmark_check` | +| `docs.yaml` | `docs` | +| `spark_sql_test_reusable.yml` | `spark_3_5`, `spark_4_0`, `spark_4_1` | +| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` | ## Modifying path filters diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 59cc05215f1..bb52803f3d4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -55,7 +55,6 @@ jobs: if: >- github.event_name != 'pull_request' || github.event.action != 'labeled' || - github.event.label.name == 'run-spark-3.4-tests' || github.event.label.name == 'run-spark-4.0-tests' || github.event.label.name == 'run-spark-4.1-tests' || github.event.label.name == 'run-iceberg-tests' @@ -110,7 +109,6 @@ jobs: build_macos: ${{ steps.compute.outputs.build_macos }} benchmark: ${{ steps.compute.outputs.benchmark }} docs: ${{ steps.compute.outputs.docs }} - spark_3_4: ${{ steps.compute.outputs.spark_3_4 }} spark_3_5: ${{ steps.compute.outputs.spark_3_5 }} spark_4_0: ${{ steps.compute.outputs.spark_4_0 }} spark_4_1: ${{ steps.compute.outputs.spark_4_1 }} @@ -137,7 +135,7 @@ jobs: run: | set -euo pipefail if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then - for key in build_linux build_macos benchmark docs spark_3_4 spark_3_5 spark_4_0 spark_4_1 iceberg_1_8 iceberg_1_9 iceberg_1_10 iceberg_1_11; do + for key in build_linux build_macos benchmark docs spark_3_5 spark_4_0 spark_4_1 iceberg_1_8 iceberg_1_9 iceberg_1_10 iceberg_1_11; do echo "${key}=true" >> "$GITHUB_OUTPUT" done exit 0 @@ -201,22 +199,6 @@ jobs: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') uses: ./.github/workflows/docs.yaml - spark_3_4: - name: Spark SQL Tests (Spark 3.4) - needs: changes - # Main-only by default; PRs need the `run-spark-3.4-tests` label. - if: | - needs.changes.outputs.spark_3_4 == 'true' && - (github.event_name == 'push' || - github.event_name == 'workflow_dispatch' || - (github.event_name == 'pull_request' && - contains(github.event.pull_request.labels.*.name, 'run-spark-3.4-tests'))) - uses: ./.github/workflows/spark_sql_test_reusable.yml - with: - spark-short: '3.4' - spark-full: '3.4.3' - java: 11 - spark_3_5: name: Spark SQL Tests (Spark 3.5) needs: changes @@ -277,9 +259,9 @@ jobs: with: iceberg-short: '1.8' iceberg-full: '1.8.1' - spark-short: '3.4' - spark-full: '3.4.3' - java: 11 + spark-short: '3.5' + spark-full: '3.5.9' + java: 17 iceberg_1_9: name: Iceberg Spark SQL Tests (Iceberg 1.9) diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml index 18dc8e4086c..1fc204c8817 100644 --- a/.github/workflows/pr_build_linux.yml +++ b/.github/workflows/pr_build_linux.yml @@ -83,9 +83,6 @@ jobs: strategy: matrix: profile: - - name: "Spark 3.4, JDK 11, Scala 2.12" - java_version: "11" - maven_opts: "-Pspark-3.4 -Pscala-2.12" - name: "Spark 3.5, JDK 17, Scala 2.12" java_version: "17" maven_opts: "-Pspark-3.5 -Pscala-2.12" @@ -276,10 +273,6 @@ jobs: # the goal with these profiles is to get coverage of all Java, Scala, and Spark # versions without testing all possible combinations, which would be overkill profile: - - name: "Spark 3.4, JDK 11, Scala 2.12" - java_version: "11" - maven_opts: "-Pspark-3.4 -Pscala-2.12" - - name: "Spark 3.5, JDK 17, Scala 2.13" java_version: "17" maven_opts: "-Pspark-3.5 -Pscala-2.13" @@ -336,7 +329,7 @@ jobs: - name: "exec" value: | org.apache.comet.exec.CometAggregateSuite - org.apache.comet.exec.CometExec3_4PlusSuite + org.apache.comet.exec.CometExecCompatibilitySuite org.apache.comet.exec.CometExecSuite org.apache.comet.exec.CometGenerateExecSuite org.apache.comet.exec.CometWindowExecSuite diff --git a/.github/workflows/pr_build_macos.yml b/.github/workflows/pr_build_macos.yml index fa82e4ebca7..24a3840cc21 100644 --- a/.github/workflows/pr_build_macos.yml +++ b/.github/workflows/pr_build_macos.yml @@ -152,7 +152,7 @@ jobs: - name: "exec" value: | org.apache.comet.exec.CometAggregateSuite - org.apache.comet.exec.CometExec3_4PlusSuite + org.apache.comet.exec.CometExecCompatibilitySuite org.apache.comet.exec.CometExecSuite org.apache.comet.exec.CometGenerateExecSuite org.apache.comet.exec.CometWindowExecSuite diff --git a/.github/workflows/pyarrow_udf_test.yml b/.github/workflows/pyarrow_udf_test.yml index 1a03962685d..bc433461db0 100644 --- a/.github/workflows/pyarrow_udf_test.yml +++ b/.github/workflows/pyarrow_udf_test.yml @@ -33,7 +33,6 @@ on: - "spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala" - "spark/src/main/scala/org/apache/spark/sql/comet/CometMapInBatchExec.scala" - "spark/src/main/scala/org/apache/spark/sql/comet/shims/MapInBatchInfo.scala" - - "spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala" - "spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala" - "spark/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala" - "spark/src/main/spark-4.1/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala" diff --git a/.github/workflows/spark_sql_writer_tests.yml b/.github/workflows/spark_sql_writer_tests.yml index c813a8b39e1..02fc41572d4 100644 --- a/.github/workflows/spark_sql_writer_tests.yml +++ b/.github/workflows/spark_sql_writer_tests.yml @@ -68,7 +68,7 @@ jobs: run: | # Map each supported Spark minor version to its full version + JDK. # Mirrors ci.yml's per-version reusable invocations (default-on PR - # versions only; 3.4 and 4.0 are label-gated and not offered here). + # versions only; 4.0 is label-gated and not offered here). case "${{ inputs.spark-version }}" in 3.5) spark_full=3.5.9; java=17 ;; 4.1) spark_full=4.1.3; java=17 ;; diff --git a/README.md b/README.md index 6eccd078161..715b2147d89 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ benefits of Comet's acceleration capabilities without disrupting your Spark appl ## Getting Started -Comet supports Apache Spark 3.4, 3.5, 4.0, and 4.1, and provides experimental support for Spark 4.2. See the +Comet supports Apache Spark 3.5, 4.0, and 4.1, and provides experimental support for Spark 4.2. See the [installation guide](https://datafusion.apache.org/comet/user-guide/installation.html) for the detailed version, Java, and Scala compatibility matrix. diff --git a/benchmarks/README.md b/benchmarks/README.md index a95d2e2d3d6..5a84ed2d763 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -62,7 +62,7 @@ docker push localhost:32000/apache/datafusion-comet-tpcbench:latest export SPARK_MASTER=k8s://https://127.0.0.1:16443 export COMET_DOCKER_IMAGE=localhost:32000/apache/datafusion-comet-tpcbench:latest # Location of Comet JAR within the Docker image -export COMET_JAR=/opt/spark/jars/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar +export COMET_JAR=/opt/spark/jars/comet-spark-spark3.5_2.12-0.5.0-SNAPSHOT.jar $SPARK_HOME/bin/spark-submit \ --master $SPARK_MASTER \ diff --git a/dev/ci/compute-changes.py b/dev/ci/compute-changes.py index 9b7cd2f1691..33134ee2b54 100644 --- a/dev/ci/compute-changes.py +++ b/dev/ci/compute-changes.py @@ -89,28 +89,6 @@ "spark/src/main/scala/org/apache/comet/expressions/**", "spark/src/main/spark-*/**", ], - "spark_3_4": [ - "native/**/src/**", - "native/**/Cargo.toml", - "native/Cargo.lock", - "common/src/main/**", - "common/pom.xml", - "spark/src/main/**", - "!spark/src/main/spark-3.5/**", - "!spark/src/main/spark-4.0/**", - "!spark/src/main/spark-4.1/**", - "!spark/src/main/spark-4.2/**", - "!spark/src/main/spark-4.x/**", - "!spark/src/main/scala/org/apache/comet/GenerateDocs.scala", - "spark/pom.xml", - "dev/diffs/3.4.3.diff", - "pom.xml", - "rust-toolchain.toml", - ".github/workflows/ci.yml", - ".github/workflows/spark_sql_test_reusable.yml", - ".github/actions/setup-builder/**", - ".github/actions/setup-spark-builder/**", - ], "spark_3_5": [ "native/**/src/**", "native/**/Cargo.toml", @@ -118,7 +96,6 @@ "common/src/main/**", "common/pom.xml", "spark/src/main/**", - "!spark/src/main/spark-3.4/**", "!spark/src/main/spark-4.0/**", "!spark/src/main/spark-4.1/**", "!spark/src/main/spark-4.2/**", @@ -140,7 +117,6 @@ "common/src/main/**", "common/pom.xml", "spark/src/main/**", - "!spark/src/main/spark-3.4/**", "!spark/src/main/spark-3.5/**", "!spark/src/main/spark-3.x/**", "!spark/src/main/spark-4.1/**", @@ -162,7 +138,6 @@ "common/src/main/**", "common/pom.xml", "spark/src/main/**", - "!spark/src/main/spark-3.4/**", "!spark/src/main/spark-3.5/**", "!spark/src/main/spark-3.x/**", "!spark/src/main/spark-4.0/**", diff --git a/dev/diffs/3.4.3.diff b/dev/diffs/3.4.3.diff deleted file mode 100644 index b90b7e2a6c8..00000000000 --- a/dev/diffs/3.4.3.diff +++ /dev/null @@ -1,3181 +0,0 @@ -diff --git a/pom.xml b/pom.xml -index d3544881af1..ff963395ec3 100644 ---- a/pom.xml -+++ b/pom.xml -@@ -148,6 +148,8 @@ - 0.10.0 - 2.5.1 - 2.0.8 -+ 3.4 -+ 1.1.0-SNAPSHOT - - Spark 3.4, 3.5, and 4.0 all have slightly different QueryExecutionErrors APIs. + Spark 3.5 and the Spark 4.x line have different QueryExecutionErrors APIs. Comet ships a separate ShimSparkErrorConverter implementation for each version. @@ -17,52 +17,39 @@ - - spark-3.4 - spark-3.5 - spark-4.0 + spark-3.5 + spark-4.x ShimSparkErrorConverter - spark-3.4 edition - Uses Spark 3.4 API signatures - (no SQLQueryContext in some calls) - - - ShimSparkErrorConverter - spark-3.5 edition - Uses Spark 3.5 API signatures - (takes SQLQueryContext directly) + spark-3.5 edition + Uses Spark 3.5 API signatures + (takes SQLQueryContext directly) ShimSparkErrorConverter - spark-4.0 edition - Uses Spark 4.0 API signatures + spark-4.x edition + Uses Spark 4.x API signatures (new functionName param in some) - QueryExecutionErrors - Spark 3.4 internal API - - - QueryExecutionErrors - Spark 3.5 internal API + Spark 3.5 internal API QueryExecutionErrors - Spark 4.0 internal API + Spark 4.x internal API - Example difference: "BinaryArithmeticOverflow" — Spark 3.x does NOT take functionName param, Spark 4.0 does. + Example difference: "BinaryArithmeticOverflow" — Spark 3.5 does NOT take functionName param, Spark 4.x does. diff --git a/docs/source/contributor-guide/spark-sql-tests.md b/docs/source/contributor-guide/spark-sql-tests.md index c90aca97f19..092633acf13 100644 --- a/docs/source/contributor-guide/spark-sql-tests.md +++ b/docs/source/contributor-guide/spark-sql-tests.md @@ -30,14 +30,14 @@ Here is an overview of the changes that we need to make to Spark: - Modify TestHive to load Comet - Modify SQLTestUtilsBase to load Comet when `ENABLE_COMET` environment variable exists -Here are the steps involved in running the Spark SQL tests with Comet, using Spark 3.4.3 for this example. +Here are the steps involved in running the Spark SQL tests with Comet, using Spark 3.5.9 for this example. ## 1. Install Comet Run `make release` in Comet to install the Comet JAR into the local Maven repository, specifying the Spark version. ```shell -PROFILES="-Pspark-3.4" make release +PROFILES="-Pspark-3.5" make release ``` ## 2. Clone Spark and Apply Diff @@ -47,9 +47,9 @@ Clone Apache Spark locally and apply the diff file from Comet. Note: this is a shallow clone of a tagged Spark commit and is not suitable for general Spark development. ```shell -git clone -b 'v3.4.3' --single-branch --depth 1 git@github.com:apache/spark.git apache-spark +git clone -b 'v3.5.9' --single-branch --depth 1 git@github.com:apache/spark.git apache-spark cd apache-spark -git apply ../datafusion-comet/dev/diffs/3.4.3.diff +git apply ../datafusion-comet/dev/diffs/3.5.9.diff ``` ## 3. Run Spark SQL Tests @@ -96,7 +96,7 @@ ENABLE_COMET=true ENABLE_COMET_ONHEAP=true sbt -J-Xmx4096m -Dspark.test.includeS 2. Set `ENABLE_COMET=true` in environment variables ![img.png](img.png) -3. After the above tests are configured, spark tests can be run with debugging enabled on spark/comet code. Note that Comet is added as a dependency and the classes are readonly while debugging from Spark. Any new changes to Comet are to be built and deployed locally through the command (`PROFILES="-Pspark-3.4" make release`) +3. After the above tests are configured, spark tests can be run with debugging enabled on spark/comet code. Note that Comet is added as a dependency and the classes are readonly while debugging from Spark. Any new changes to Comet are to be built and deployed locally through the command (`PROFILES="-Pspark-3.5" make release`) ## Creating a diff file for a new Spark version diff --git a/docs/source/contributor-guide/spark_configs_support.md b/docs/source/contributor-guide/spark_configs_support.md index b2c55bc1462..5e3584eb857 100644 --- a/docs/source/contributor-guide/spark_configs_support.md +++ b/docs/source/contributor-guide/spark_configs_support.md @@ -93,7 +93,7 @@ from passing through silent Spark fallback. **Affected expressions.** Determined by tracing `TimestampFormatterHelper`, `TimestampFormatter(...)`, and `DateFormatter(...)` usage in `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala` -across Spark 3.4, 3.5, 4.0, and 4.1. Three expression classes mix in +across Spark 3.5, 4.0, and 4.1. Three expression classes mix in `TimestampFormatterHelper`: - `DateFormatClass` -- `date_format` diff --git a/docs/source/contributor-guide/sql_error_propagation.md b/docs/source/contributor-guide/sql_error_propagation.md index a27408510dd..bcd1d41b50b 100644 --- a/docs/source/contributor-guide/sql_error_propagation.md +++ b/docs/source/contributor-guide/sql_error_propagation.md @@ -398,9 +398,8 @@ def convertToSparkException(e: CometQueryExecutionException): Throwable = { ### `ShimSparkErrorConverter` calls the real Spark API -Because Spark's `QueryExecutionErrors` API changes between Spark versions (3.4, 3.5, and the 4.x line), -there is a separate implementation per branch (in `spark-3.4/`, `spark-3.5/`, and `spark-4.x/`, which is -shared by Spark 4.0, 4.1, and 4.2). +Because Spark's `QueryExecutionErrors` API changes between Spark 3.5 and the 4.x line, there is a +separate implementation in `spark-3.5/` and `spark-4.x/` (shared by Spark 4.0, 4.1, and 4.2). ![Shim pattern for per-version Spark API bridging](./shim_pattern.svg) diff --git a/docs/source/user-guide/latest/compatibility/expressions/index.md b/docs/source/user-guide/latest/compatibility/expressions/index.md index 35746991951..6c0e926682c 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/index.md +++ b/docs/source/user-guide/latest/compatibility/expressions/index.md @@ -26,7 +26,6 @@ version you are running: ```{toctree} :maxdepth: 2 -Spark 3.4 Spark 3.5 Spark 4.0 Spark 4.1 diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md deleted file mode 100644 index 40fe54c4eab..00000000000 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md +++ /dev/null @@ -1,43 +0,0 @@ - - -# Expression Compatibility (Spark 3.4) - -Compatibility notes for Comet running on Apache Spark 3.4. Expressions that are not 100% -Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch -path, which stay in Comet's native pipeline and match Spark exactly. Set -`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark -expression class name, to run Comet's native implementation despite its differences -from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md) -for more information on this configuration setting. - -```{toctree} -:maxdepth: 1 - -aggregate -array -datetime -map -math -misc -string -struct -url -cast -``` diff --git a/docs/source/user-guide/latest/compatibility/spark-versions.md b/docs/source/user-guide/latest/compatibility/spark-versions.md index 090a1df85e8..1bd55ffdef7 100644 --- a/docs/source/user-guide/latest/compatibility/spark-versions.md +++ b/docs/source/user-guide/latest/compatibility/spark-versions.md @@ -26,25 +26,6 @@ compatibility guide. For the rule that governs how long each Spark minor is supp [Apache Spark version support section](../../../about/versioning_policy.md#apache-spark-version-support) of the versioning policy. -## Spark 3.4 - -Spark 3.4.3 is supported with Java 11/17 and Scala 2.12/2.13. - -```{warning} -Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in the 1.1.0 release. -``` - -### Known Limitations - -- **Reading `TimestampLTZ` as `TimestampNTZ`**: Spark 3.4 raises an error for this operation - (SPARK-36182), but Comet's Parquet scan silently returns the raw UTC value instead. - See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details. - -- **Unsupported Parquet type conversions**: Spark 3.4 raises schema incompatibility errors for - certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision changes), but Comet's - Comet's Parquet scan may not detect these and could return unexpected values. - See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details. - ## Spark 3.5 Spark 3.5.9 is supported with Java 11/17 and Scala 2.12/2.13. diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md index 192eae807eb..9404c7779bc 100644 --- a/docs/source/user-guide/latest/expressions.md +++ b/docs/source/user-guide/latest/expressions.md @@ -701,4 +701,4 @@ This list is illustrative, not exhaustive: the per-function tables are not the c ## See also - [Comet Compatibility Guide](compatibility/index.md) - known incompatibilities and edge cases for supported expressions. -- [Expression Audits (contributor guide)](../../contributor-guide/expression-audits/index.md) - per-version (Spark 3.4 / 3.5 / 4.0 / 4.1) audit notes for audited expressions. +- [Expression Audits (contributor guide)](../../contributor-guide/expression-audits/index.md) - per-version (Spark 3.5 / 4.0 / 4.1) audit notes for audited expressions. diff --git a/docs/source/user-guide/latest/iceberg-writes.md b/docs/source/user-guide/latest/iceberg-writes.md index a2bdbc091b2..fcfbeec30bb 100644 --- a/docs/source/user-guide/latest/iceberg-writes.md +++ b/docs/source/user-guide/latest/iceberg-writes.md @@ -75,7 +75,7 @@ supports: The mechanism behind row-level DML differs by Spark version: on Spark 4.0+ the analyzer emits operation-coded rows that Comet's writer dispatches through `ReplaceData`'s projections, while -on Spark 3.4/3.5 the rewritten rows are written as a plain row stream. The supported set of +on Spark 3.5 the rewritten rows are written as a plain row stream. The supported set of operations is the same either way. On Spark 4.1+ the split plan matches two further stock-Spark behaviours: MERGE metrics are @@ -90,8 +90,6 @@ The rewrite is skipped — and the write runs through Spark's stock combined ope - `spark.comet.write.iceberg.splitOperator.enabled` is `false` (the default); - the write is not an Iceberg `SparkWrite` (any other V2 data source); - the table uses merge-on-read: delta writes (Iceberg `WriteDelta`) are not intercepted; -- the statement is CTAS / RTAS on Spark 3.4, where the staged exec writes inline; on Spark - 3.5+ those statements re-plan their inner append, which is intercepted normally; - the write requires Spark's commit coordinator, which Comet's per-task commit protocol does not use; - Comet cannot reflect the Iceberg internals needed to build the two-operator plan (for diff --git a/docs/source/user-guide/latest/installation.md b/docs/source/user-guide/latest/installation.md index 7fc4a81d897..5029f2902e2 100644 --- a/docs/source/user-guide/latest/installation.md +++ b/docs/source/user-guide/latest/installation.md @@ -46,13 +46,12 @@ We recommend only using Comet with Spark versions where we currently have both C Other versions may work well enough for development and evaluation purposes. ```{warning} -JDK 11 and Spark 3.4 support are deprecated as of the 1.0.0 release and will be removed in the 1.1.0 release. -We recommend moving to JDK 17 or later and Spark 3.5 or later. +JDK 11 support is deprecated as of the 1.0.0 release and will be removed in the 1.1.0 release. +We recommend moving to JDK 17 or later. ``` | Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI | | ------------- | ------------ | ------------- | ----------------- | --------------------- | -| 3.4.3 | 11/17 | 2.12/2.13 | Yes | Yes | | 3.5.9 | 11/17 | 2.12/2.13 | Yes | Yes | | 4.0.4 | 17/21 | 2.13 | Yes | Yes | | 4.1.3 | 17/21 | 2.13 | Yes | Yes | @@ -91,8 +90,6 @@ repository describing your environment, and [build from source] for your target Here are the direct links for downloading the Comet $COMET_VERSION jar file. -- [Comet plugin for Spark 3.4 / Scala 2.12](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.4_2.12/$COMET_VERSION/comet-spark-spark3.4_2.12-$COMET_VERSION.jar) -- [Comet plugin for Spark 3.4 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.4_2.13/$COMET_VERSION/comet-spark-spark3.4_2.13-$COMET_VERSION.jar) - [Comet plugin for Spark 3.5 / Scala 2.12](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.12/$COMET_VERSION/comet-spark-spark3.5_2.12-$COMET_VERSION.jar) - [Comet plugin for Spark 3.5 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.13/$COMET_VERSION/comet-spark-spark3.5_2.13-$COMET_VERSION.jar) - [Comet plugin for Spark 4.0 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar) diff --git a/docs/source/user-guide/latest/pyarrow-udfs.md b/docs/source/user-guide/latest/pyarrow-udfs.md index ff493de23c0..cd0404de783 100644 --- a/docs/source/user-guide/latest/pyarrow-udfs.md +++ b/docs/source/user-guide/latest/pyarrow-udfs.md @@ -188,7 +188,7 @@ on the unoptimized path. `org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager` and enable `spark.comet.shuffle.enabled=true` at session startup. With a vanilla Spark `Exchange` in the plan the data leaves the shuffle as rows and the optimization cannot fire. -- Spark 4.0 or newer is required. On Spark 3.4 and 3.5 the optimization is a no-op even when +- Spark 4.0 or newer is required. On Spark 3.5 the optimization is a no-op even when enabled; vanilla `PythonMapInArrowExec` / `MapInPandasExec` handle the operation. The Spark 3.5 `PythonArrowInput` trait has a different contract than 4.x and a separate implementation has not been written. Track 3.5 support as a future follow-on if there is user demand. diff --git a/docs/source/user-guide/latest/s3-credential-providers.md b/docs/source/user-guide/latest/s3-credential-providers.md index 33777c497af..18592ad83a6 100644 --- a/docs/source/user-guide/latest/s3-credential-providers.md +++ b/docs/source/user-guide/latest/s3-credential-providers.md @@ -73,7 +73,7 @@ Catalog configuration you set under `spark.sql.catalog..*` (which may i Spark's RPC channel is plaintext by default and offers two opt-in encryption mechanisms, both off by default: - [`spark.network.crypto.enabled`](https://spark.apache.org/docs/latest/security.html#authentication-and-encryption) for AES-based RPC encryption keyed off the auth shared secret. -- [`spark.ssl.rpc.enabled`](https://spark.apache.org/docs/latest/security.html#ssl-configuration) for TLS on the same channel (Spark 3.4+). +- [`spark.ssl.rpc.enabled`](https://spark.apache.org/docs/latest/security.html#ssl-configuration) for TLS on the same channel. The same defaults apply to Hadoop delegation tokens, `CloudCredentialsProvider` JWTs, and any other secrets Spark already ships driver-to-executor, so this is a deployment-wide call rather than something specific to this SPI. See Spark's [security guide](https://spark.apache.org/docs/latest/security.html) for the full set of knobs. diff --git a/native/spark-expr/src/agg_funcs/quantile_summaries.rs b/native/spark-expr/src/agg_funcs/quantile_summaries.rs index 2ed6b320049..1d6a102015f 100644 --- a/native/spark-expr/src/agg_funcs/quantile_summaries.rs +++ b/native/spark-expr/src/agg_funcs/quantile_summaries.rs @@ -127,7 +127,7 @@ impl QuantileSummaries { 0 } else { // Spark: `math.floor(2 * relativeError * currentCount).toLong` - // (verified `.toLong` in 3.4/3.5/4.0/4.1), matching our i64. + // (verified `.toLong` in 3.5/4.0/4.1), matching our i64. (2.0 * self.relative_error * current_count as f64).floor() as i64 }; new_samples.push(Stats { diff --git a/native/spark-expr/src/math_funcs/negative.rs b/native/spark-expr/src/math_funcs/negative.rs index a0da6f52fb3..575d5eb97fe 100644 --- a/native/spark-expr/src/math_funcs/negative.rs +++ b/native/spark-expr/src/math_funcs/negative.rs @@ -108,7 +108,7 @@ impl PhysicalExpr for NegativeExpr { // The shims render this as `{from_type} overflow` under // `ARITHMETIC_OVERFLOW`. For byte/short that is byte-identical to Spark // 4.x, which routes them through `MathUtils.negateExact` ("byte overflow" / - // "short overflow"). Spark 3.4/3.5 instead throw + // "short overflow"). Spark 3.5 instead throws // `_LEGACY_ERROR_TEMP_2043` ("- caused overflow."); that error // class is out of reach here, so no string can match every version. let from_type = match array.data_type() { diff --git a/native/spark-expr/src/nondetermenistic_funcs/uuid.rs b/native/spark-expr/src/nondetermenistic_funcs/uuid.rs index c2f9944b0a7..3ba690e2710 100644 --- a/native/spark-expr/src/nondetermenistic_funcs/uuid.rs +++ b/native/spark-expr/src/nondetermenistic_funcs/uuid.rs @@ -221,7 +221,7 @@ mod tests { /// from this Rust code instead of from Commons Math3 would make the test circular and silently /// stop it testing anything. The backstop is `CometUuidExpressionSuite`, which builds /// `Uuid(Some(seed))` directly and compares Comet against Spark bit for bit on *every* - /// supported profile -- including 3.4 and 3.5, where the SQL `uuid(seed)` form does not exist + /// supported profile -- including 3.5, where the SQL `uuid(seed)` form does not exist /// and `uuid_with_seed.sql` is skipped. If these values ever disagree with that suite, trust /// the suite. #[test] diff --git a/native/spark-expr/src/string_funcs/base64.rs b/native/spark-expr/src/string_funcs/base64.rs index d394cfdccfb..62461f6a4ed 100644 --- a/native/spark-expr/src/string_funcs/base64.rs +++ b/native/spark-expr/src/string_funcs/base64.rs @@ -30,8 +30,8 @@ use datafusion::physical_plan::ColumnarValue; /// Spark `base64(bin)`: encodes a binary value as a padded base64 string. /// /// The second argument is a boolean `chunk` flag mirroring Spark's -/// `spark.sql.chunkBase64String.enabled`. When `chunk` is true (Spark's default, and the only -/// behavior on Spark 3.4), the output matches `java.util.Base64.getMimeEncoder()`: lines of at most +/// `spark.sql.chunkBase64String.enabled`. When `chunk` is true (Spark's default), the output +/// matches `java.util.Base64.getMimeEncoder()`: lines of at most /// 76 characters joined by a CRLF (`\r\n`), with no trailing separator. When false, the output is a /// single unwrapped line, matching `java.util.Base64.getMimeEncoder(-1, [])`. pub fn spark_base64(args: &[ColumnarValue]) -> Result { diff --git a/pom.xml b/pom.xml index a50d94543a2..9d138f44339 100644 --- a/pom.xml +++ b/pom.xml @@ -651,25 +651,6 @@ under the License. - - spark-3.4 - - 2.12.17 - 2.12 - 3.4.3 - 3.4 - 1.13.1 - 4.8.8 - 2.0.6 - spark-3.x - spark-3.4 - spark-none - 11 - ${java.version} - ${java.version} - - - spark-3.5 diff --git a/spark/pom.xml b/spark/pom.xml index 08f4f3752f2..e4ddbe79b57 100644 --- a/spark/pom.xml +++ b/spark/pom.xml @@ -210,32 +210,7 @@ under the License. - - - spark-3.4 - - - org.apache.iceberg - iceberg-spark-runtime-${spark.version.short}_${scala.binary.version} - 1.5.2 - test - - - - org.eclipse.jetty - jetty-server - 9.4.53.v20231009 - test - - - org.eclipse.jetty - jetty-servlet - 9.4.53.v20231009 - test - - - - + spark-3.5 diff --git a/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala b/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala index 4dc36194331..f15f76fa7d7 100644 --- a/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala +++ b/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala @@ -33,7 +33,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.comet.CometConf._ import org.apache.comet.iceberg.IcebergWriteStrategy -import org.apache.comet.rules.{CometExecRule, CometPlanAdaptiveDynamicPruningFilters, CometReuseSubquery, CometScanRule, CometSpark34AqeDppFallbackRule, EliminateRedundantTransitions, RevertNativeForTransitionHeavyStages} +import org.apache.comet.rules.{CometExecRule, CometPlanAdaptiveDynamicPruningFilters, CometReuseSubquery, CometScanRule, EliminateRedundantTransitions, RevertNativeForTransitionHeavyStages} import org.apache.comet.shims.ShimCometSparkSessionExtensions /** @@ -57,7 +57,7 @@ import org.apache.comet.shims.ShimCometSparkSessionExtensions * 5. ReuseExchangeAndSubquery -- Spark deduplicates subqueries (sees Comet nodes) * }}} * - * AQE (AdaptiveSparkPlanExec, Spark 3.5+): + * AQE (AdaptiveSparkPlanExec): * {{{ * Initial plan: * PlanAdaptiveSubqueries: creates SubqueryAdaptiveBroadcastExec (SAB) for AQE DPP @@ -79,11 +79,6 @@ import org.apache.comet.shims.ShimCometSparkSessionExtensions * c. postColumnarTransitions: RevertNativeForTransitionHeavyStages, * EliminateRedundantTransitions * }}} - * - * On Spark 3.4, injectQueryStageOptimizerRule is unavailable. CometExecRule does not wrap SABs, - * and CometPlanAdaptiveDynamicPruningFilters/CometReuseSubquery are not registered. AQE DPP scans - * fall back to Spark so that Spark's PlanAdaptiveDynamicPruningFilters handles them natively - * (with DPP). */ class CometSparkSessionExtensions extends (SparkSessionExtensions => Unit) @@ -92,10 +87,6 @@ class CometSparkSessionExtensions override def apply(extensions: SparkSessionExtensions): Unit = { extensions.injectColumnar { session => CometScanColumnar(session) } extensions.injectColumnar { session => CometExecColumnar(session) } - // Pre-3.5 only: tag AQE DPP regions so the conversion rules below leave them Spark-native. - // Registered before CometScanRule/CometExecRule so tags are in place when conversion runs. - // No-op on Spark 3.5+; see CometSpark34AqeDppFallbackRule's class docstring. - injectPreSpark35QueryStagePrepRuleShim(extensions, CometSpark34AqeDppFallbackRule) extensions.injectQueryStagePrepRule { session => CometScanRule(session) } extensions.injectQueryStagePrepRule { session => CometExecRule(session) } injectQueryStageOptimizerRuleShim(extensions, CometPlanAdaptiveDynamicPruningFilters) @@ -189,10 +180,6 @@ object CometSparkSessionExtensions extends Logging { op.isInstanceOf[CometBatchScanExec] || op.isInstanceOf[CometScanExec] } - def isSpark35Plus: Boolean = { - org.apache.spark.SPARK_VERSION >= "3.5" - } - def isSpark40Plus: Boolean = { org.apache.spark.SPARK_VERSION >= "4.0" } diff --git a/spark/src/main/scala/org/apache/comet/GenerateDocs.scala b/spark/src/main/scala/org/apache/comet/GenerateDocs.scala index 921e7b4f3e2..cc8b4168365 100644 --- a/spark/src/main/scala/org/apache/comet/GenerateDocs.scala +++ b/spark/src/main/scala/org/apache/comet/GenerateDocs.scala @@ -135,7 +135,7 @@ object GenerateDocs { * Args: * - args(0): user guide root directory (e.g. `docs/source/user-guide/latest/`). * - args(1) (optional): per-Spark-version subdirectory for compatibility pages (e.g. - * `spark-3.4`). When omitted, compat pages are written to `compatibility/expressions/` + * `spark-3.5`). When omitted, compat pages are written to `compatibility/expressions/` * directly, preserving the legacy flat layout used by released-version doc trees. */ def main(args: Array[String]): Unit = { diff --git a/spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala b/spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala index 078cb61f9da..92d2c78e9ff 100644 --- a/spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala +++ b/spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala @@ -60,8 +60,6 @@ object IcebergReflection extends Logging { val SPARK_WRITE = "org.apache.iceberg.spark.source.SparkWrite" val TABLE_PROPERTIES = "org.apache.iceberg.TableProperties" - // Iceberg 1.5.2 uses its own `ReplaceIcebergData` due to lack of `ReplaceData` in Spark 3.4. - val REPLACE_ICEBERG_DATA = "org.apache.spark.sql.catalyst.plans.logical.ReplaceIcebergData" } /** @@ -170,34 +168,6 @@ object IcebergReflection extends Logging { } } - def isReplaceIcebergData(plan: Any): Boolean = - plan != null && plan.getClass.getName == ClassNames.REPLACE_ICEBERG_DATA - - private def reflectField(plan: Any, fieldName: String): Option[AnyRef] = - try { - val field = plan.getClass.getDeclaredField(fieldName) - field.setAccessible(true) - Option(field.get(plan)) - } catch { - case e: Exception => - logError( - s"Iceberg reflection failure: $fieldName on ${plan.getClass.getName}: ${e.getMessage}") - None - } - - def extractReplaceIcebergDataFields(plan: Any): Option[(AnyRef, AnyRef, AnyRef, AnyRef)] = { - if (!isReplaceIcebergData(plan)) return None - for { - table <- reflectField(plan, "table") - query <- reflectField(plan, "query") - originalTable <- reflectField(plan, "originalTable") - write <- reflectField( - plan, - "write" - ) // Option[Write]; field can be Some(null) so kept AnyRef - } yield (table, query, originalTable, write) - } - /** * Loads a class using the thread context classloader first, then falls back to the system * classloader. diff --git a/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala b/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala index c501c900973..eca4a047115 100644 --- a/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala +++ b/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala @@ -52,17 +52,6 @@ case class IcebergWriteStrategy(session: SparkSession) extends SparkStrategy { rd.write, rd.query, replaceDataDispatch = IcebergReplaceDataShim.extractProjections(rd)).toList - case plan if IcebergReflection.isReplaceIcebergData(plan) => - IcebergReflection - .extractReplaceIcebergDataFields(plan) - .flatMap { case (_, query, originalTable, write) => - matchedSparkWrite( - originalTable.asInstanceOf[org.apache.spark.sql.catalyst.analysis.NamedRelation], - write.asInstanceOf[Option[Write]], - query.asInstanceOf[LogicalPlan], - replaceDataDispatch = None) - } - .toList // Hit by AQE. case l @ IcebergWriteLogical(child, batchWrite, replaceDataDispatch) => Seq(IcebergWriteExec(batchWrite, l.output, planLater(child), replaceDataDispatch)) diff --git a/spark/src/main/scala/org/apache/comet/iceberg/ReplaceDataDispatchInfo.scala b/spark/src/main/scala/org/apache/comet/iceberg/ReplaceDataDispatchInfo.scala index 8f9274a9e2b..0c802d975bc 100644 --- a/spark/src/main/scala/org/apache/comet/iceberg/ReplaceDataDispatchInfo.scala +++ b/spark/src/main/scala/org/apache/comet/iceberg/ReplaceDataDispatchInfo.scala @@ -22,9 +22,9 @@ package org.apache.comet.iceberg import org.apache.spark.sql.catalyst.ProjectingInternalRow /** - * Mirror of Spark 4.x's `ReplaceDataProjections`, defined here so we can compile against Spark - * 3.4 / 3.5 source trees (where the class doesn't exist). Populated by `IcebergReplaceDataShim` - * on 4.x and left `None` on 3.x; consumed by `IcebergWriteExec`. + * Mirror of Spark 4.x's `ReplaceDataProjections`, defined here so we can compile against the + * Spark 3.5 source tree (where the class doesn't exist). Populated by `IcebergReplaceDataShim` on + * 4.x and left `None` on 3.x; consumed by `IcebergWriteExec`. */ case class ReplaceDataDispatchInfo( rowProjection: ProjectingInternalRow, diff --git a/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala b/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala index cbf6c6a1e89..8c63292feed 100644 --- a/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala +++ b/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala @@ -108,13 +108,6 @@ object CometExecRule { val SKIP_COMET_SHUFFLE_TAG: org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit] = org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit]("comet.skipCometShuffle") - /** - * Tag set on a `BroadcastExchangeExec` that should be left as a plain Spark broadcast rather - * than converted to `CometBroadcastExchangeExec`. Written by [[CometSpark34AqeDppFallbackRule]] - * on Spark < 3.5. See that rule's class docstring for the rationale. - */ - val SKIP_COMET_BROADCAST_TAG: org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit] = - org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit]("comet.skipCometBroadcast") } /** @@ -317,11 +310,6 @@ case class CometExecRule(session: SparkSession) // broadcast exchange is forced to be enabled by Comet config. case plan if plan.children.exists(_.isInstanceOf[BroadcastExchangeExec]) => val newChildren = plan.children.map { - // Tagged by CometSpark34AqeDppFallbackRule on Spark < 3.5 to keep the build-side - // broadcast Spark-native so Spark's PlanAdaptiveDynamicPruningFilters can match it. - case b: BroadcastExchangeExec - if b.getTagValue(CometExecRule.SKIP_COMET_BROADCAST_TAG).isDefined => - b case b: BroadcastExchangeExec if b.children.forall(_.isInstanceOf[CometNativeExec]) => convertToComet(b, CometBroadcastExchangeExec).getOrElse(b) case other => other @@ -480,17 +468,13 @@ case class CometExecRule(session: SparkSession) } case _ => inSub } - case sab: SubqueryAdaptiveBroadcastExec if isSpark35Plus => + case sab: SubqueryAdaptiveBroadcastExec => // Wrap SABs to prevent Spark's PlanAdaptiveDynamicPruningFilters from // converting them to Literal.TrueLiteral. Spark's rule pattern-matches for // BroadcastHashJoinExec, which Comet replaced with CometBroadcastHashJoinExec. // Without wrapping, DPP is disabled for both Comet native scans and non-Comet // scans (e.g., V2 BatchScan). CometPlanAdaptiveDynamicPruningFilters - // (queryStageOptimizerRule, 3.5+) unwraps and converts them later. - // - // On Spark 3.4, injectQueryStageOptimizerRule is unavailable. The isSpark35Plus - // guard leaves SABs unwrapped; CometSpark34AqeDppFallbackRule then tags the - // matching BHJ's build broadcast so Spark's rule can match it natively. + // (queryStageOptimizerRule) unwraps and converts them later. assert( sab.buildKeys.nonEmpty, s"SubqueryAdaptiveBroadcastExec '${sab.name}' has empty buildKeys") diff --git a/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala b/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala index 2c15bb5e4e7..0bf7d45d098 100644 --- a/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala +++ b/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala @@ -31,7 +31,7 @@ import scala.jdk.CollectionConverters._ import org.apache.hadoop.conf.Configuration import org.apache.spark.internal.Logging import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.catalyst.expressions.{Attribute, DynamicPruningExpression, Expression, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName, PlanExpression} +import org.apache.spark.sql.catalyst.expressions.{Attribute, DynamicPruningExpression, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName} import org.apache.spark.sql.catalyst.rules.Rule import org.apache.spark.sql.catalyst.util.{sideBySide, ArrayBasedMapData, GenericArrayData, MetadataColumnHelper} import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues @@ -45,7 +45,7 @@ import org.apache.spark.sql.types._ import org.apache.comet.{CometConf, DataTypeSupport, NativeBase} import org.apache.comet.CometConf._ -import org.apache.comet.CometSparkSessionExtensions.{isCometLoaded, isSpark35Plus, withFallbackReason, withFallbackReasons} +import org.apache.comet.CometSparkSessionExtensions.{isCometLoaded, withFallbackReason, withFallbackReasons} import org.apache.comet.DataTypeSupport.isComplexType import org.apache.comet.iceberg.{CometIcebergNativeScanMetadata, IcebergReflection} import org.apache.comet.objectstore.NativeConfig @@ -116,11 +116,6 @@ case class CometScanRule(session: SparkSession) val fullPlan = plan def transformScan(scanNode: SparkPlan): SparkPlan = scanNode match { - // Tagged by CometSpark34AqeDppFallbackRule on Spark < 3.5 to keep a peer scan - // Spark-native for canonical symmetry in SMJ self-joins (SPARK-32509). - case scan if scan.getTagValue(CometScanRule.SKIP_COMET_SCAN_TAG).isDefined => - withFallbackReason(scan, "AQE DPP region fallback (Spark < 3.5)") - case scan if !CometConf.COMET_NATIVE_SCAN_ENABLED.get(conf) => withFallbackReason(scan, "Comet Scan is not enabled") @@ -157,19 +152,6 @@ case class CometScanRule(session: SparkSession) s"Metadata column(s) ${unsupportedMetadataColNames.mkString(", ")} is not supported") } - // On Spark 3.4, injectQueryStageOptimizerRule is unavailable, so - // CometPlanAdaptiveDynamicPruningFilters cannot run. Fall back this scan to Spark so that - // Spark's PlanAdaptiveDynamicPruningFilters handles the SAB natively. Comet's narrower - // CometSpark34AqeDppFallbackRule (queryStagePrepRule on 3.4) then tags any matching BHJ's - // build-side broadcast so Spark's rule can match it via sameResult. See - // CometSpark34AqeDppFallbackRule's class docstring. - // - // On 3.5+, CometPlanAdaptiveDynamicPruningFilters rewrites SABs directly and this fallback - // is not needed. - if (!isSpark35Plus && scanExec.partitionFilters.exists(isAqeDynamicPruningFilter)) { - return withFallbackReason(scanExec, "AQE Dynamic Partition Pruning requires Spark 3.5+") - } - scanExec.relation match { case r: HadoopFsRelation => if (!CometScanExec.isFileFormatSupported(r.fileFormat)) { @@ -857,24 +839,6 @@ case class CometScanRule(session: SparkSession) } } - private def isDynamicPruningFilter(e: Expression): Boolean = - e.exists(_.isInstanceOf[PlanExpression[_]]) - - /** - * Detects AQE DPP (SubqueryAdaptiveBroadcastExec), as opposed to non-AQE DPP. - * - * Non-AQE DPP (PlanDynamicPruningFilters) runs before Comet rules and produces - * SubqueryBroadcastExec/SubqueryExec which Spark's execution framework resolves. AQE DPP - * (PlanAdaptiveDynamicPruningFilters) runs after Comet rules and searches for - * BroadcastHashJoinExec. It doesn't recognize Comet operators, so it can't create DPP filters - * correctly. - */ - private def isAqeDynamicPruningFilter(e: Expression): Boolean = - e.exists { - case sub: InSubqueryExec => sub.plan.isInstanceOf[SubqueryAdaptiveBroadcastExec] - case _ => false - } - private def metadataCols(plan: SparkPlan): Seq[String] = { plan.expressions.collect { case a: Attribute if a.isMetadataCol => a.name @@ -966,15 +930,6 @@ object CometScanRule extends Logging { .booleanValue() } - /** - * Tag set on a scan (`FileSourceScanExec` or `BatchScanExec`) that should be left as a plain - * Spark scan rather than converted to a Comet scan. Written by - * [[CometSpark34AqeDppFallbackRule]] on Spark < 3.5. See that rule's class docstring for the - * rationale. - */ - val SKIP_COMET_SCAN_TAG: org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit] = - org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit]("comet.skipCometScan") - /** * Single-pass validation of Iceberg FileScanTasks. * diff --git a/spark/src/main/scala/org/apache/comet/rules/CometSpark34AqeDppFallbackRule.scala b/spark/src/main/scala/org/apache/comet/rules/CometSpark34AqeDppFallbackRule.scala deleted file mode 100644 index afedc1601c9..00000000000 --- a/spark/src/main/scala/org/apache/comet/rules/CometSpark34AqeDppFallbackRule.scala +++ /dev/null @@ -1,392 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.rules - -import org.apache.spark.internal.Logging -import org.apache.spark.sql.catalyst.expressions.DynamicPruningExpression -import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight} -import org.apache.spark.sql.catalyst.rules.Rule -import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, SparkPlan, SubqueryAdaptiveBroadcastExec, SubqueryBroadcastExec} -import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper -import org.apache.spark.sql.execution.datasources.HadoopFsRelation -import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec, ShuffleExchangeExec} -import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec - -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus - -/** - * Preserves AQE Dynamic Partition Pruning on Spark < 3.5 by tagging specific nodes to stay - * Spark-native, so that Spark's PlanAdaptiveDynamicPruningFilters can match them natively. - * - * On Spark 3.5+, CometPlanAdaptiveDynamicPruningFilters (queryStageOptimizerRule) rewrites AQE - * DPP SABs into CometSubqueryBroadcastExec after Spark's own rule has run. Spark 3.4 does not - * expose injectQueryStageOptimizerRule (SPARK-45785 added it in 3.5), so the rewrite cannot run - * at the correct time. Rewriting the SAB at queryStagePrepRule time does not work either: AQE - * rebuilds plan nodes between prep and execution in ways that drop the `@transient` inner scan we - * would need to update. See the dual-filter handling in - * CometPlanAdaptiveDynamicPruningFilters.convertNativeScanDPP for why both filters need updating. - * - * Instead, on 3.4 we arrange for Spark's PlanAdaptiveDynamicPruningFilters to succeed on its own - * by keeping the BHJ's build-side exchange (or the peer branch of a self-join SMJ) Spark-native. - * This rule only writes skip-tags on nodes; it never rewrites expressions or plan structure. Tags - * survive AQE per-stage re-entry, matching the contract of SKIP_COMET_SHUFFLE_TAG from PR #4010. - * - * Registered via injectPreSpark35QueryStagePrepRuleShim before CometScanRule/CometExecRule in - * CometSparkSessionExtensions, so tags are in place when conversion runs. No-op on Spark 3.5+. - * - * Four cases handled: - * - * 1. SAB + matching BHJ (non-V1 fact scans: Hive / V2 / V2Filter). The cascade up from a non-V1 - * scan reaches a CometBroadcastHashJoinExec + CometBroadcastExchangeExec build side; Spark's - * class-sensitive sameResult check in PlanAdaptiveDynamicPruningFilters.scala:50-57 fails to - * match. Tag the BHJ's build-side BroadcastExchangeExec with SKIP_COMET_BROADCAST_TAG. With - * the build exchange Spark-native, Comet's BHJ conversion fails its - * forall(_.isInstanceOf[CometNativeExec]) guard and the BHJ stays Spark. Spark's rule then - * matches and creates SubqueryBroadcastExec. - * - * 2. SAB + matching BHJ on V1. CometScanRule.transformV1Scan already rejects the V1 fact scan via - * isAqeDynamicPruningFilter; the cascade keeps the BHJ and its BroadcastExchangeExec - * Spark-native. No tagging needed; this rule is a no-op for V1 BHJ. TPC-DS Q7 on 3.4 stays the - * same shape it has today, including the Comet acceleration on dim scans below the Spark - * broadcast. - * - * 3. SAB with no matching BHJ (V1 SMJ self-join, SPARK-32509 with - * AUTO_BROADCASTJOIN_THRESHOLD=-1). The logical Partition-Pruning rule attaches the SAB to only - * one branch of the self-join. transformV1Scan falls back the SAB-bearing scan; the peer scan (no - * SAB) Cometizes, producing canonical asymmetry that breaks shuffle exchange reuse. Tag the peer - * scan with SKIP_COMET_SCAN_TAG and any shuffle whose subtree contains the peer scan with - * SKIP_COMET_SHUFFLE_TAG. Both branches end up Spark-native with matching canonical forms; - * Spark's rule replaces the SAB with TrueLiteral, and FileSourceScanExec.doCanonicalize strips it - * via filterUnusedDynamicPruningExpressions (DataSourceScanExec.scala:731,736), restoring - * canonical symmetry for reuse. - * - * 4. SubqueryBroadcastExec-bearing scans (AQE re-optimize). On re-optimize cycles, - * ASPE.preprocessingRules (PlanAdaptiveSubqueries) fills the DPP slot with the already- - * materialized SubqueryBroadcastExec (produced by Spark's PlanAdaptiveDynamicPruningFilters on a - * previous pass) rather than the original SAB. The freshly-planned BroadcastExchangeExec on the - * main BHJ's build side is a new instance with no SKIP_COMET_BROADCAST_TAG carried over, so - * CometExecRule would Cometize it and lose AQE stageCache broadcast reuse with the DPP subquery's - * Spark broadcast. The rule also scans for SubqueryBroadcastExec (descending into QueryStageExec - * via AdaptiveSparkPlanHelper since the fact scan is already inside a materialized stage), - * extracts its buildKeys, finds the matching BHJ by exprId, and tags the build BE. - * - * Non-AQE DPP (#4011) is untouched: it produces SubqueryBroadcastExec, not the adaptive variant, - * and is handled by CometExecRule.convertSubqueryBroadcasts. - * - * Known limitation on 3.4: cross-plan scalar-subquery DPP. An SAB in a scalar subquery cannot see - * a matching BHJ in the main query because at prep-rule time each AdaptiveSparkPlanExec sees only - * its own plan. When the match fails, Spark's own rule falls back to TrueLiteral or aggregate - * SubqueryExec (same behavior as Spark-without-Comet on 3.4). - * - * @see - * PlanAdaptiveDynamicPruningFilters (Spark's rule this code arranges to succeed) - * @see - * CometPlanAdaptiveDynamicPruningFilters (Spark 3.5+ equivalent via queryStageOptimizerRule) - */ -case object CometSpark34AqeDppFallbackRule - extends Rule[SparkPlan] - with AdaptiveSparkPlanHelper - with Logging { - - override def apply(plan: SparkPlan): SparkPlan = { - // Registered only on Spark < 3.5 via injectPreSpark35QueryStagePrepRuleShim. If the - // 3.5+ shim mis-registers this rule, fail loud rather than silently disable Comet. - assert( - !isSpark35Plus, - "CometSpark34AqeDppFallbackRule must only be registered on Spark < 3.5; " + - "see ShimCometSparkSessionExtensions.injectPreSpark35QueryStagePrepRuleShim") - - if (!conf.dynamicPartitionPruningEnabled) return plan - - val sabScans = findSabScans(plan) - val sbScans = findSubqueryBroadcastScans(plan) - if (sabScans.isEmpty && sbScans.isEmpty) return plan - - sabScans.foreach { case (scan, sab) => - tagForSab(plan, scan, sab) - } - - // AQE re-optimization path: see case 4 in the class-level docstring. On subsequent - // re-optimize cycles Spark's PlanAdaptiveSubqueries hands us a SubqueryBroadcastExec (not - // the original SAB), and the freshly-planned main-BHJ build BE has no tag carried over. - sbScans.foreach { case (scan, sb) => - tagForSubqueryBroadcast(plan, scan, sb) - } - - // This rule only tags; it never rewrites the plan structurally. - plan - } - - /** - * Find every scan whose `partitionFilters` contain a `SubqueryAdaptiveBroadcastExec`. - * - * Mirrors the SAB pattern matched by Spark's `PlanAdaptiveDynamicPruningFilters.apply` (lines - * 41-43): - * {{{ - * case DynamicPruningExpression(InSubqueryExec( - * value, SubqueryAdaptiveBroadcastExec(...), ...)) - * }}} - * - * Returns the scan node itself and the SAB found on it. - */ - private def findSabScans(plan: SparkPlan): Seq[(SparkPlan, SubqueryAdaptiveBroadcastExec)] = { - val buf = scala.collection.mutable.ArrayBuffer[(SparkPlan, SubqueryAdaptiveBroadcastExec)]() - foreach(plan) { node => - extractFirstSab(node).foreach(sab => buf += ((node, sab))) - } - buf.toSeq - } - - private def extractFirstSab(node: SparkPlan): Option[SubqueryAdaptiveBroadcastExec] = { - node.expressions - .flatMap(_.collect { - case DynamicPruningExpression(inSub: InSubqueryExec) - if inSub.plan.isInstanceOf[SubqueryAdaptiveBroadcastExec] => - inSub.plan.asInstanceOf[SubqueryAdaptiveBroadcastExec] - }) - .headOption - } - - /** - * Find every scan whose DPP `partitionFilters` contain a `SubqueryBroadcastExec` - i.e. a DPP - * subquery that Spark's `PlanAdaptiveDynamicPruningFilters` has already materialized on a - * previous AQE pass. See the comment block in `apply` for why we need this in addition to the - * SAB path. - * - * Uses `AdaptiveSparkPlanHelper.foreach`, which descends through `QueryStageExec.plan` and - * `AdaptiveSparkPlanExec.executedPlan`. By the re-optimize pass where this matters, the fact - * scan is inside a `ShuffleQueryStageExec` whose `children` is `Seq.empty`, so a plain - * `plan.foreach` would stop there and miss the scan's `SubqueryBroadcastExec`. - */ - private def findSubqueryBroadcastScans( - plan: SparkPlan): Seq[(SparkPlan, SubqueryBroadcastExec)] = { - val buf = scala.collection.mutable.ArrayBuffer[(SparkPlan, SubqueryBroadcastExec)]() - foreach(plan) { node => - extractFirstSubqueryBroadcast(node).foreach(sb => buf += ((node, sb))) - } - buf.toSeq - } - - private def extractFirstSubqueryBroadcast(node: SparkPlan): Option[SubqueryBroadcastExec] = { - node.expressions - .flatMap(_.collect { - case DynamicPruningExpression(inSub: InSubqueryExec) - if inSub.plan.isInstanceOf[SubqueryBroadcastExec] => - inSub.plan.asInstanceOf[SubqueryBroadcastExec] - }) - .headOption - } - - /** - * Place tags for a single SAB-bearing scan. Behavior depends on whether a matching - * broadcast-hash join exists in the plan: - * - Matching BHJ found: tag its build-side `BroadcastExchangeExec` (case 1 above). - * - No matching BHJ: tag peer scans + their shuffles for canonical symmetry (case 3). - */ - private def tagForSab( - plan: SparkPlan, - scan: SparkPlan, - sab: SubqueryAdaptiveBroadcastExec): Unit = { - val sabKeyIds: Set[Any] = sab.buildKeys.flatMap(_.references.map(_.exprId)).toSet - if (sabKeyIds.isEmpty) { - logWarning(s"SAB '${sab.name}' has empty buildKeys; skipping") - return - } - - findMatchingBroadcastJoin(plan, sabKeyIds) match { - case Some(buildSide) => - tagBhjBuildBroadcast(buildSide, sab.name) - case None => - tagPeerScansAndShuffles(plan, scan, sab.name) - } - } - - /** - * Tag the matching BHJ's build-side `BroadcastExchangeExec` for a scan whose DPP filter holds a - * `SubqueryBroadcastExec` (post-PADPF form). Only the matching-BHJ case applies here: if PADPF - * already ran on a previous AQE cycle, it would have fallen back to aggregate `SubqueryExec` or - * `Literal.TrueLiteral` instead of producing a `SubqueryBroadcastExec` when no BHJ matched - so - * a `SubqueryBroadcastExec` always implies a BHJ with compatible build keys somewhere in the - * plan. No peer-scan tagging path is needed because the SMJ self-join case (SPARK-32509) never - * produces a `SubqueryBroadcastExec`. - */ - private def tagForSubqueryBroadcast( - plan: SparkPlan, - scan: SparkPlan, - sb: SubqueryBroadcastExec): Unit = { - val keyIds: Set[Any] = sb.buildKeys.flatMap(_.references.map(_.exprId)).toSet - if (keyIds.isEmpty) { - logWarning(s"SubqueryBroadcast '${sb.name}' has empty buildKeys; skipping") - return - } - - findMatchingBroadcastJoin(plan, keyIds) match { - case Some(buildSide) => - tagBhjBuildBroadcast(buildSide, sb.name) - case None => - // Nothing to tag. Either the BHJ has a different buildKey exprId set (e.g. AQE - // rewrote attributes across stages) or the matching join isn't in this plan snapshot. - logDebug( - s"SubqueryBroadcast '${sb.name}': no matching BHJ on this plan snapshot; no tag placed") - } - } - - /** - * Walk from a BHJ's build-side subtree root to the first `BroadcastExchangeExec` and tag it - * with `SKIP_COMET_BROADCAST_TAG`. If the subtree root already IS a BroadcastExchangeExec (most - * common), that's the one. Otherwise we walk until we find one or give up. - * - * Tagging the exchange is enough: when it stays Spark-native, `CometBroadcastHashJoinExec`'s - * conversion guard (`forall(_.isInstanceOf[CometNativeExec])`) fails because a Spark - * `BroadcastExchangeExec` is not a `CometNativeExec`, so the BHJ stays Spark too. Spark's - * `PlanAdaptiveDynamicPruningFilters` can then match it via `sameResult` - * (`PlanAdaptiveDynamicPruningFilters.scala:50-57`) and create a `SubqueryBroadcastExec`. - */ - private def tagBhjBuildBroadcast(buildSide: SparkPlan, dppName: String): Unit = { - val found = buildSide.find { - case _: BroadcastExchangeExec => true - case _ => false - } - found match { - case Some(be: BroadcastExchangeExec) => - be.setTagValue(CometExecRule.SKIP_COMET_BROADCAST_TAG, ()) - logDebug(s"Tagged BroadcastExchangeExec for DPP '$dppName' (BHJ build side)") - case _ => - logWarning( - s"DPP '$dppName': matched BHJ but could not locate BroadcastExchangeExec on " + - "build side; skipping") - } - } - - /** - * For SMJ-shaped DPP (SPARK-32509 with AUTO_BROADCASTJOIN_THRESHOLD=-1), there is no BHJ. The - * logical Partition-Pruning rule attaches the SAB to only one branch of a self-join; the peer - * branch has no SAB, so `transformV1Scan`'s V1 AQE DPP fallback fires for the SAB-bearing scan - * only. Peer scans Cometize normally, leading to canonical asymmetry that breaks shuffle - * exchange reuse (0 `ReusedExchangeExec` instead of 1). - * - * Tag every peer scan (same relation as the SAB-bearing scan, different instance) with - * `SKIP_COMET_SCAN_TAG`, and every `ShuffleExchangeExec` whose subtree contains a peer scan - * with `SKIP_COMET_SHUFFLE_TAG`. Both branches end up Spark-native with matching canonical - * forms. - * - * For canonical equality the SAB-bearing scan's side is already handled: - * `FileSourceScanExec.doCanonicalize` strips `DynamicPruningExpression(Literal.TrueLiteral)` - * via `filterUnusedDynamicPruningExpressions` (`DataSourceScanExec.scala:731,736`), and Spark's - * rule replaces the SAB with `TrueLiteral` when it can't match. - */ - private def tagPeerScansAndShuffles( - plan: SparkPlan, - sabScan: SparkPlan, - sabName: String): Unit = { - val sabRelation = sabScan match { - case f: FileSourceScanExec => Some(f.relation) - case _ => None - } - if (sabRelation.isEmpty) { - logDebug( - s"SAB '$sabName': non-V1 scan with no BHJ match; no peer-tagging heuristic available") - return - } - val sabRel = sabRelation.get - - var taggedScans = 0 - foreach(plan) { - case peer: FileSourceScanExec if (peer ne sabScan) && sameRelation(peer.relation, sabRel) => - peer.setTagValue(CometScanRule.SKIP_COMET_SCAN_TAG, ()) - taggedScans += 1 - case _ => - } - - var taggedShuffles = 0 - foreach(plan) { - case sh: ShuffleExchangeExec if shuffleSubtreeContainsMatchingScan(sh, sabRel, sabScan) => - sh.setTagValue(CometExecRule.SKIP_COMET_SHUFFLE_TAG, ()) - taggedShuffles += 1 - case _ => - } - - logDebug( - s"SAB '$sabName' (no BHJ match): tagged $taggedScans peer scan(s) + " + - s"$taggedShuffles shuffle(s)") - } - - private def sameRelation(a: HadoopFsRelation, b: HadoopFsRelation): Boolean = { - (a eq b) || - (a.location.rootPaths == b.location.rootPaths && - a.dataSchema == b.dataSchema && - a.partitionSchema == b.partitionSchema) - } - - private def shuffleSubtreeContainsMatchingScan( - shuffle: ShuffleExchangeExec, - sabRelation: HadoopFsRelation, - sabScan: SparkPlan): Boolean = { - find(shuffle.child) { - case scan: FileSourceScanExec => - (scan eq sabScan) || sameRelation(scan.relation, sabRelation) - case _ => false - }.isDefined - } - - /** - * Mirrors `PlanAdaptiveDynamicPruningFilters.apply` lines 50-57: - * {{{ - * find(rootPlan) { - * case BroadcastHashJoinExec(_, _, _, BuildLeft, _, left, _, _) => - * left.sameResult(exchange) - * case BroadcastHashJoinExec(_, _, _, BuildRight, _, _, right, _) => - * right.sameResult(exchange) - * } - * }}} - * - * Our rule runs BEFORE `CometScanRule`/`CometExecRule`, so the plan is entirely Spark-native at - * this point. We only match `BroadcastHashJoinExec`. Instead of `sameResult` we match on - * join-side exprId equality with the SAB's buildKeys; this is semantically equivalent because - * SAB buildKeys originate from the same logical plan as the join's build-side keys. - * - * Returns the BHJ's build-side subtree (the one we want to keep Spark-native), or None if no - * matching join is found. - */ - private def findMatchingBroadcastJoin( - plan: SparkPlan, - sabKeyIds: Set[Any]): Option[SparkPlan] = { - var result: Option[SparkPlan] = None - find(plan) { - case j: BroadcastHashJoinExec => - val joinBuildKeys = j.buildSide match { - case BuildLeft => j.leftKeys - case BuildRight => j.rightKeys - } - val joinKeyIds: Set[Any] = joinBuildKeys.flatMap(_.references.map(_.exprId)).toSet - if (sabKeyIds == joinKeyIds) { - result = Some(j.buildSide match { - case BuildLeft => j.left - case BuildRight => j.right - }) - true - } else { - false - } - case _ => false - } - result - } -} diff --git a/spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala b/spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala index ec277cfc7bc..025a7c80c0a 100644 --- a/spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala +++ b/spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala @@ -108,7 +108,7 @@ case class EliminateRedundantTransitions(session: SparkSession) // Replace MapInBatchExec (PythonMapInArrowExec / MapInArrowExec / MapInPandasExec) that has // a ColumnarToRow child with CometMapInBatchExec, eliminating the input and output // UnsafeProjection copies and keeping the stage columnar. The matchers are - // version-shimmed: Spark 3.4 / 3.5 return None (they lack the required APIs) and Spark + // version-shimmed: Spark 3.5 returns None (it lacks the required APIs) and Spark // 4.1+ matches the renamed `MapInArrowExec`. // // Falls back to vanilla Spark when `spark.sql.execution.arrow.useLargeVarTypes` is enabled: diff --git a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala index 85601a9e0c9..f3196a7bbd7 100644 --- a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala +++ b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala @@ -55,7 +55,7 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim { private[comet] val arrayExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map( // ArrayAppend is a concrete expression only on Spark 3.x. Spark 4.0+ marks it // RuntimeReplaceable and rewrites it to ArrayInsert before serde, so this entry is - // unreachable there and is kept solely for Spark 3.4/3.5. + // unreachable there and is kept solely for Spark 3.5. classOf[ArrayAppend] -> CometArrayAppend, // ArrayCompact is RuntimeReplaceable in all supported Spark versions (rewritten to // ArrayFilter(arr, IsNotNull(...))), so it never reaches serde directly; dispatch flows @@ -260,7 +260,6 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim { classOf[Overlay] -> CometOverlay, classOf[SoundEx] -> CometSoundEx, classOf[StringLocate] -> CometStringLocate, - classOf[Base64] -> CometBase64, classOf[UnBase64] -> CometUnBase64, classOf[ToCharacter] -> CometToCharacter, classOf[ToNumber] -> CometToNumber, diff --git a/spark/src/main/scala/org/apache/comet/serde/aggregates.scala b/spark/src/main/scala/org/apache/comet/serde/aggregates.scala index db435e5b5d5..651f2ac5335 100644 --- a/spark/src/main/scala/org/apache/comet/serde/aggregates.scala +++ b/spark/src/main/scala/org/apache/comet/serde/aggregates.scala @@ -839,8 +839,8 @@ object CometCollectSet extends CometAggregateExpressionSerde[CollectSet] { override def getSupportLevel(expr: CollectSet): SupportLevel = { // The native path always drops null inputs. Spark 4.2 added an `ignoreNulls` field to // CollectSet that `RESPECT NULLS` sets to false, preserving nulls in the result; Comet - // cannot match that, so fall back. This branch is only reachable on Spark 4.2+: on 3.4 - // through 4.1 the field does not exist, `RESPECT NULLS`/`IGNORE NULLS` are rejected at + // cannot match that, so fall back. This branch is only reachable on Spark 4.2+: through + // Spark 4.1 the field does not exist, `RESPECT NULLS`/`IGNORE NULLS` are rejected at // analysis time, and CometCollectShim.ignoreNulls hardcodes true, making this a no-op. if (!CometCollectShim.ignoreNulls(expr)) { Unsupported(Some("collect_set with RESPECT NULLS (ignoreNulls = false) is not supported")) @@ -890,7 +890,7 @@ object CometCollectList extends CometAggregateExpressionSerde[CollectList] { // The native path delegates to SparkCollectList, which always drops null inputs. Spark 4.2 // added an `ignoreNulls` field to CollectList that `RESPECT NULLS` sets to false, preserving // nulls in the result; Comet cannot match that, so fall back. This branch is only reachable - // on Spark 4.2+: on 3.4 through 4.1 the field does not exist, `RESPECT NULLS`/`IGNORE NULLS` + // on Spark 4.2+: through Spark 4.1 the field does not exist, `RESPECT NULLS`/`IGNORE NULLS` // are rejected at analysis time, and CometCollectShim.ignoreNulls hardcodes true, making this // a no-op. if (!CometCollectShim.ignoreNulls(expr)) { diff --git a/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala b/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala index 0240bd4a076..def2286db80 100644 --- a/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala +++ b/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala @@ -23,17 +23,17 @@ import scala.collection.mutable.ListBuffer import scala.jdk.CollectionConverters._ import org.apache.spark.internal.Logging -import org.apache.spark.sql.catalyst.expressions.{Expression, Literal} +import org.apache.spark.sql.catalyst.expressions.Literal import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues import org.apache.spark.sql.comet.{CometNativeExec, CometNativeScanExec, CometScanExec} -import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, SubqueryAdaptiveBroadcastExec} +import org.apache.spark.sql.execution.FileSourceScanExec import org.apache.spark.sql.execution.datasources.parquet.ParquetUtils import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.StructField import org.apache.comet.{CometConf, ConfigEntry} import org.apache.comet.CometConf.COMET_EXEC_ENABLED -import org.apache.comet.CometSparkSessionExtensions.{hasFallbackReason, isSpark35Plus, isSpark41Plus, withFallbackReason} +import org.apache.comet.CometSparkSessionExtensions.{hasFallbackReason, isSpark41Plus, withFallbackReason} import org.apache.comet.objectstore.NativeConfig import org.apache.comet.parquet.CometParquetUtils import org.apache.comet.serde.{CometOperatorSerde, Compatible, OperatorOuterClass, SupportLevel} @@ -65,17 +65,10 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with Logging { } // AQE DPP (SubqueryAdaptiveBroadcastExec) is converted to CometSubqueryBroadcastExec - // by CometPlanAdaptiveDynamicPruningFilters (queryStageOptimizerRule, Spark 3.5+). + // by CometPlanAdaptiveDynamicPruningFilters (queryStageOptimizerRule). // Non-AQE DPP (SubqueryBroadcastExec/SubqueryExec) is converted by // CometExecRule.convertSubqueryBroadcasts. Both are resolved through the lazy // partition serialization path in CometNativeScanExec. - // - // On Spark 3.4, injectQueryStageOptimizerRule is unavailable, so the AQE DPP conversion - // rule can't run. CometScanRule.transformV1Scan rejects AQE DPP on 3.4, so this check - // is a safety net: if the scan somehow reached here with AQE DPP on 3.4, reject it. - if (!isSpark35Plus && scanExec.partitionFilters.exists(isAqeDynamicPruningFilter)) { - withFallbackReason(scanExec, "Native DataFusion scan does not support AQE DPP on Spark 3.4") - } if (SQLConf.get.ignoreCorruptFiles || scanExec.relation.options @@ -96,13 +89,6 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with Logging { !hasFallbackReason(scanExec) } - /** Detects AQE DPP (SubqueryAdaptiveBroadcastExec), as opposed to non-AQE DPP. */ - private def isAqeDynamicPruningFilter(e: Expression): Boolean = - e.exists { - case sub: InSubqueryExec => sub.plan.isInstanceOf[SubqueryAdaptiveBroadcastExec] - case _ => false - } - override def enabledConfig: Option[ConfigEntry[Boolean]] = None override def getSupportLevel(operator: CometScanExec): SupportLevel = { diff --git a/spark/src/main/scala/org/apache/comet/serde/predicates.scala b/spark/src/main/scala/org/apache/comet/serde/predicates.scala index 0e7bb02cd68..596b6c02cb6 100644 --- a/spark/src/main/scala/org/apache/comet/serde/predicates.scala +++ b/spark/src/main/scala/org/apache/comet/serde/predicates.scala @@ -26,7 +26,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.BooleanType import org.apache.comet.CometConf -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus import org.apache.comet.serde.ExprOuterClass.Expr import org.apache.comet.serde.QueryPlanSerde._ @@ -342,14 +342,14 @@ object ComparisonUtils { s"effect (`$legacyNullInEmptyListConfig`), because a `NULL` operand then evaluates to " + "`NULL` rather than `false`." - // Spark 3.4 has no config and always uses the legacy behavior, Spark 3.5 defaults the config to - // true, and Spark 4.0+ makes it optional with a default of `!spark.sql.ansi.enabled`. Read by - // string key so this compiles against every supported Spark version. + // Spark 3.5 defaults the config to true, and Spark 4.0+ makes it optional with a default of + // `!spark.sql.ansi.enabled`. Read by string key so this compiles against every supported Spark + // version. private def legacyNullInEmptyListBehavior: Boolean = - !isSpark35Plus || { - val default = !isSpark40Plus || !SQLConf.get.ansiEnabled - CometConf.getBooleanConf(legacyNullInEmptyListConfig, default, SQLConf.get) - } + CometConf.getBooleanConf( + legacyNullInEmptyListConfig, + !isSpark40Plus || !SQLConf.get.ansiEnabled, + SQLConf.get) /** * Support level shared by the `In` and `InSet` serdes: `list` is the set of values being tested diff --git a/spark/src/main/scala/org/apache/comet/serde/statics.scala b/spark/src/main/scala/org/apache/comet/serde/statics.scala index 6501a1d46cb..6146676b7a2 100644 --- a/spark/src/main/scala/org/apache/comet/serde/statics.scala +++ b/spark/src/main/scala/org/apache/comet/serde/statics.scala @@ -29,8 +29,7 @@ import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, scalarFunctio object CometStaticInvoke extends CometExpressionSerde[StaticInvoke] { - // With Spark 3.4, CharVarcharCodegenUtils.readSidePadding gets called to pad spaces for - // char types. + // CharVarcharCodegenUtils.readSidePadding gets called to pad spaces for char types. // See https://github.com/apache/spark/pull/38151 private val staticInvokeExpressions : Map[(String, Class[_]), CometExpressionSerde[StaticInvoke]] = @@ -46,9 +45,8 @@ object CometStaticInvoke extends CometExpressionSerde[StaticInvoke] { // carrying the `legacyCharsets` / `legacyErrorAction` flags. Routing through the codegen // dispatcher runs Spark's own decoder so both flags are honored. See #4465. ("decode", classOf[StringDecode]) -> CometStaticInvokeCodegenDispatch, - // Spark 3.5+ makes `Base64` RuntimeReplaceable, lowering `base64(bin)` to - // `StaticInvoke(Base64.encode, Seq(child, chunkBase64), ...)`. On Spark 3.4 the `Base64` - // node survives and is handled directly (see CometBase64). + // `Base64` is RuntimeReplaceable, lowering `base64(bin)` to + // `StaticInvoke(Base64.encode, Seq(child, chunkBase64), ...)`. ("encode", classOf[Base64]) -> CometBase64StaticInvoke) override def convert( diff --git a/spark/src/main/scala/org/apache/comet/serde/strings.scala b/spark/src/main/scala/org/apache/comet/serde/strings.scala index ebf45089882..3b2d2c1a78f 100644 --- a/spark/src/main/scala/org/apache/comet/serde/strings.scala +++ b/spark/src/main/scala/org/apache/comet/serde/strings.scala @@ -19,7 +19,7 @@ package org.apache.comet.serde -import org.apache.spark.sql.catalyst.expressions.{Attribute, Base64, BitLength, Cast, Concat, ConcatWs, Contains, Elt, Empty2Null, EndsWith, Expression, FindInSet, FormatNumber, FormatString, GetJsonObject, InitCap, Left, Length, Levenshtein, Like, Literal, Lower, Mask, OctetLength, Overlay, RegExpExtract, RegExpExtractAll, RegExpInStr, RegExpReplace, Right, RLike, SoundEx, StartsWith, StringLocate, StringLPad, StringRepeat, StringReplace, StringRPad, StringSplit, StringTranslate, Substring, SubstringIndex, ToCharacter, ToNumber, TryToNumber, UnBase64, Upper} +import org.apache.spark.sql.catalyst.expressions.{Attribute, BitLength, Cast, Concat, ConcatWs, Contains, Elt, Empty2Null, EndsWith, Expression, FindInSet, FormatNumber, FormatString, GetJsonObject, InitCap, Left, Length, Levenshtein, Like, Literal, Lower, Mask, OctetLength, Overlay, RegExpExtract, RegExpExtractAll, RegExpInStr, RegExpReplace, Right, RLike, SoundEx, StartsWith, StringLocate, StringLPad, StringRepeat, StringReplace, StringRPad, StringSplit, StringTranslate, Substring, SubstringIndex, ToCharacter, ToNumber, TryToNumber, UnBase64, Upper} import org.apache.spark.sql.types.{BinaryType, DataTypes, IntegerType, LongType, StringType} import org.apache.comet.CometConf @@ -71,7 +71,7 @@ class CometCaseConversionBase[T <: Expression](function: String) super.convert(expr, inputs, binding) } else { // Default: route through the codegen dispatcher so Spark's own doGenCode runs inside the - // Comet pipeline. This guarantees Spark-compatible behavior across 3.4 / 3.5 / 4.0. + // Comet pipeline. This guarantees Spark-compatible behavior across supported versions. // Falls through to Spark when the dispatcher is disabled. CometScalaUDF.emitJvmCodegenDispatch(expr, inputs, binding) } @@ -167,7 +167,7 @@ object CometInitCap extends CometScalarFunction[InitCap]("initcap") with NativeO super.convert(expr, inputs, binding) } else { // Default: route through the codegen dispatcher so Spark's own doGenCode runs inside the - // Comet pipeline. This guarantees Spark-compatible behavior across 3.4 / 3.5 / 4.0. + // Comet pipeline. This guarantees Spark-compatible behavior across supported versions. // Falls through to Spark when the dispatcher is disabled. CometScalaUDF.emitJvmCodegenDispatch(expr, inputs, binding) } @@ -695,24 +695,6 @@ object CometSoundEx extends CometScalarFunction[SoundEx]("soundex") object CometStringLocate extends CometCodegenDispatch[StringLocate] -// On Spark 3.4 `Base64` is a plain expression node and always chunks the output (it uses -// `java.util.Base64.getMimeEncoder()` with no arguments). On Spark 3.5+ it is RuntimeReplaceable -// and lowers to a `StaticInvoke`, handled by CometBase64StaticInvoke instead. -object CometBase64 extends CometExpressionSerde[Base64] { - override def convert(expr: Base64, inputs: Seq[Attribute], binding: Boolean): Option[Expr] = { - val childExpr = exprToProtoInternal(expr.child, inputs, binding) - val chunkExpr = exprToProtoInternal(Literal(true), inputs, binding) - val optExpr = - scalarFunctionExprToProtoWithReturnType( - "base64", - StringType, - failOnError = false, - childExpr, - chunkExpr) - optExpr - } -} - object CometUnBase64 extends CometCodegenDispatch[UnBase64] object CometToCharacter extends CometCodegenDispatch[ToCharacter] diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala index 9ae92845c9c..d6b1d386f14 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala @@ -165,7 +165,7 @@ case class CometBroadcastExchangeExec( val beforeBroadcast = System.nanoTime() longMetric("buildTime") += NANOSECONDS.toMillis(beforeBroadcast - beforeBuild) - // (3.4 only) SPARK-39983 - Broadcast the relation without caching the unserialized object. + // SPARK-39983 - Broadcast the relation without caching the unserialized object. val broadcasted = sparkContext .broadcastInternal(batches, true) .asInstanceOf[broadcast.Broadcast[Any]] diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeWriteExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeWriteExec.scala index f0d10b17667..8fed1ef9abe 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeWriteExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeWriteExec.scala @@ -198,7 +198,7 @@ case class CometNativeWriteExec( // Get the work directory for temp files // Spark 4.1 made the (taskContext, dir, ext: String) overload throw by default; - // the FileNameSpec overload is the supported one and exists in 3.4+. + // the FileNameSpec overload is the supported one across supported versions. val workPath = committer.newTaskTempFile(taskContext, None, FileNameSpec("", "")) val workDir = new Path(workPath).getParent.toString diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryAdaptiveBroadcastExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryAdaptiveBroadcastExec.scala index 99063e8370c..2e14ba6e227 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryAdaptiveBroadcastExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryAdaptiveBroadcastExec.scala @@ -55,8 +55,8 @@ case class CometSubqueryAdaptiveBroadcastExec( with UnaryExecNode { // This node must be converted to CometSubqueryBroadcastExec by - // CometPlanAdaptiveDynamicPruningFilters before execution. If we reach doExecute(), - // the rule didn't run (e.g., Spark 3.4 where injectQueryStageOptimizerRule is unavailable). + // CometPlanAdaptiveDynamicPruningFilters before execution. Reaching doExecute() means the + // conversion rule did not run. protected override def doExecute(): RDD[InternalRow] = { throw QueryExecutionErrors.executeCodePathUnsupportedError( "CometSubqueryAdaptiveBroadcastExec (should have been converted by " + diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala b/spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala index 08efcfc9c9c..d479791f5ea 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala @@ -28,7 +28,7 @@ import org.apache.spark.sql.types.DecimalType * * Spark itself stopped wrapping these in `CheckOverflow` in 3.4 (SPARK-39316), but Comet's native * `CheckOverflow` only validates precision (it does not rescale), so the target type must equal - * the child's actual `dataType`. Always using `expr.dataType` is the safe choice: on Spark 3.4 - + * the child's actual `dataType`. Always using `expr.dataType` is the safe choice: on Spark 3.5 - * 4.0 it equals the value the rule would otherwise recompute from `SQLConf`, and on Spark 4.1+ * (SPARK-53968) it preserves the per-expression `allowDecimalPrecisionLoss` captured at view * creation time. Recomputing from the live `SQLConf` would re-label a stored DEC(38, 17) result diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala index 9505443fb31..9632ac0f0cb 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala @@ -333,9 +333,9 @@ object CometShuffleExchangeExec // A Comet shuffle wrapped around a stage that still contains a Spark FileSourceScanExec // with DPP produces inefficient row<->columnar transitions. This only happens when the - // scan fell back to Spark (e.g., AQE DPP on Spark 3.4, or unsupported scan type). - // On 3.5+ with AQE DPP, the scan converts to CometNativeScanExec and - // stageContainsDPPScan won't match (it checks FileSourceScanExec). + // scan fell back to Spark because of an unsupported scan type. AQE DPP scans convert to + // CometNativeScanExec, so stageContainsDPPScan won't match them (it checks + // FileSourceScanExec). if (stageContainsDPPScan(s)) { withFallbackReasons(s, Set("Stage contains a scan with Dynamic Partition Pruning")) return None diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala index 398ed66b6a5..adc807a665f 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala @@ -90,9 +90,8 @@ class CometShuffleManager(conf: SparkConf) extends ShuffleManager with Logging { } override val shuffleBlockResolver: IndexShuffleBlockResolver = { - // The patch versions of Spark 3.4 have different constructor signatures: - // See https://github.com/apache/spark/commit/5180694705be3508bd21dd9b863a59b8cb8ba193 - // We look for proper constructor by reflection. + // Supported Spark versions have different constructor signatures, so select the matching + // constructor by reflection. classOf[IndexShuffleBlockResolver].getDeclaredConstructors .filter(c => List(2, 3).contains(c.getParameterCount())) .map { c => diff --git a/spark/src/main/spark-3.4/org/apache/comet/shims/CometExprShim.scala b/spark/src/main/spark-3.4/org/apache/comet/shims/CometExprShim.scala deleted file mode 100644 index 1ad9ec75bf0..00000000000 --- a/spark/src/main/spark-3.4/org/apache/comet/shims/CometExprShim.scala +++ /dev/null @@ -1,62 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.shims - -import org.apache.spark.sql.catalyst.expressions._ -import org.apache.spark.sql.catalyst.expressions.aggregate.Sum - -import org.apache.comet.expressions.CometEvalMode -import org.apache.comet.serde.{CometExpressionSerde, CometStringDecode} -import org.apache.comet.serde.ExprOuterClass.{BinaryOutputStyle, Expr} - -/** - * `CometExprShim` acts as a shim for parsing expressions from different Spark versions. - */ -trait CometExprShim { - protected def evalMode(c: Cast): CometEvalMode.Value = - CometEvalModeUtil.fromSparkEvalMode(c.evalMode) - - def binaryOutputStyle: BinaryOutputStyle = BinaryOutputStyle.HEX_DISCRETE - - def sparkVersionSpecificStringExpressions - : Map[Class[_ <: Expression], CometExpressionSerde[_]] = - Map(classOf[StringDecode] -> CometStringDecode) - def sparkVersionSpecificMathExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = - Map.empty - def sparkVersionSpecificMiscExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = - Map.empty - def sparkVersionSpecificMapExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = - Map.empty - - def sparkVersionSpecificExprToProtoInternal( - expr: Expression, - inputs: Seq[Attribute], - binding: Boolean): Option[Expr] = None -} - -object CometEvalModeUtil { - def fromSparkEvalMode(evalMode: EvalMode.Value): CometEvalMode.Value = evalMode match { - case EvalMode.LEGACY => CometEvalMode.LEGACY - case EvalMode.TRY => CometEvalMode.TRY - case EvalMode.ANSI => CometEvalMode.ANSI - } - - def sumEvalMode(s: Sum): EvalMode.Value = s.evalMode -} diff --git a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimBatchReader.scala b/spark/src/main/spark-3.4/org/apache/comet/shims/ShimBatchReader.scala deleted file mode 100644 index 17b60e0e5a1..00000000000 --- a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimBatchReader.scala +++ /dev/null @@ -1,37 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.shims - -import org.apache.spark.paths.SparkPath -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.execution.datasources.PartitionedFile - -object ShimBatchReader { - - def newPartitionedFile(partitionValues: InternalRow, file: String): PartitionedFile = - PartitionedFile( - partitionValues, - SparkPath.fromPathString(file), - -1, // -1 means we read the entire file - -1, - Array.empty[String], - 0, - 0) -} diff --git a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimCometBroadcastExchangeExec.scala b/spark/src/main/spark-3.4/org/apache/comet/shims/ShimCometBroadcastExchangeExec.scala deleted file mode 100644 index 93eee60923d..00000000000 --- a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimCometBroadcastExchangeExec.scala +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.shims - -import org.apache.spark.SparkContext -import org.apache.spark.network.util.JavaUtils -import org.apache.spark.sql.execution.exchange.BroadcastExchangeLike -import org.apache.spark.sql.internal.SQLConf - -import org.apache.comet.shims.ShimCometBroadcastExchangeExec.SPARK_MAX_BROADCAST_TABLE_SIZE - -trait ShimCometBroadcastExchangeExec { - - def setJobGroupOrTag(sc: SparkContext, broadcastExchange: BroadcastExchangeLike): Unit = { - // Setup a job group here so later it may get cancelled by groupId if necessary. - sc.setJobGroup( - broadcastExchange.runId.toString, - s"broadcast exchange (runId ${broadcastExchange.runId})", - interruptOnCancel = true) - } - - def cancelJobGroup(sc: SparkContext, broadcastExchange: BroadcastExchangeLike): Unit = { - sc.cancelJobGroup(broadcastExchange.runId.toString) - } - - def maxBroadcastTableBytes(conf: SQLConf): Long = { - JavaUtils.byteStringAsBytes(conf.getConfString(SPARK_MAX_BROADCAST_TABLE_SIZE, "8GB")) - } - -} - -object ShimCometBroadcastExchangeExec { - val SPARK_MAX_BROADCAST_TABLE_SIZE = "spark.sql.maxBroadcastTableSize" -} diff --git a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala b/spark/src/main/spark-3.4/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala deleted file mode 100644 index 55e3e1cf8a3..00000000000 --- a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala +++ /dev/null @@ -1,60 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.shims - -import org.apache.spark.sql.SparkSessionExtensions -import org.apache.spark.sql.catalyst.rules.Rule -import org.apache.spark.sql.execution.{QueryExecution, SparkPlan} - -trait ShimCometSparkSessionExtensions { - - protected val EXTENDED_EXPLAIN_PROVIDERS_KEY = "spark.sql.extendedExplainProviders" - - def supportsExtendedExplainInfo(qe: QueryExecution): Boolean = { - try { - qe.getClass.getDeclaredMethod( - "extendedExplainInfo", - classOf[String => Unit], - classOf[SparkPlan]) - } catch { - case _: NoSuchMethodException | _: SecurityException => return false - } - true - } - - // injectQueryStageOptimizerRule not available on Spark 3.4. - // CometPlanAdaptiveDynamicPruningFilters and CometReuseSubquery are not registered. - // On 3.4, Spark's PlanAdaptiveDynamicPruningFilters handles SABs directly (converting to - // TrueLiteral when it can't find BroadcastHashJoinExec, disabling DPP). - def injectQueryStageOptimizerRuleShim( - extensions: SparkSessionExtensions, - rule: Rule[SparkPlan]): Unit = {} - - // Registers a queryStagePrepRule only on Spark < 3.5. The 3.5+ variants no-op this shim. - // Used by CometSpark34AqeDppFallbackRule, which is a correctness workaround for AQE DPP on - // Spark 3.4 where injectQueryStageOptimizerRule (and therefore - // CometPlanAdaptiveDynamicPruningFilters) is unavailable. See - // CometSpark34AqeDppFallbackRule's class docstring for details. - def injectPreSpark35QueryStagePrepRuleShim( - extensions: SparkSessionExtensions, - rule: Rule[SparkPlan]): Unit = { - extensions.injectQueryStagePrepRule(_ => rule) - } -} diff --git a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimFileFormat.scala b/spark/src/main/spark-3.4/org/apache/comet/shims/ShimFileFormat.scala deleted file mode 100644 index 6906a39fbfc..00000000000 --- a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimFileFormat.scala +++ /dev/null @@ -1,67 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.shims - -import org.apache.hadoop.fs.Path -import org.apache.spark.sql.catalyst.expressions.Literal -import org.apache.spark.sql.execution.datasources.{FileFormat, PartitionedFile, RowIndexUtil} -import org.apache.spark.sql.types.{DataType, StructType} - -object ShimFileFormat { - - // A name for a temporary column that holds row indexes computed by the file format reader - // until they can be placed in the _metadata struct. - val ROW_INDEX_TEMPORARY_COLUMN_NAME: String = FileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME - - def findRowIndexColumnIndexInSchema(sparkSchema: StructType): Int = - RowIndexUtil.findRowIndexColumnIndexInSchema(sparkSchema) - - // Spark 3.4 has no per-format metadata extractor concept (added in Spark 3.5, SPARK-43868); - // it derives these values inline in FileFormat.updateMetadataInternalRow. Replicate that - // logic here so callers can use the same extractor-map shape across all Spark versions. - private val baseMetadataExtractors: Map[String, PartitionedFile => Any] = Map( - FileFormat.FILE_PATH -> { pf: PartitionedFile => - // Use `new Path(Path.toString)` as a form of canonicalization - new Path(pf.filePath.toPath.toString).toUri.toString - }, - FileFormat.FILE_NAME -> { pf: PartitionedFile => - pf.filePath.toUri.getRawPath.split("/").lastOption.getOrElse("") - }, - FileFormat.FILE_SIZE -> { pf: PartitionedFile => pf.fileSize }, - FileFormat.FILE_BLOCK_START -> { pf: PartitionedFile => pf.start }, - FileFormat.FILE_BLOCK_LENGTH -> { pf: PartitionedFile => pf.length }, - // The modificationTime from the file has millisecond granularity, but the TimestampType for - // `file_modification_time` has microsecond granularity. - FileFormat.FILE_MODIFICATION_TIME -> { pf: PartitionedFile => pf.modificationTime * 1000 }) - - def fileConstantMetadataExtractors( - fileFormat: FileFormat): Map[String, PartitionedFile => Any] = - baseMetadataExtractors - - // dataType is unused on this Spark version; getFileConstantMetadataColumnValue only gained - // a dataType parameter in Spark 4.2 (SPARK-56931). Accepted here so callers can pass it - // uniformly across Spark versions. - def getFileConstantMetadataColumnValue( - name: String, - file: PartitionedFile, - extractors: Map[String, PartitionedFile => Any], - dataType: DataType): Literal = - Literal(extractors(name)(file)) -} diff --git a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimSQLConf.scala b/spark/src/main/spark-3.4/org/apache/comet/shims/ShimSQLConf.scala deleted file mode 100644 index 3ab5e5cbaca..00000000000 --- a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimSQLConf.scala +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.shims - -import org.apache.spark.sql.internal.SQLConf -import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy - -trait ShimSQLConf { - protected val LEGACY = LegacyBehaviorPolicy.LEGACY - protected val CORRECTED = LegacyBehaviorPolicy.CORRECTED - - /** - * Reads `spark.sql.execution.arrow.useLargeVarTypes`. Spark 3.4 has no typed accessor for this - * conf, so read by raw key. The conf only governs the destination Arrow IPC root width on Spark - * 4.x, so the value returned here matters only to callers that look it up explicitly. - */ - protected def arrowUseLargeVarTypes(conf: SQLConf): Boolean = - conf.getConfString("spark.sql.execution.arrow.useLargeVarTypes", "false").toBoolean -} diff --git a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimSubqueryBroadcast.scala b/spark/src/main/spark-3.4/org/apache/comet/shims/ShimSubqueryBroadcast.scala deleted file mode 100644 index 20663bb05ba..00000000000 --- a/spark/src/main/spark-3.4/org/apache/comet/shims/ShimSubqueryBroadcast.scala +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.shims - -import org.apache.spark.sql.catalyst.expressions.Expression -import org.apache.spark.sql.execution.{SparkPlan, SubqueryAdaptiveBroadcastExec, SubqueryBroadcastExec} - -trait ShimSubqueryBroadcast { - - def getSubqueryBroadcastIndices(sab: SubqueryAdaptiveBroadcastExec): Seq[Int] = { - Seq(sab.index) - } - - def getSubqueryBroadcastExecIndices(sub: SubqueryBroadcastExec): Seq[Int] = { - Seq(sub.index) - } - - def createSubqueryBroadcastExec( - name: String, - indices: Seq[Int], - buildKeys: Seq[Expression], - child: SparkPlan): SubqueryBroadcastExec = { - assert(indices.length == 1, s"Multi-index DPP not supported on Spark 3.4: indices=$indices") - SubqueryBroadcastExec(name, indices.head, buildKeys, child) - } -} diff --git a/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometScanExec.scala b/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometScanExec.scala deleted file mode 100644 index 7d8ba9f0f8b..00000000000 --- a/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometScanExec.scala +++ /dev/null @@ -1,83 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.spark.sql.comet.shims - -import org.apache.hadoop.fs.{FileStatus, Path} -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression} -import org.apache.spark.sql.execution.{FileSourceScanExec, PartitionedFileUtil} -import org.apache.spark.sql.execution.datasources._ -import org.apache.spark.sql.execution.datasources.parquet.ParquetOptions -import org.apache.spark.sql.sources.Filter -import org.apache.spark.sql.types.StructType - -import org.apache.comet.shims.ShimFileFormat - -trait ShimCometScanExec { - def wrapped: FileSourceScanExec - - lazy val fileConstantMetadataColumns: Seq[AttributeReference] = - wrapped.fileConstantMetadataColumns - - protected def newFileScanRDD( - fsRelation: HadoopFsRelation, - readFunction: PartitionedFile => Iterator[InternalRow], - filePartitions: Seq[FilePartition], - readSchema: StructType, - options: ParquetOptions): FileScanRDD = new FileScanRDD( - fsRelation.sparkSession, - readFunction, - filePartitions, - readSchema, - fileConstantMetadataColumns, - options) - - protected def isNeededForSchema(sparkSchema: StructType): Boolean = { - // TODO: remove after PARQUET-2161 becomes available in Parquet (tracked in SPARK-39634) - ShimFileFormat.findRowIndexColumnIndexInSchema(sparkSchema) >= 0 - } - - protected def getPartitionedFile(f: FileStatus, p: PartitionDirectory): PartitionedFile = - PartitionedFileUtil.getPartitionedFile(f, f.getPath, p.values) - - protected def splitFiles( - sparkSession: SparkSession, - file: FileStatus, - filePath: Path, - isSplitable: Boolean, - maxSplitBytes: Long, - partitionValues: InternalRow): Seq[PartitionedFile] = - PartitionedFileUtil.splitFiles( - sparkSession, - file, - filePath, - isSplitable, - maxSplitBytes, - partitionValues) - - protected def getPushedDownFilters( - relation: HadoopFsRelation, - dataFilters: Seq[Expression]): Seq[Filter] = { - val supportNestedPredicatePushdown = DataSourceUtils.supportNestedPredicatePushdown(relation) - dataFilters.flatMap(DataSourceStrategy.translateFilter(_, supportNestedPredicatePushdown)) - } - -} diff --git a/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometUnionExec.scala b/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometUnionExec.scala deleted file mode 100644 index 15c480b057c..00000000000 --- a/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometUnionExec.scala +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.spark.sql.comet.shims - -import scala.reflect.ClassTag - -import org.apache.spark.SparkContext -import org.apache.spark.rdd.RDD -import org.apache.spark.sql.catalyst.plans.physical.Partitioning - -object ShimCometUnionExec { - - /** - * Unions a sequence of RDDs while preserving the declared output partitioning. Before Spark - * 4.1, [[org.apache.spark.sql.execution.UnionExec]] always reports - * [[org.apache.spark.sql.catalyst.plans.physical.UnknownPartitioning]], so this shim simply - * concatenates partitions via `SparkContext.union`. The partitioning-aware path is only needed - * on Spark 4.1+ (see SPARK-52921). - */ - def unionRDDs[T: ClassTag]( - sc: SparkContext, - rdds: Seq[RDD[T]], - @annotation.nowarn("cat=unused") outputPartitioning: Partitioning): RDD[T] = { - sc.union(rdds) - } -} diff --git a/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala b/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala deleted file mode 100644 index 6cb64486725..00000000000 --- a/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala +++ /dev/null @@ -1,405 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.spark.sql.comet.shims - -import java.io.FileNotFoundException - -import scala.util.matching.Regex - -import org.apache.spark.{QueryContext, SparkDateTimeException, SparkException} -import org.apache.spark.sql.catalyst.trees.SQLQueryContext -import org.apache.spark.sql.errors.QueryExecutionErrors -import org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException -import org.apache.spark.sql.types._ -import org.apache.spark.unsafe.types.UTF8String - -object ShimSparkErrorConverter { - val ObjectLocationPattern: Regex = "Object at location (.+?) not found".r -} - -/** - * Spark 3.4 implementation for converting error types to proper Spark exceptions. - * - * Handles all error cases using the Spark 3.4 QueryExecutionErrors API. Differs from the 3.5 - * version in 4 places where the API changed between 3.4 and 3.5. - */ -trait ShimSparkErrorConverter { - - private def sqlCtx(context: Array[QueryContext]): SQLQueryContext = - context.headOption.map(_.asInstanceOf[SQLQueryContext]).getOrElse(null) - - private def parseFloatLiteral(value: String): Float = { - value.toLowerCase match { - case "inf" | "+inf" | "infinity" | "+infinity" => Float.PositiveInfinity - case "-inf" | "-infinity" => Float.NegativeInfinity - case "nan" | "+nan" | "-nan" => Float.NaN - case _ => value.toFloat - } - } - - private def parseDoubleLiteral(value: String): Double = { - val normalized = value.toLowerCase.stripSuffix("d") - normalized match { - case "inf" | "+inf" | "infinity" | "+infinity" => Double.PositiveInfinity - case "-inf" | "-infinity" => Double.NegativeInfinity - case "nan" | "+nan" | "-nan" => Double.NaN - case _ => normalized.toDouble - } - } - - def convertErrorType( - errorType: String, - errorClass: String, - params: Map[String, Any], - context: Array[QueryContext], - summary: String): Option[Throwable] = { - val _ = (errorClass, summary) - - errorType match { - - case "DivideByZero" => - Some(QueryExecutionErrors.divideByZeroError(sqlCtx(context))) - - case "RemainderByZero" => - Some(QueryExecutionErrors.divideByZeroError(sqlCtx(context))) - - case "IntervalDividedByZero" => - Some(QueryExecutionErrors.intervalDividedByZeroError(sqlCtx(context))) - - case "BinaryArithmeticOverflow" => - // Spark 3.x does not take functionName parameter - Some( - QueryExecutionErrors.binaryArithmeticCauseOverflowError( - params("value1").toString.toShort, - params("symbol").toString, - params("value2").toString.toShort)) - - case "ArithmeticOverflow" => - val fromType = params("fromType").toString - Some( - QueryExecutionErrors - .arithmeticOverflowError(fromType + " overflow", "", sqlCtx(context))) - - case "IntegralDivideOverflow" => - Some(QueryExecutionErrors.overflowInIntegralDivideError(sqlCtx(context))) - - case "DecimalSumOverflow" => - // Spark 3.x takes SQLQueryContext, not QueryContext - Some(QueryExecutionErrors.overflowInSumOfDecimalError(sqlCtx(context))) - - case "NumericValueOutOfRange" => - val decimal = Decimal(params("value").toString) - Some( - QueryExecutionErrors.cannotChangeDecimalPrecisionError( - decimal, - params("precision").toString.toInt, - params("scale").toString.toInt, - sqlCtx(context))) - - case "DatetimeOverflow" => - Some( - new SparkException( - errorClass = "DATETIME_OVERFLOW", - messageParameters = params.map { case (k, v) => (k, v.toString) }, - cause = null)) - - case "InvalidArrayIndex" => - Some( - QueryExecutionErrors.invalidArrayIndexError( - params("indexValue").toString.toInt, - params("arraySize").toString.toInt, - sqlCtx(context))) - - case "InvalidElementAtIndex" => - Some( - QueryExecutionErrors.invalidElementAtIndexError( - params("indexValue").toString.toInt, - params("arraySize").toString.toInt, - sqlCtx(context))) - - case "InvalidIndexOfZero" => - Some(QueryExecutionErrors.invalidIndexOfZeroError(sqlCtx(context))) - - case "InvalidBitmapPosition" => - // invalidBitmapPositionError does not exist in Spark 3.4 - Some( - new SparkException( - errorClass = "INVALID_BITMAP_POSITION", - messageParameters = params.map { case (k, v) => (k, v.toString) }, - cause = null)) - - case "DuplicatedMapKey" => - Some(QueryExecutionErrors.duplicateMapKeyFoundError(params("key"))) - - case "NullMapKey" => - Some(QueryExecutionErrors.nullAsMapKeyNotAllowedError()) - - case "MapKeyValueDiffSizes" => - Some(QueryExecutionErrors.mapDataKeyArrayLengthDiffersFromValueArrayLengthError()) - - case "ExceedMapSizeLimit" => - Some(QueryExecutionErrors.exceedMapSizeLimitError(params("size").toString.toInt)) - - case "CollectionSizeLimitExceeded" => - // createArrayWithElementsExceedLimitError takes (count: Any) in Spark 3.4 - Some( - QueryExecutionErrors.createArrayWithElementsExceedLimitError( - params("numElements").toString.toLong)) - - case "NotNullAssertViolation" => - Some( - QueryExecutionErrors.foundNullValueForNotNullableFieldError( - params("fieldName").toString)) - - case "ValueIsNull" => - Some( - QueryExecutionErrors.fieldCannotBeNullError( - params.getOrElse("rowIndex", 0).toString.toInt, - params("fieldName").toString)) - - case "CannotParseTimestamp" => - Some( - QueryExecutionErrors.ansiDateTimeParseError(new Exception(params("message").toString))) - - case "IllegalDayOfWeek" => - Some( - QueryExecutionErrors - .ansiIllegalArgumentError(s"Illegal input for day of week: ${params("string")}")) - - case "DatetimeFieldOutOfBounds" => - Some( - QueryExecutionErrors.ansiDateTimeError( - new java.time.DateTimeException(params("rangeMessage").toString))) - - case "InvalidFractionOfSecond" => - Some(QueryExecutionErrors.invalidFractionOfSecondError()) - - case "CastInvalidValue" => - val str = UTF8String.fromString(params("value").toString) - val targetType = getDataType(params("toType").toString) - Some( - QueryExecutionErrors - .invalidInputInCastToNumberError(targetType, str, sqlCtx(context))) - - case "InvalidInputInCastToDatetime" => - val expression = - s"'${params("value").toString.replace("\\", "\\\\").replace("'", "\\'")}'" - val sourceType = s""""${params("fromType").toString}"""" - val targetType = s""""${params("toType").toString}"""" - Some( - new SparkDateTimeException( - errorClass = "CAST_INVALID_INPUT", - messageParameters = Map( - "expression" -> expression, - "sourceType" -> sourceType, - "targetType" -> targetType, - "ansiConfig" -> "\"spark.sql.ansi.enabled\""), - context = context, - summary = summary)) - - case "CastOverFlow" => - val fromType = getDataType(params("fromType").toString) - val toType = getDataType(params("toType").toString) - val valueStr = params("value").toString - - val typedValue: Any = fromType match { - case _: DecimalType => - val cleanStr = if (valueStr.endsWith("BD")) valueStr.dropRight(2) else valueStr - Decimal(cleanStr) - case ByteType => - val cleanStr = if (valueStr.endsWith("T")) valueStr.dropRight(1) else valueStr - cleanStr.toByte - case ShortType => - val cleanStr = if (valueStr.endsWith("S")) valueStr.dropRight(1) else valueStr - cleanStr.toShort - case IntegerType => valueStr.toInt - case LongType => - val cleanStr = if (valueStr.endsWith("L")) valueStr.dropRight(1) else valueStr - cleanStr.toLong - case FloatType => parseFloatLiteral(valueStr) - case DoubleType => parseDoubleLiteral(valueStr) - case StringType => UTF8String.fromString(valueStr) - case _ => valueStr - } - - Some(QueryExecutionErrors.castingCauseOverflowError(typedValue, fromType, toType)) - - case "CannotParseDecimal" => - Some(QueryExecutionErrors.cannotParseDecimalError()) - - case "InvalidUtf8String" => - // invalidUTF8StringError does not exist in Spark 3.x; use generic fallback - Some( - new SparkException( - errorClass = "INVALID_UTF8_STRING", - messageParameters = params.map { case (k, v) => (k, v.toString) }, - cause = null)) - - case "UnexpectedPositiveValue" => - Some( - QueryExecutionErrors.unexpectedValueForStartInFunctionError( - params("parameterName").toString)) - - case "UnexpectedNegativeValue" => - Some( - QueryExecutionErrors.unexpectedValueForLengthInFunctionError( - params("parameterName").toString)) - - case "InvalidRegexGroupIndex" => - // invalidRegexGroupIndexError does not exist in Spark 3.4 - Some( - new SparkException( - errorClass = "INVALID_REGEX_GROUP_INDEX", - messageParameters = params.map { case (k, v) => (k, v.toString) }, - cause = null)) - - case "DatatypeCannotOrder" => - // orderedOperationUnsupportedByDataTypeError takes DataType in Spark 3.4, not String - Some( - QueryExecutionErrors.orderedOperationUnsupportedByDataTypeError( - getDataType(params("dataType").toString))) - - case "ScalarSubqueryTooManyRows" => - // multipleRowScalarSubqueryError was renamed to multipleRowSubqueryError in Spark 3.x - Some(QueryExecutionErrors.multipleRowSubqueryError(sqlCtx(context))) - - case "IntervalArithmeticOverflowWithSuggestion" => - // Spark 3.x uses a single intervalArithmeticOverflowError method - Some( - QueryExecutionErrors.intervalArithmeticOverflowError( - "Interval arithmetic overflow", - params.get("functionName").map(_.toString).getOrElse(""), - sqlCtx(context))) - - case "IntervalArithmeticOverflowWithoutSuggestion" => - Some( - QueryExecutionErrors - .intervalArithmeticOverflowError("Interval arithmetic overflow", "", sqlCtx(context))) - - case "DuplicateFieldCaseInsensitive" => - Some( - QueryExecutionErrors.foundDuplicateFieldInCaseInsensitiveModeError( - params("requiredFieldName").toString, - params("matchedOrcFields").toString)) - - case "DuplicateFieldByFieldId" => - // Mirror Spark's `ParquetReadSupport.matchIdField` which calls - // `foundDuplicateFieldInFieldIdLookupModeError` when more than one Parquet field - // shares an id requested by the read schema. - Some( - QueryExecutionErrors.foundDuplicateFieldInFieldIdLookupModeError( - params("requiredId").toString.toInt, - params("matchedFields").toString)) - - case "ParquetMissingFieldIds" => - // Mirror Spark's `ParquetReadSupport.inferSchema`, which throws a plain - // `RuntimeException` (not a SparkException) when the read schema requests field - // ids and the file carries none. - Some( - new RuntimeException( - "Spark read schema expects field Ids, but Parquet file schema doesn't " + - "contain any field Ids. Please remove the field ids from Spark schema or " + - "ignore missing ids by setting " + - "`spark.sql.parquet.fieldId.read.ignoreMissing = true`")) - - case "ParquetSchemaConvert" => - // Mirror Spark's FileScanRDD: wrap the SchemaColumnConvertNotSupportedException - // in a SparkException whose message is "Parquet column cannot be converted in - // file ...". The native side may not have the file path; an empty path - // still produces a message that contains "Parquet column cannot be converted in - // file" (which is what Spark's own SQL tests assert). - val column = params("column").toString - val physicalType = params("physicalType").toString - val logicalType = params("sparkType").toString - val filePath = params.get("filePath").map(_.toString).getOrElse("") - val cause = - new SchemaColumnConvertNotSupportedException(column, physicalType, logicalType) - Some( - QueryExecutionErrors.unsupportedSchemaColumnConvertError( - filePath, - column, - logicalType, - physicalType, - cause)) - - case "FileNotFound" => - val msg = params("message").toString - // Extract file path from native error message and format like Hadoop's - // FileNotFoundException: "File does not exist" - val path = ShimSparkErrorConverter.ObjectLocationPattern - .findFirstMatchIn(msg) - .map(_.group(1)) - .getOrElse(msg) - Some( - QueryExecutionErrors.readCurrentFileNotFoundError( - new FileNotFoundException(s"File $path does not exist"))) - - case "CannotReadFile" => - // A per-file read failure of a readable-but-broken file (corrupt/truncated parquet, - // object_store, IO) classified by typed DataFusionError variant on the native side. Wrap - // in the FAILED_READ_FILE SparkException Spark itself produces when its own parquet reader - // fails. (A genuinely-missing file is "FileNotFound" above.) `filePath` is filled by - // SparkErrorConverter from the per-task file list when the native error carried none. - val message = params.get("message").map(_.toString).getOrElse("") - val filePath = params.get("filePath").map(_.toString).getOrElse("") - Some(QueryExecutionErrors.cannotReadFilesError(new SparkException(message), filePath)) - - case _ => - None - } - } - - private def getDataType(typeName: String): DataType = { - typeName.toUpperCase match { - case "BYTE" | "TINYINT" => ByteType - case "SHORT" | "SMALLINT" => ShortType - case "INT" | "INTEGER" => IntegerType - case "LONG" | "BIGINT" => LongType - case "FLOAT" | "REAL" => FloatType - case "DOUBLE" => DoubleType - case "DECIMAL" => DecimalType.SYSTEM_DEFAULT - case "STRING" | "VARCHAR" => StringType - case "BINARY" => BinaryType - case "BOOLEAN" => BooleanType - case "DATE" => DateType - case "TIMESTAMP" => TimestampType - case _ => - try { - DataType.fromDDL(typeName) - } catch { - case _: Exception => - // fromDDL rejects types that are syntactically invalid in SQL DDL, such as - // DECIMAL(p,s) with a negative scale (valid when allowNegativeScaleOfDecimal=true). - // Parse those manually rather than silently falling back to StringType. - if (typeName.toUpperCase.startsWith("DECIMAL(") && typeName.endsWith(")")) { - val inner = typeName.substring("DECIMAL(".length, typeName.length - 1) - val parts = inner.split(",") - if (parts.length == 2) { - try { - DataTypes.createDecimalType(parts(0).trim.toInt, parts(1).trim.toInt) - } catch { - case _: Exception => StringType - } - } else StringType - } else StringType - } - } - } -} diff --git a/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimTaskMetrics.scala b/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimTaskMetrics.scala deleted file mode 100644 index 5b2a5fb5bf5..00000000000 --- a/spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimTaskMetrics.scala +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.spark.sql.comet.shims - -import org.apache.spark.executor.TaskMetrics -import org.apache.spark.util.AccumulatorV2 - -object ShimTaskMetrics { - - def getTaskAccumulator(taskMetrics: TaskMetrics): Option[AccumulatorV2[_, _]] = - taskMetrics.externalAccums.lastOption -} diff --git a/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala b/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala index e28bdfc55b5..c33e7447a0c 100644 --- a/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala +++ b/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala @@ -46,9 +46,4 @@ trait ShimCometSparkSessionExtensions { extensions.injectQueryStageOptimizerRule(_ => rule) } - // No-op on Spark >= 3.5. See the Spark 3.4 shim and - // CometSpark34AqeDppFallbackRule's class docstring for why this shim exists. - def injectPreSpark35QueryStagePrepRuleShim( - extensions: SparkSessionExtensions, - rule: Rule[SparkPlan]): Unit = {} } diff --git a/spark/src/main/spark-3.x/org/apache/comet/shims/CometCollectShim.scala b/spark/src/main/spark-3.x/org/apache/comet/shims/CometCollectShim.scala index 69f33a1b29a..f07245263b3 100644 --- a/spark/src/main/spark-3.x/org/apache/comet/shims/CometCollectShim.scala +++ b/spark/src/main/spark-3.x/org/apache/comet/shims/CometCollectShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.catalyst.expressions.aggregate.{CollectList, CollectSet} /** - * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.4 through 4.1 have no + * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.5 through 4.1 have no * such field: these aggregates always drop null inputs, so this shim reports `true`. Spark 4.2 * added `ignoreNulls` (settable to `false` via `RESPECT NULLS`), handled by the spark-4.2 shim. */ diff --git a/spark/src/main/spark-3.x/org/apache/comet/shims/CometSampleShim.scala b/spark/src/main/spark-3.x/org/apache/comet/shims/CometSampleShim.scala index ff927120687..52cde14eeaa 100644 --- a/spark/src/main/spark-3.x/org/apache/comet/shims/CometSampleShim.scala +++ b/spark/src/main/spark-3.x/org/apache/comet/shims/CometSampleShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.execution.SampleExec /** - * Shim for the seed that `SampleExec` samples with. Spark 3.4 through 4.1 carry it as a plain + * Shim for the seed that `SampleExec` samples with. Spark 3.5 through 4.1 carry it as a plain * `Long`. Spark 4.2 made it an `Option[Long]` resolved into a `resolvedSeed` field, handled by * the spark-4.2 shim. */ diff --git a/spark/src/main/spark-3.x/org/apache/comet/shims/ShimCometShuffleExchangeExec.scala b/spark/src/main/spark-3.x/org/apache/comet/shims/ShimCometShuffleExchangeExec.scala index 3198035446b..a1b41412657 100644 --- a/spark/src/main/spark-3.x/org/apache/comet/shims/ShimCometShuffleExchangeExec.scala +++ b/spark/src/main/spark-3.x/org/apache/comet/shims/ShimCometShuffleExchangeExec.scala @@ -26,19 +26,14 @@ import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec import org.apache.spark.sql.types.{StructField, StructType} trait ShimCometShuffleExchangeExec { - // TODO: remove after dropping Spark 3.4 support def apply(s: ShuffleExchangeExec, shuffleType: ShuffleType): CometShuffleExchangeExec = { - val advisoryPartitionSize = s.getClass.getDeclaredMethods - .filter(_.getName == "advisoryPartitionSize") - .flatMap(_.invoke(s).asInstanceOf[Option[Long]]) - .headOption CometShuffleExchangeExec( s.outputPartitioning, s.child, s, s.shuffleOrigin, shuffleType, - advisoryPartitionSize) + s.advisoryPartitionSize) } // TODO: remove after dropping Spark 3.x support diff --git a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometBatchScanExec.scala b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometBatchScanExec.scala index 9f163da9b2a..229dcb85ca7 100644 --- a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometBatchScanExec.scala +++ b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometBatchScanExec.scala @@ -25,7 +25,7 @@ import org.apache.spark.sql.execution.datasources.v2.BatchScanExec trait ShimCometBatchScanExec { protected def wrappedScan: BatchScanExec - // Spark 3.4 and 3.5 expose `partitions` as Seq[Seq[InputPartition]] (same as 4.0/4.1); + // Spark 3.5 exposes `partitions` as Seq[Seq[InputPartition]] (same as 4.0/4.1); // 4.2 changed it to Seq[Option[InputPartition]]. The concrete subclass overrides // `partitions` to delegate to `shimPartitions`, which carries the version-specific // element type. diff --git a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala index 59d8c7f251b..d8972d6da8e 100644 --- a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala +++ b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala @@ -30,16 +30,14 @@ import org.apache.spark.sql.vectorized.ColumnarBatch /** * Spark 3.x stub for the PyArrow UDF acceleration support. * - * The columnar runner introduced in #4234 only targets Spark 4.0+. On Spark 3.4 / 3.5 the - * matchers return `None`, the rewrite does not fire, and vanilla Spark handles `mapInArrow` / + * The columnar runner introduced in #4234 only targets Spark 4.0+. On Spark 3.5 the matchers + * return `None`, the rewrite does not fire, and vanilla Spark handles `mapInArrow` / * `mapInPandas` unchanged. The runner factory throws; it is never called because the matchers * always return `None`. 3.x support can be added later if there is user demand. * - * Shared across spark-3.4 and spark-3.5 because both are identical: 3.4 lacks the modern - * `ArrowPythonRunner` constructor and `arrowUseLargeVarTypes`, and 3.5's `PythonArrowInput` trait - * has a different contract (`writeIteratorToArrowStream` one-shot vs 4.x's - * `writeNextBatchToArrowStream` batch-at-a-time), so neither version can host the columnar input - * implementation without a separate rewrite. + * Spark 3.5's `PythonArrowInput` trait has a different contract (`writeIteratorToArrowStream` + * one-shot vs 4.x's `writeNextBatchToArrowStream` batch-at-a-time), so it cannot host the + * columnar input implementation without a separate rewrite. */ trait ShimCometMapInBatch { diff --git a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimDataSourceRDDPartition.scala b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimDataSourceRDDPartition.scala index dd1d30e09ee..a7173c6d8e2 100644 --- a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimDataSourceRDDPartition.scala +++ b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimDataSourceRDDPartition.scala @@ -23,7 +23,7 @@ import org.apache.spark.sql.connector.read.InputPartition import org.apache.spark.sql.execution.datasources.v2.DataSourceRDDPartition object ShimDataSourceRDDPartition { - // In Spark 3.4, 3.5, 4.0, and 4.1 a DataSourceRDDPartition holds a Seq[InputPartition] + // In Spark 3.5, 4.0, and 4.1 a DataSourceRDDPartition holds a Seq[InputPartition] // under `inputPartitions`. In 4.2 it was renamed to a single `inputPartition`. def inputPartitions(p: DataSourceRDDPartition): Seq[InputPartition] = p.inputPartitions } diff --git a/spark/src/main/spark-4.0/org/apache/comet/shims/CometCollectShim.scala b/spark/src/main/spark-4.0/org/apache/comet/shims/CometCollectShim.scala index 69f33a1b29a..f07245263b3 100644 --- a/spark/src/main/spark-4.0/org/apache/comet/shims/CometCollectShim.scala +++ b/spark/src/main/spark-4.0/org/apache/comet/shims/CometCollectShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.catalyst.expressions.aggregate.{CollectList, CollectSet} /** - * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.4 through 4.1 have no + * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.5 through 4.1 have no * such field: these aggregates always drop null inputs, so this shim reports `true`. Spark 4.2 * added `ignoreNulls` (settable to `false` via `RESPECT NULLS`), handled by the spark-4.2 shim. */ diff --git a/spark/src/main/spark-4.0/org/apache/comet/shims/CometSampleShim.scala b/spark/src/main/spark-4.0/org/apache/comet/shims/CometSampleShim.scala index ff927120687..52cde14eeaa 100644 --- a/spark/src/main/spark-4.0/org/apache/comet/shims/CometSampleShim.scala +++ b/spark/src/main/spark-4.0/org/apache/comet/shims/CometSampleShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.execution.SampleExec /** - * Shim for the seed that `SampleExec` samples with. Spark 3.4 through 4.1 carry it as a plain + * Shim for the seed that `SampleExec` samples with. Spark 3.5 through 4.1 carry it as a plain * `Long`. Spark 4.2 made it an `Option[Long]` resolved into a `resolvedSeed` field, handled by * the spark-4.2 shim. */ diff --git a/spark/src/main/spark-4.1/org/apache/comet/shims/CometCollectShim.scala b/spark/src/main/spark-4.1/org/apache/comet/shims/CometCollectShim.scala index 69f33a1b29a..f07245263b3 100644 --- a/spark/src/main/spark-4.1/org/apache/comet/shims/CometCollectShim.scala +++ b/spark/src/main/spark-4.1/org/apache/comet/shims/CometCollectShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.catalyst.expressions.aggregate.{CollectList, CollectSet} /** - * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.4 through 4.1 have no + * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.5 through 4.1 have no * such field: these aggregates always drop null inputs, so this shim reports `true`. Spark 4.2 * added `ignoreNulls` (settable to `false` via `RESPECT NULLS`), handled by the spark-4.2 shim. */ diff --git a/spark/src/main/spark-4.1/org/apache/comet/shims/CometSampleShim.scala b/spark/src/main/spark-4.1/org/apache/comet/shims/CometSampleShim.scala index ff927120687..52cde14eeaa 100644 --- a/spark/src/main/spark-4.1/org/apache/comet/shims/CometSampleShim.scala +++ b/spark/src/main/spark-4.1/org/apache/comet/shims/CometSampleShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.execution.SampleExec /** - * Shim for the seed that `SampleExec` samples with. Spark 3.4 through 4.1 carry it as a plain + * Shim for the seed that `SampleExec` samples with. Spark 3.5 through 4.1 carry it as a plain * `Long`. Spark 4.2 made it an `Option[Long]` resolved into a `resolvedSeed` field, handled by * the spark-4.2 shim. */ diff --git a/spark/src/main/spark-4.x/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala b/spark/src/main/spark-4.x/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala index c78cc6b2d17..ed4e1236b43 100644 --- a/spark/src/main/spark-4.x/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala +++ b/spark/src/main/spark-4.x/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala @@ -39,9 +39,4 @@ trait ShimCometSparkSessionExtensions { extensions.injectQueryStageOptimizerRule(_ => rule) } - // No-op on Spark >= 3.5. See the Spark 3.4 shim and - // CometSpark34AqeDppFallbackRule's class docstring for why this shim exists. - def injectPreSpark35QueryStagePrepRuleShim( - extensions: SparkSessionExtensions, - rule: Rule[SparkPlan]): Unit = {} } diff --git a/spark/src/main/spark-4.x/org/apache/comet/shims/ShimSQLConf.scala b/spark/src/main/spark-4.x/org/apache/comet/shims/ShimSQLConf.scala index 3157889b431..023340a5055 100644 --- a/spark/src/main/spark-4.x/org/apache/comet/shims/ShimSQLConf.scala +++ b/spark/src/main/spark-4.x/org/apache/comet/shims/ShimSQLConf.scala @@ -26,10 +26,9 @@ trait ShimSQLConf { protected val CORRECTED = LegacyBehaviorPolicy.CORRECTED /** - * Reads `spark.sql.execution.arrow.useLargeVarTypes`. Spark 4.x exposes a typed accessor; 3.4 - * lacks it (a 3.5 backport added it, but Comet's 3.x shim collapses both into a single string - * fallback). Forward to the accessor here so callers do not depend on which version they're - * compiled against. + * Reads `spark.sql.execution.arrow.useLargeVarTypes`. Spark 4.x exposes a typed accessor while + * Comet's 3.x shim uses a string fallback. Forward to the accessor here so callers do not + * depend on which version they're compiled against. */ protected def arrowUseLargeVarTypes(conf: SQLConf): Boolean = conf.arrowUseLargeVarTypes } diff --git a/spark/src/test/resources/pyspark/benchmark_pyarrow_udf.py b/spark/src/test/resources/pyspark/benchmark_pyarrow_udf.py index 19d9dac5d32..d52a245ff99 100644 --- a/spark/src/test/resources/pyspark/benchmark_pyarrow_udf.py +++ b/spark/src/test/resources/pyspark/benchmark_pyarrow_udf.py @@ -20,7 +20,7 @@ End-to-end wall-clock benchmark for Comet's PyArrow UDF acceleration. Requires PySpark 4.0.1+ (Comet's columnar runner targets Spark 4.0+ only; -3.5 and 3.4 are documented no-ops). +3.5 is a documented no-op). Times `df.mapInArrow(passthrough, schema).count()` and the equivalent `mapInPandas` query with `spark.comet.exec.pyarrowUDF.enabled` set diff --git a/spark/src/test/resources/sql-tests/expressions/aggregate/collect_list.sql b/spark/src/test/resources/sql-tests/expressions/aggregate/collect_list.sql index e00d2b19696..31ea3e4427a 100644 --- a/spark/src/test/resources/sql-tests/expressions/aggregate/collect_list.sql +++ b/spark/src/test/resources/sql-tests/expressions/aggregate/collect_list.sql @@ -408,7 +408,7 @@ SELECT sort_array(collect_list(a)), sort_array(collect_list(b)) FROM cl_src_1764 -- FILTER (WHERE ...): only rows that pass the predicate are collected. -- The Spark analyzer routes this through `applyIgnoreNulls`, which the -- collect_list serde handler is expected to accept as long as ignoreNulls --- remains at its default (true on 3.4-4.1; explicit on 4.2). +-- remains at its default (true through 4.1; explicit on 4.2). -- ============================================================ query diff --git a/spark/src/test/resources/sql-tests/expressions/array/explode.sql b/spark/src/test/resources/sql-tests/expressions/array/explode.sql index 68a2bcc9434..abc7a224316 100644 --- a/spark/src/test/resources/sql-tests/expressions/array/explode.sql +++ b/spark/src/test/resources/sql-tests/expressions/array/explode.sql @@ -342,7 +342,7 @@ INSERT INTO test_explode_map VALUES query expect_fallback(Comet only supports explode/explode_outer for arrays, not maps) SELECT id, explode_outer(m) FROM test_explode_map --- ===== TIMESTAMP_NTZ (Spark 3.4+): distinct Arrow layout (no tz) from LTZ ===== +-- ===== TIMESTAMP_NTZ: distinct Arrow layout (no tz) from LTZ ===== statement CREATE TABLE test_explode_ts_ntz(id int, arr array) USING parquet diff --git a/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql b/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql index ceb29e5a395..53c93950af4 100644 --- a/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql +++ b/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql @@ -301,7 +301,7 @@ SELECT sort_array(arr) FROM test_sort_array_nested_struct query SELECT sort_array(arr, false) FROM test_sort_array_nested_struct --- Non-literal (foldable) ascendingOrder is exercised in sort_array_min_spark_4_0.sql: Spark 3.4/3.5 +-- Non-literal (foldable) ascendingOrder is exercised in sort_array_min_spark_4_0.sql: Spark 3.5 -- reject a non-literal sort order at analysis time, so those cases are gated to Spark 4.0+. -- literal arguments diff --git a/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql b/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql index e3e0f6342c0..9b6aef10135 100644 --- a/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql +++ b/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql @@ -17,7 +17,7 @@ -- ConfigMatrix: parquet.enable.dictionary=false,true --- Spark 3.4/3.5's SortArray requires the ascendingOrder argument to be a boolean literal and +-- Spark 3.5's SortArray requires the ascendingOrder argument to be a boolean literal and -- rejects any other expression at analysis time. Spark 4.0 relaxed this to accept any foldable -- boolean, so the non-literal-ascendingOrder cases below can only be exercised on Spark 4.0+. -- MinSparkVersion: 4.0 diff --git a/spark/src/test/resources/sql-tests/expressions/cast/cast_string_trim.sql b/spark/src/test/resources/sql-tests/expressions/cast/cast_string_trim.sql index c14b893b0c2..27889bb7c4c 100644 --- a/spark/src/test/resources/sql-tests/expressions/cast/cast_string_trim.sql +++ b/spark/src/test/resources/sql-tests/expressions/cast/cast_string_trim.sql @@ -30,7 +30,7 @@ CREATE TABLE cast_trim_pad(name string, pad string) USING parquet -- The padding is materialized here rather than in the queries, so that the cast operands below -- stay inside expressions Comet runs natively. The two multi-byte pads are spelled as casts of --- their UTF-8 bytes: `decode(bin, charset)` resolves to a RuntimeReplaceable on Spark 3.4 and +-- their UTF-8 bytes: `decode(bin, charset)` resolves to a RuntimeReplaceable on Spark 3.5 and -- 4.x, which is not foldable, and an inline table only accepts foldable expressions. statement INSERT INTO cast_trim_pad VALUES diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/make_timestamp_ansi.sql b/spark/src/test/resources/sql-tests/expressions/datetime/make_timestamp_ansi.sql index 16ffa09d8b6..e693e5e917b 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/make_timestamp_ansi.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/make_timestamp_ansi.sql @@ -23,7 +23,7 @@ -- class string `DATETIME_FIELD_OUT_OF_BOUNDS` is NOT preserved when the exception is -- thrown from a task on the executor side (only the wrapped `Job aborted ... Lost task -- ... SparkDateTimeException: ` form is preserved), so we match the JDK --- field names which are stable from Spark 3.4 through 4.x. +-- field names which are stable from Spark 3.5 through 4.x. -- Config: spark.sql.session.timeZone=UTC -- Config: spark.sql.ansi.enabled=true -- Config: spark.comet.exec.scalaUDF.codegen.enabled=true diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/to_unix_timestamp_ansi_spark34.sql b/spark/src/test/resources/sql-tests/expressions/datetime/to_unix_timestamp_ansi_spark34.sql deleted file mode 100644 index 4fed3ff6546..00000000000 --- a/spark/src/test/resources/sql-tests/expressions/datetime/to_unix_timestamp_ansi_spark34.sql +++ /dev/null @@ -1,39 +0,0 @@ --- Licensed to the Apache Software Foundation (ASF) under one --- or more contributor license agreements. See the NOTICE file --- distributed with this work for additional information --- regarding copyright ownership. The ASF licenses this file --- to you under the Apache License, Version 2.0 (the --- "License"); you may not use this file except in compliance --- with the License. You may obtain a copy of the License at --- --- http://www.apache.org/licenses/LICENSE-2.0 --- --- Unless required by applicable law or agreed to in writing, --- software distributed under the License is distributed on an --- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY --- KIND, either express or implied. See the License for the --- specific language governing permissions and limitations --- under the License. - --- ANSI mode: to_unix_timestamp throws on parse failure. Same coverage as --- to_unix_timestamp_ansi.sql, but the expect_error substring targets the JDK --- DateTimeParseException message text because Spark 3.4 does not yet wrap these in the --- CANNOT_PARSE_TIMESTAMP error class (that classification arrived in Spark 3.5). The JDK --- message "Text 'not a date' could not be parsed" is stable on Spark 3.4. --- Config: spark.sql.session.timeZone=UTC --- Config: spark.sql.ansi.enabled=true --- Config: spark.sql.legacy.timeParserPolicy=CORRECTED --- Config: spark.comet.exec.scalaUDF.codegen.enabled=true --- MaxSparkVersion: 3.4 - -query expect_error(could not be parsed) -SELECT to_unix_timestamp('not a date', 'yyyy-MM-dd') - -query expect_error(could not be parsed) -SELECT to_unix_timestamp('2024-13-99', 'yyyy-MM-dd') - --- Sentinel: confirms Comet ran the expression natively. If the dispatcher silently rejects --- ToUnixTimestamp, the error queries above pass vacuously via Spark fallback. This valid --- query uses `checkSparkAnswerAndOperator` and fails if Comet did not execute it natively. -query -SELECT to_unix_timestamp('2024-06-15 10:30:45', 'yyyy-MM-dd HH:mm:ss') diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy.sql b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy.sql index 99725e8b6f9..40477d6322b 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy.sql @@ -16,7 +16,6 @@ -- under the License. -- Convergent try_to_timestamp(string, format) behavior across all three policies. --- MinSparkVersion: 3.4 -- ConfigMatrix: spark.sql.legacy.timeParserPolicy=LEGACY,CORRECTED,EXCEPTION -- Config: spark.sql.session.timeZone=UTC diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_corrected.sql b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_corrected.sql index ee2b97968b1..c9a6b0dcdba 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_corrected.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_corrected.sql @@ -17,7 +17,6 @@ -- try_to_timestamp() under CORRECTED timeParserPolicy. -- Strict java.time parsing returns null for inputs that legacy would accept. --- MinSparkVersion: 3.4 -- Config: spark.sql.legacy.timeParserPolicy=CORRECTED -- Config: spark.sql.session.timeZone=UTC diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_exception.sql b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_exception.sql index ef4c4c464e2..9936f7992f3 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_exception.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_exception.sql @@ -19,7 +19,6 @@ -- try_to_timestamp swallows DateTimeException/ParseException, but SparkUpgradeException -- raised by checkParsedDiff propagates through the catch -- so the EXCEPTION case still -- throws on legacy/new divergent inputs. --- MinSparkVersion: 3.4 -- Config: spark.sql.legacy.timeParserPolicy=EXCEPTION -- Config: spark.sql.session.timeZone=UTC diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_legacy.sql b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_legacy.sql index cee5ed4689d..59b2a7f061a 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_legacy.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_legacy.sql @@ -17,7 +17,6 @@ -- try_to_timestamp() under LEGACY timeParserPolicy. -- Lenient SimpleDateFormat parsing accepts inputs that the new formatter rejects. --- MinSparkVersion: 3.4 -- Config: spark.sql.legacy.timeParserPolicy=LEGACY -- Config: spark.sql.session.timeZone=UTC diff --git a/spark/src/test/resources/sql-tests/expressions/misc/aes_cbc.sql b/spark/src/test/resources/sql-tests/expressions/misc/aes_cbc.sql index fb660b9a718..5e86f359e57 100644 --- a/spark/src/test/resources/sql-tests/expressions/misc/aes_cbc.sql +++ b/spark/src/test/resources/sql-tests/expressions/misc/aes_cbc.sql @@ -16,8 +16,7 @@ -- under the License. -- AES-CBC round-trip for aes_encrypt / aes_decrypt. Comet routes the underlying StaticInvoke --- through the JVM codegen dispatcher. AES-CBC was added to Spark in 3.5 (SPARK-43042) and --- throws on 3.4, so this file is gated. +-- through the JVM codegen dispatcher. AES-CBC was added to Spark in 3.5 (SPARK-43042). -- MinSparkVersion: 3.5 statement diff --git a/spark/src/test/resources/sql-tests/expressions/misc/equal_null.sql b/spark/src/test/resources/sql-tests/expressions/misc/equal_null.sql index 7ce2b346461..3cdae409abe 100644 --- a/spark/src/test/resources/sql-tests/expressions/misc/equal_null.sql +++ b/spark/src/test/resources/sql-tests/expressions/misc/equal_null.sql @@ -15,7 +15,6 @@ -- specific language governing permissions and limitations -- under the License. --- MinSparkVersion: 3.4 statement CREATE TABLE test_equal_null(a int, b int) USING parquet diff --git a/spark/src/test/resources/sql-tests/expressions/misc/uuid_with_seed.sql b/spark/src/test/resources/sql-tests/expressions/misc/uuid_with_seed.sql index 260e729f43a..be71c7cfe96 100644 --- a/spark/src/test/resources/sql-tests/expressions/misc/uuid_with_seed.sql +++ b/spark/src/test/resources/sql-tests/expressions/misc/uuid_with_seed.sql @@ -69,7 +69,7 @@ SELECT uuid(42) FROM (SELECT id FROM test_uuid_parts DISTRIBUTE BY id) -- cannot be asserted here: this suite compares Comet against Spark, so if the plan ever collapsed -- to one partition both engines would collapse identically and any check would still pass. The -- partition count is asserted directly in CometUuidExpressionSuite ("across multiple partitions"), --- which checks getNumPartitions > 1 before comparing, and covers 3.4/3.5 as well. +-- which checks getNumPartitions > 1 before comparing. -- Aliased seeded projections: both nodes carry the same explicit seed, so freshCopyIfContainsStatefulExpression -- must not accidentally hoist a shared instance -- each row must satisfy uuid(0) = uuid(0). diff --git a/spark/src/test/resources/sql-tests/expressions/string/base64_unchunked.sql b/spark/src/test/resources/sql-tests/expressions/string/base64_unchunked.sql index 79a252bb868..d4c109b82eb 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/base64_unchunked.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/base64_unchunked.sql @@ -17,8 +17,6 @@ -- Exercises the unchunked base64 path (chunk = false in the native function). -- On Spark 3.5+ this config produces single-line output with no CRLF separators. --- On Spark 3.4 the config does not exist and is ignored, so both engines still chunk; --- the file is harmless there and Comet still matches Spark. -- Config: spark.sql.chunkBase64String.enabled=false statement diff --git a/spark/src/test/resources/sql-tests/expressions/string/decode_invalid_utf8.sql b/spark/src/test/resources/sql-tests/expressions/string/decode_invalid_utf8.sql index 5186a171db5..58bd98fb2c9 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/decode_invalid_utf8.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/decode_invalid_utf8.sql @@ -17,7 +17,7 @@ -- decode() over invalid UTF-8 byte sequences with legacy / replacement-character semantics. -- --- On Spark 3.4 and 3.5 `decode(bin, charset)` always substitutes the Unicode replacement +-- On Spark 3.5 `decode(bin, charset)` always substitutes the Unicode replacement -- character for malformed sequences (it lowers to `new String(bytes, charset)`, which uses the -- JVM's default replace-on-error decoder). -- On Spark 4.0+ the same substitute behavior is selected by enabling both diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q44/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q44/extended.txt deleted file mode 100644 index 0db84abb506..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q44/extended.txt +++ /dev/null @@ -1,64 +0,0 @@ -TakeOrderedAndProject -+- Project - +- BroadcastHashJoin - :- Project - : +- BroadcastHashJoin - : :- Project - : : +- SortMergeJoin - : : :- Sort - : : : +- Project - : : : +- Filter - : : : +- Window - : : : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - : : : +- CometColumnarToRow - : : : +- CometSort - : : : +- CometColumnarExchange - : : : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - : : : +- CometColumnarToRow - : : : +- CometSort - : : : +- CometFilter - : : : : +- Subquery - : : : : +- CometColumnarToRow - : : : : +- CometHashAggregate - : : : : +- CometExchange - : : : : +- CometHashAggregate - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometHashAggregate - : : : +- CometExchange - : : : +- CometHashAggregate - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- Sort - : : +- Project - : : +- Filter - : : +- Window - : : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - : : +- CometColumnarToRow - : : +- CometSort - : : +- CometColumnarExchange - : : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - : : +- CometColumnarToRow - : : +- CometSort - : : +- CometFilter - : : : +- ReusedSubquery - : : +- CometHashAggregate - : : +- CometExchange - : : +- CometHashAggregate - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : +- BroadcastExchange - : +- CometColumnarToRow - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- BroadcastExchange - +- CometColumnarToRow - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item - -Comet accelerated 32 out of 54 eligible operators (59%). Final plan contains 7 transitions between Spark and Comet. Accelerated expressions: 14 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q58/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q58/extended.txt deleted file mode 100644 index 8e467d7f2c1..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q58/extended.txt +++ /dev/null @@ -1,111 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometHashAggregate - : : +- CometExchange - : : +- CometHashAggregate - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometBroadcastHashJoin - : : : : :- CometFilter - : : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : : +- Subquery - : : : : : +- CometColumnarToRow - : : : : : +- CometProject - : : : : : +- CometFilter - : : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.item - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : : +- Subquery - : : : +- CometColumnarToRow - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometFilter - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.catalog_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.item - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : : +- Subquery - : : +- CometColumnarToRow - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometFilter - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.web_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometFilter - : +- Subquery - : +- CometColumnarToRow - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometNativeScan parquet spark_catalog.default.date_dim - -Comet accelerated 98 out of 102 eligible operators (96%). Final plan contains 5 transitions between Spark and Comet. Accelerated expressions: 16 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q67/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q67/extended.txt deleted file mode 100644 index 83193e5e657..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q67/extended.txt +++ /dev/null @@ -1,41 +0,0 @@ -TakeOrderedAndProject -+- Filter - +- Window - +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - +- CometColumnarToRow - +- CometSort - +- CometColumnarExchange - +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - +- CometColumnarToRow - +- CometSort - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometExpand - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item - -Comet accelerated 32 out of 37 eligible operators (86%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q70/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q70/extended.txt deleted file mode 100644 index ed3e7d1b913..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q70/extended.txt +++ /dev/null @@ -1,59 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometWindowExec - +- CometSort - +- CometExchange - +- CometHashAggregate - +- CometColumnarExchange - +- HashAggregate - +- Expand - +- Project - +- BroadcastHashJoin - :- CometColumnarToRow - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometSubqueryBroadcast - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- BroadcastExchange - +- Project - +- BroadcastHashJoin - :- CometColumnarToRow - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- BroadcastExchange - +- Project - +- Filter - +- Window - +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - +- CometColumnarToRow - +- CometSort - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim - -Comet accelerated 40 out of 52 eligible operators (76%). Final plan contains 4 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt deleted file mode 100644 index 9722b079d25..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt +++ /dev/null @@ -1,104 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometHashAggregate - : : +- CometExchange - : : +- CometHashAggregate - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_returns - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometBroadcastHashJoin - : : : : :- CometFilter - : : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometBroadcastHashJoin - : : : : :- CometNativeScan parquet spark_catalog.default.date_dim - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.item - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.catalog_returns - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.item - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.web_returns - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim - -Comet accelerated 99 out of 99 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt index 1f0664d2b1c..0db84abb506 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt @@ -1,60 +1,64 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometSortMergeJoin - : : :- CometSort - : : : +- CometProject - : : : +- CometFilter - : : : +- CometWindowExec - : : : +- CometSort - : : : +- CometExchange - : : : +- CometFilter - : : : : +- Subquery - : : : : +- CometColumnarToRow - : : : : +- CometHashAggregate - : : : : +- CometExchange - : : : : +- CometHashAggregate - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometHashAggregate - : : : +- CometExchange - : : : +- CometHashAggregate - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometSort - : : +- CometProject - : : +- CometFilter - : : +- CometWindowExec - : : +- CometSort - : : +- CometExchange - : : +- CometFilter - : : : +- Subquery - : : : +- CometColumnarToRow - : : : +- CometHashAggregate - : : : +- CometExchange - : : : +- CometHashAggregate - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometHashAggregate - : : +- CometExchange - : : +- CometHashAggregate - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometBroadcastExchange +TakeOrderedAndProject ++- Project + +- BroadcastHashJoin + :- Project + : +- BroadcastHashJoin + : :- Project + : : +- SortMergeJoin + : : :- Sort + : : : +- Project + : : : +- Filter + : : : +- Window + : : : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + : : : +- CometColumnarToRow + : : : +- CometSort + : : : +- CometColumnarExchange + : : : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + : : : +- CometColumnarToRow + : : : +- CometSort + : : : +- CometFilter + : : : : +- Subquery + : : : : +- CometColumnarToRow + : : : : +- CometHashAggregate + : : : : +- CometExchange + : : : : +- CometHashAggregate + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- CometHashAggregate + : : : +- CometExchange + : : : +- CometHashAggregate + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : +- Sort + : : +- Project + : : +- Filter + : : +- Window + : : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + : : +- CometColumnarToRow + : : +- CometSort + : : +- CometColumnarExchange + : : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + : : +- CometColumnarToRow + : : +- CometSort + : : +- CometFilter + : : : +- ReusedSubquery + : : +- CometHashAggregate + : : +- CometExchange + : : +- CometHashAggregate + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store_sales + : +- BroadcastExchange + : +- CometColumnarToRow + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + +- BroadcastExchange + +- CometColumnarToRow +- CometProject +- CometFilter +- CometNativeScan parquet spark_catalog.default.item -Comet accelerated 53 out of 55 eligible operators (96%). Final plan contains 3 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 32 out of 54 eligible operators (59%). Final plan contains 7 transitions between Spark and Comet. Accelerated expressions: 14 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt index 781032c145e..8e467d7f2c1 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt @@ -58,17 +58,7 @@ CometColumnarToRow : : +- CometBroadcastHashJoin : : :- CometFilter : : : +- CometNativeScan parquet spark_catalog.default.catalog_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : : +- ReusedSubquery - : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- ReusedSubquery : : +- CometBroadcastExchange : : +- CometProject : : +- CometFilter @@ -81,7 +71,11 @@ CometColumnarToRow : +- CometBroadcastExchange : +- CometProject : +- CometFilter - : : +- ReusedSubquery + : : +- Subquery + : : +- CometColumnarToRow + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim : +- CometNativeScan parquet spark_catalog.default.date_dim +- CometBroadcastExchange +- CometFilter @@ -107,7 +101,11 @@ CometColumnarToRow +- CometBroadcastExchange +- CometProject +- CometFilter - : +- ReusedSubquery + : +- Subquery + : +- CometColumnarToRow + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.date_dim +- CometNativeScan parquet spark_catalog.default.date_dim -Comet accelerated 102 out of 104 eligible operators (98%). Final plan contains 3 transitions between Spark and Comet. Accelerated expressions: 16 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 98 out of 102 eligible operators (96%). Final plan contains 5 transitions between Spark and Comet. Accelerated expressions: 16 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt index 6a46986af81..83193e5e657 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt @@ -1,37 +1,41 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometFilter - +- CometWindowExec - +- CometSort - +- CometExchange - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometExpand - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item +TakeOrderedAndProject ++- Filter + +- Window + +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + +- CometColumnarToRow + +- CometSort + +- CometColumnarExchange + +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + +- CometColumnarToRow + +- CometSort + +- CometHashAggregate + +- CometExchange + +- CometHashAggregate + +- CometExpand + +- CometProject + +- CometBroadcastHashJoin + :- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- CometSubqueryBroadcast + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- CometBroadcastExchange + +- CometProject + +- CometFilter + +- CometNativeScan parquet spark_catalog.default.item -Comet accelerated 34 out of 34 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 32 out of 37 eligible operators (86%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt index 2402bd8ebcf..ed3e7d1b913 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt @@ -5,51 +5,55 @@ CometColumnarToRow +- CometSort +- CometExchange +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometExpand - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometSubqueryBroadcast - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometWindowExec - +- CometSort - +- CometHashAggregate - +- CometExchange + +- CometColumnarExchange + +- HashAggregate + +- Expand + +- Project + +- BroadcastHashJoin + :- CometColumnarToRow + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : +- CometSubqueryBroadcast + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.date_dim + +- BroadcastExchange + +- Project + +- BroadcastHashJoin + :- CometColumnarToRow + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- BroadcastExchange + +- Project + +- Filter + +- Window + +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + +- CometColumnarToRow + +- CometSort +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim + +- CometExchange + +- CometHashAggregate + +- CometProject + +- CometBroadcastHashJoin + :- CometProject + : +- CometBroadcastHashJoin + : :- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : +- ReusedSubquery + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- CometBroadcastExchange + +- CometProject + +- CometFilter + +- CometNativeScan parquet spark_catalog.default.date_dim -Comet accelerated 51 out of 51 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 40 out of 52 eligible operators (76%). Final plan contains 4 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt index 591d6a51460..9722b079d25 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt @@ -54,20 +54,7 @@ CometColumnarToRow : : +- CometBroadcastHashJoin : : :- CometFilter : : : +- CometNativeScan parquet spark_catalog.default.catalog_returns - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- ReusedSubquery : : +- CometBroadcastExchange : : +- CometProject : : +- CometFilter @@ -114,4 +101,4 @@ CometColumnarToRow +- CometFilter +- CometNativeScan parquet spark_catalog.default.date_dim -Comet accelerated 113 out of 113 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 99 out of 99 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q67a/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q67a/extended.txt deleted file mode 100644 index 6f95655293b..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q67a/extended.txt +++ /dev/null @@ -1,289 +0,0 @@ -TakeOrderedAndProject -+- Filter - +- Window - +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - +- CometColumnarToRow - +- CometSort - +- CometColumnarExchange - +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - +- CometColumnarToRow - +- CometSort - +- CometUnion - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item - -Comet accelerated 280 out of 285 eligible operators (98%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q70a/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q70a/extended.txt deleted file mode 100644 index 25cf0af8d17..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q70a/extended.txt +++ /dev/null @@ -1,168 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometWindowExec - +- CometSort - +- CometExchange - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometUnion - :- CometHashAggregate - : +- CometColumnarExchange - : +- HashAggregate - : +- Project - : +- BroadcastHashJoin - : :- CometColumnarToRow - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- BroadcastExchange - : +- Project - : +- BroadcastHashJoin - : :- CometColumnarToRow - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- BroadcastExchange - : +- Project - : +- Filter - : +- Window - : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - : +- CometColumnarToRow - : +- CometSort - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometColumnarExchange - : +- HashAggregate - : +- Project - : +- BroadcastHashJoin - : :- CometColumnarToRow - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- BroadcastExchange - : +- Project - : +- BroadcastHashJoin - : :- CometColumnarToRow - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- BroadcastExchange - : +- Project - : +- Filter - : +- Window - : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - : +- CometColumnarToRow - : +- CometSort - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometHashAggregate - +- CometColumnarExchange - +- HashAggregate - +- Project - +- BroadcastHashJoin - :- CometColumnarToRow - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometSubqueryBroadcast - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- BroadcastExchange - +- Project - +- BroadcastHashJoin - :- CometColumnarToRow - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- BroadcastExchange - +- Project - +- Filter - +- Window - +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] - +- CometColumnarToRow - +- CometSort - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim - -Comet accelerated 120 out of 153 eligible operators (78%). Final plan contains 10 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt index 501b342bf8c..6f95655293b 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt @@ -1,285 +1,289 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometFilter - +- CometWindowExec - +- CometSort - +- CometExchange - +- CometUnion - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item +TakeOrderedAndProject ++- Filter + +- Window + +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + +- CometColumnarToRow + +- CometSort + +- CometColumnarExchange + +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + +- CometColumnarToRow + +- CometSort + +- CometUnion + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + +- CometHashAggregate + +- CometExchange + +- CometHashAggregate + +- CometHashAggregate + +- CometExchange + +- CometHashAggregate + +- CometProject + +- CometBroadcastHashJoin + :- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- CometSubqueryBroadcast + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- CometBroadcastExchange + +- CometProject + +- CometFilter + +- CometNativeScan parquet spark_catalog.default.item -Comet accelerated 282 out of 282 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 280 out of 285 eligible operators (98%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt index b3435650dae..25cf0af8d17 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt @@ -9,148 +9,160 @@ CometColumnarToRow +- CometHashAggregate +- CometUnion :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometWindowExec - : +- CometSort - : +- CometHashAggregate - : +- CometExchange + : +- CometColumnarExchange + : +- HashAggregate + : +- Project + : +- BroadcastHashJoin + : :- CometColumnarToRow + : : +- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- CometSubqueryBroadcast + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- BroadcastExchange + : +- Project + : +- BroadcastHashJoin + : :- CometColumnarToRow + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- BroadcastExchange + : +- Project + : +- Filter + : +- Window + : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + : +- CometColumnarToRow + : +- CometSort : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- ReusedSubquery + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.date_dim :- CometHashAggregate : +- CometExchange : +- CometHashAggregate : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometWindowExec - : +- CometSort - : +- CometHashAggregate - : +- CometExchange + : +- CometColumnarExchange + : +- HashAggregate + : +- Project + : +- BroadcastHashJoin + : :- CometColumnarToRow + : : +- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- CometSubqueryBroadcast + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- BroadcastExchange + : +- Project + : +- BroadcastHashJoin + : :- CometColumnarToRow + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- BroadcastExchange + : +- Project + : +- Filter + : +- Window + : +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + : +- CometColumnarToRow + : +- CometSort : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- ReusedSubquery + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.date_dim +- CometHashAggregate +- CometExchange +- CometHashAggregate +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometSubqueryBroadcast - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometWindowExec - +- CometSort - +- CometHashAggregate - +- CometExchange + +- CometColumnarExchange + +- HashAggregate + +- Project + +- BroadcastHashJoin + :- CometColumnarToRow + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : +- CometSubqueryBroadcast + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.date_dim + +- BroadcastExchange + +- Project + +- BroadcastHashJoin + :- CometColumnarToRow + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- BroadcastExchange + +- Project + +- Filter + +- Window + +- WindowGroupLimit [COMET: WindowGroupLimit is not supported] + +- CometColumnarToRow + +- CometSort +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim + +- CometExchange + +- CometHashAggregate + +- CometProject + +- CometBroadcastHashJoin + :- CometProject + : +- CometBroadcastHashJoin + : :- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : +- ReusedSubquery + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- CometBroadcastExchange + +- CometProject + +- CometFilter + +- CometNativeScan parquet spark_catalog.default.date_dim -Comet accelerated 150 out of 150 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 120 out of 153 eligible operators (78%). Final plan contains 10 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala index 7377f4fe4fc..ab5699609f2 100644 --- a/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala @@ -30,7 +30,7 @@ import org.apache.spark.sql.functions._ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.ArrayType -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus import org.apache.comet.DataTypeSupport.isComplexType import org.apache.comet.serde.{CometArrayExcept, CometArrayRemove, CometArrayReverse, CometFlatten} import org.apache.comet.testing.{DataGenOptions, ParquetGenerator, SchemaGenOptions} @@ -179,7 +179,6 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp } test("array_prepend") { - assume(isSpark35Plus) // in Spark 3.5 array_prepend is implemented via array_insert withSQLConf(CometConf.getExprAllowIncompatConfigKey(classOf[ArrayInsert]) -> "true") { Seq(true, false).foreach { dictionaryEnabled => withTempDir { dir => @@ -1168,7 +1167,6 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp } test("array_insert on non-null element array from local table scan (#4789)") { - assume(isSpark35Plus) withLocalTableScanNoFold { import testImplicits._ val df = Seq(Seq(1, 2, 3), Seq(4, 5)).toDF("x") diff --git a/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala index 80f92c4b7f5..69ed84d2002 100644 --- a/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala @@ -1713,7 +1713,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { test("null IN empty list honours legacy null-in-empty behavior") { // Spark returns NULL for a NULL operand against an empty IN list when the legacy behavior is - // in effect (SPARK-44550): always on Spark 3.4, on by default on Spark 3.5, and whenever ANSI + // in effect (SPARK-44550): on by default on Spark 3.5 and whenever ANSI // mode is disabled on Spark 4.0+. Comet's native `in` kernel always returns false, so the // legacy case must leave the native path. Empty IN lists are not expressible in SQL and // `OptimizeIn` folds them away, so build the expression via the DataFrame API with that rule @@ -2322,7 +2322,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { CometConf.COMET_EXEC_ENABLED.key -> "true")(f) } - // Spark 3.4/3.5 throw `_LEGACY_ERROR_TEMP_2043` ("- caused overflow.") for byte and + // Spark 3.5 throws `_LEGACY_ERROR_TEMP_2043` ("- caused overflow.") for byte and // short. Spark 4.x routes them through `MathUtils.negateExact` and agrees with Comet, which // always renders `SparkError::ArithmeticOverflow` with the Spark type name. def sparkOverflowMsg(dtype: String): String = diff --git a/spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala b/spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala index 1d7ddfdd5db..84ad4a22a71 100644 --- a/spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala @@ -70,7 +70,7 @@ class CometFuzzTestSuite extends CometFuzzTestBase { val defaultValueType = defaultValueRow.schema.fields(0).dataType.sql // Construct the string for the default value based on the column type. val defaultValueString = defaultValueType match { - // These explicit type definitions for TINYINT, SMALLINT, FLOAT, DOUBLE, and DATE are only needed for 3.4. + // Keep explicit type definitions so generated SQL literals retain the source type. case "TINYINT" | "SMALLINT" => s"$defaultValueType(${defaultValueRow.get(0)})" case "FLOAT" => diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala b/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala index 362f843ccaf..aff147c6385 100644 --- a/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala @@ -38,7 +38,7 @@ import org.apache.spark.sql.execution.exchange.{ReusedExchangeExec, ShuffleExcha import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{StringType, TimestampType} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark42Plus} import org.apache.comet.iceberg.{IcebergReflection, RESTCatalogHelper} import org.apache.comet.serde.OperatorOuterClass import org.apache.comet.testing.{FuzzDataGenerator, SchemaGenOptions} @@ -3238,14 +3238,12 @@ class CometIcebergNativeSuite assert(numPartitions == 1, s"Expected DPP to prune to 1 partition but got $numPartitions") // Verify AQE DPP used CometSubqueryBroadcastExec with broadcast reuse - if (isSpark35Plus) { - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.size == 2, s"Expected 2 DPP subqueries but got ${subqueries.size}") - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.size == 2, s"Expected 2 DPP subqueries but got ${subqueries.size}") + subqueries.foreach { sub => + assert( + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") } spark.sql("DROP TABLE runtime_cat.db.multi_dpp_fact") @@ -3327,11 +3325,9 @@ class CometIcebergNativeSuite assert(numPartitions == 1, s"Expected DPP to prune to 1 partition but got $numPartitions") // Verify AQE DPP used CometSubqueryBroadcastExec with broadcast reuse - if (isSpark35Plus) { - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") - assertCsbBroadcastReuse(subqueries, cometPlan) - } + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") + assertCsbBroadcastReuse(subqueries, cometPlan) spark.sql("DROP TABLE runtime_cat.db.fact_table") } @@ -3794,31 +3790,29 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Verify CometSubqueryBroadcastExec replaced SubqueryAdaptiveBroadcastExec - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } - - // Verify broadcast reuse: subquery child should be an ASPE that contains a - // BroadcastQueryStageExec (via AQE stageCache, possibly wrapped in - // ReusedExchangeExec). Reference equality (eq) on the join's BQS does not - // hold because convertSAB wraps a fresh exchange in a new ASPE; the actual - // reuse manifests as ReusedExchangeExec inside the ASPE's final plan. - assertCsbBroadcastReuse(subqueries, cometPlan) - - // Verify correct results and partition pruning - val icebergScans = collectIcebergNativeScans(cometPlan) - assert(icebergScans.nonEmpty, "Expected CometIcebergNativeScanExec in plan") + // Verify CometSubqueryBroadcastExec replaced SubqueryAdaptiveBroadcastExec + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") + subqueries.foreach { sub => assert( - icebergScans.head.numPartitions == 1, - s"Expected DPP to prune to 1 partition but got ${icebergScans.head.numPartitions}") + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") } + // Verify broadcast reuse: subquery child should be an ASPE that contains a + // BroadcastQueryStageExec (via AQE stageCache, possibly wrapped in + // ReusedExchangeExec). Reference equality (eq) on the join's BQS does not + // hold because convertSAB wraps a fresh exchange in a new ASPE; the actual + // reuse manifests as ReusedExchangeExec inside the ASPE's final plan. + assertCsbBroadcastReuse(subqueries, cometPlan) + + // Verify correct results and partition pruning + val icebergScans = collectIcebergNativeScans(cometPlan) + assert(icebergScans.nonEmpty, "Expected CometIcebergNativeScanExec in plan") + assert( + icebergScans.head.numPartitions == 1, + s"Expected DPP to prune to 1 partition but got ${icebergScans.head.numPartitions}") + spark.sql("DROP TABLE aqe_cat.db.dpp_reuse_fact") } } @@ -3874,22 +3868,20 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Both DPP filters should use CometSubqueryBroadcastExec - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.size == 2, s"Expected 2 DPP subqueries but got ${subqueries.size}") - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } - - // Both should reuse the dim broadcast. Each subquery child is an ASPE that - // contains a BroadcastQueryStageExec (or ReusedExchangeExec) - AQE stageCache - // dedupes via canonical form rather than Java reference identity. - assertCsbBroadcastReuse(subqueries, cometPlan) + // Both DPP filters should use CometSubqueryBroadcastExec + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.size == 2, s"Expected 2 DPP subqueries but got ${subqueries.size}") + subqueries.foreach { sub => + assert( + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") } + // Both should reuse the dim broadcast. Each subquery child is an ASPE that + // contains a BroadcastQueryStageExec (or ReusedExchangeExec) - AQE stageCache + // dedupes via canonical form rather than Java reference identity. + assertCsbBroadcastReuse(subqueries, cometPlan) + spark.sql("DROP TABLE aqe_cat.db.multi_dpp_reuse") } } @@ -3954,41 +3946,39 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Should have DPP subqueries for both joins - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") + // Should have DPP subqueries for both joins + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") - // Each should be CometSubqueryBroadcastExec with an ASPE child wrapping the - // build-side broadcast (reuse manifests as ReusedExchangeExec inside the ASPE). - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } - - val subqueryCsbs = subqueries.collect { case csb: CometSubqueryBroadcastExec => csb } - assertCsbBroadcastReuse(subqueryCsbs, cometPlan) + // Each should be CometSubqueryBroadcastExec with an ASPE child wrapping the + // build-side broadcast (reuse manifests as ReusedExchangeExec inside the ASPE). + subqueries.foreach { sub => + assert( + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") + } - // buildKeys disambiguation: the two DPP subqueries should not share canonical - // form (different join keys -> different broadcasts). Compare canonicalized - // plans rather than Java reference identity. - if (subqueryCsbs.size >= 2) { - val distinctCanonical = subqueryCsbs.map(_.child.canonicalized).distinct - assert( - distinctCanonical.size == subqueryCsbs.size, - s"Expected ${subqueryCsbs.size} distinct broadcasts (by canonical form) " + - s"but got ${distinctCanonical.size}. buildKeys disambiguation may not be working.") - } + val subqueryCsbs = subqueries.collect { case csb: CometSubqueryBroadcastExec => csb } + assertCsbBroadcastReuse(subqueryCsbs, cometPlan) - // Verify correct results: date=2024-01-02 AND category=A returns row (3, 2024-01-02, A, 30) - val icebergScans = collectIcebergNativeScans(cometPlan) - assert(icebergScans.nonEmpty, "Expected CometIcebergNativeScanExec in plan") + // buildKeys disambiguation: the two DPP subqueries should not share canonical + // form (different join keys -> different broadcasts). Compare canonicalized + // plans rather than Java reference identity. + if (subqueryCsbs.size >= 2) { + val distinctCanonical = subqueryCsbs.map(_.child.canonicalized).distinct assert( - icebergScans.head.numPartitions == 1, - s"Expected DPP to prune to 1 partition but got ${icebergScans.head.numPartitions}") + distinctCanonical.size == subqueryCsbs.size, + s"Expected ${subqueryCsbs.size} distinct broadcasts (by canonical form) " + + s"but got ${distinctCanonical.size}. buildKeys disambiguation may not be working.") } + // Verify correct results: date=2024-01-02 AND category=A returns row (3, 2024-01-02, A, 30) + val icebergScans = collectIcebergNativeScans(cometPlan) + assert(icebergScans.nonEmpty, "Expected CometIcebergNativeScanExec in plan") + assert( + icebergScans.head.numPartitions == 1, + s"Expected DPP to prune to 1 partition but got ${icebergScans.head.numPartitions}") + spark.sql("DROP TABLE aqe_cat.db.two_join_fact") } } @@ -4090,14 +4080,12 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Verify the rule still converted the SAB - val subqueries = collectIcebergDPPSubqueries(cometPlan) - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } + // Verify the rule still converted the SAB + val subqueries = collectIcebergDPPSubqueries(cometPlan) + subqueries.foreach { sub => + assert( + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") } spark.sql("DROP TABLE aqe_cat.db.empty_dpp_fact") @@ -4149,9 +4137,8 @@ class CometIcebergNativeSuite // Should produce correct results regardless of DPP path val (_, cometPlan) = checkSparkAnswer(query) assertIcebergNativeScanPresent(cometPlan) - // Even when no BHJ exists, no CSAB should survive the rule. On 3.5+ the rule - // emits TrueLiteral or aggregate SubqueryExec; on 3.4 the wrapper is never - // created in the first place. A leftover CSAB would crash at runtime. + // Even when no BHJ exists, no CSAB should survive the rule. It emits TrueLiteral or an + // aggregate SubqueryExec; a leftover CSAB would crash at runtime. assertNoLeftoverCSAB(cometPlan) spark.sql("DROP TABLE aqe_cat.db.smj_fact") @@ -4225,27 +4212,25 @@ class CometIcebergNativeSuite val (_, cometPlan) = checkSparkAnswer(query) assertIcebergNativeScanPresent(cometPlan) - // No CSAB should survive on either version (would crash at execution otherwise). + // No CSAB should survive (it would crash at execution otherwise). assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Subquery dedup: exactly 1 CometSubqueryBroadcastExec shared across both - // fact scans, plus at least 1 ReusedSubqueryExec pointer. - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.size == 1, - "Expected exactly 1 CometSubqueryBroadcastExec (shared), got " + - s"${cometSubqueries.size}:\n${cometPlan.treeString}") - val reusedCsbs = collectWithSubqueries(cometPlan) { - case r @ ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => r - } - assert( - reusedCsbs.nonEmpty, - "Expected at least one ReusedSubqueryExec(CometSubqueryBroadcastExec):" + - s"\n${cometPlan.treeString}") + // Subquery dedup: exactly 1 CometSubqueryBroadcastExec shared across both + // fact scans, plus at least 1 ReusedSubqueryExec pointer. + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.size == 1, + "Expected exactly 1 CometSubqueryBroadcastExec (shared), got " + + s"${cometSubqueries.size}:\n${cometPlan.treeString}") + val reusedCsbs = collectWithSubqueries(cometPlan) { + case r @ ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => r + } + assert( + reusedCsbs.nonEmpty, + "Expected at least one ReusedSubqueryExec(CometSubqueryBroadcastExec):" + + s"\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.fact1") spark.sql("DROP TABLE aqe_cat.db.fact2") @@ -4318,25 +4303,20 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - // Cross-plan dedup is a known 3.4 limitation (each ASPE sees only its own - // plan at prep-rule time, so the scalar subquery's SAB cannot find the main - // query's BHJ). Strict shape checks only on 3.5+. - if (isSpark35Plus) { - val countBroadcasts = collectWithSubqueries(cometPlan) { - case _: CometSubqueryBroadcastExec => 1 - }.sum - val countReused = collectWithSubqueries(cometPlan) { - case ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => 1 - }.sum + val countBroadcasts = collectWithSubqueries(cometPlan) { + case _: CometSubqueryBroadcastExec => 1 + }.sum + val countReused = collectWithSubqueries(cometPlan) { + case ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => 1 + }.sum - assert( - countBroadcasts == 1, - "Expected 1 CometSubqueryBroadcastExec (shared across plans), " + - s"got $countBroadcasts:\n${cometPlan.treeString}") - assert( - countReused == 1, - s"Expected 1 ReusedSubqueryExec, got $countReused:\n${cometPlan.treeString}") - } + assert( + countBroadcasts == 1, + "Expected 1 CometSubqueryBroadcastExec (shared across plans), " + + s"got $countBroadcasts:\n${cometPlan.treeString}") + assert( + countReused == 1, + s"Expected 1 ReusedSubqueryExec, got $countReused:\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.scalar_fact") } @@ -4459,17 +4439,15 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Two DPP filters from the same join must not duplicate the dim broadcast. - // Count broadcasts across the whole plan including subquery contexts. - val cometBroadcasts = collectWithSubqueries(cometPlan) { - case b: CometBroadcastExchangeExec => b - } - assert( - cometBroadcasts.size == 1, - "Expected exactly 1 CometBroadcastExchangeExec across whole plan, " + - s"got ${cometBroadcasts.size}:\n${cometPlan.treeString}") + // Two DPP filters from the same join must not duplicate the dim broadcast. + // Count broadcasts across the whole plan including subquery contexts. + val cometBroadcasts = collectWithSubqueries(cometPlan) { + case b: CometBroadcastExchangeExec => b } + assert( + cometBroadcasts.size == 1, + "Expected exactly 1 CometBroadcastExchangeExec across whole plan, " + + s"got ${cometBroadcasts.size}:\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.exchg_fact") } @@ -4516,14 +4494,12 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.nonEmpty, - s"Expected CometSubqueryBroadcastExec for DPP, got 0:\n${cometPlan.treeString}") + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.nonEmpty, + s"Expected CometSubqueryBroadcastExec for DPP, got 0:\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.q34637_fact") } @@ -4538,14 +4514,9 @@ class CometIcebergNativeSuite // DynamicPruningExpression(TrueLiteral) - analogous to FileSourceScanExec.doCanonicalize // on V1. // - // The "exactly once" property manifests differently across Spark versions: - // - 3.5+: EnsureRequirements inserts a hash shuffle for SMJ; AQE recognizes the matching - // canonical form on the peer side and emits ReusedExchangeExec -> 1 shuffle, 1 reuse. - // Same shape as V1's CometExecSuite version of this test. - // - 3.4: planner doesn't insert shuffles for this query shape (V2-specific outputPartitioning - // interaction with EnsureRequirements). Result: 0 shuffles. Still "data read once" - just - // via a different mechanism. - // Either shape satisfies the invariant; assertion accepts both. + // EnsureRequirements inserts a hash shuffle for SMJ; AQE recognizes the matching canonical + // form on the peer side and emits ReusedExchangeExec -> 1 shuffle, 1 reuse. This is the same + // shape as V1's CometExecSuite version of this test. test("AQE DPP - unused DPP filter and exchange reuse (SPARK-32509)") { assume(icebergAvailable, "Iceberg not available") withTempIcebergDir { warehouseDir => @@ -4590,8 +4561,7 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - // Accept either shape (see test docstring): single shuffle with reuse, or no - // shuffles at all. Both prove the two scans canonicalize identically after DPP + // A single shuffle plus one reuse proves the two scans canonicalize identically after DPP // degrades to TrueLiteral, so fact data is read exactly once. val shuffleExchanges = collect(cometPlan) { case e: ShuffleExchangeExec => e @@ -4599,13 +4569,10 @@ class CometIcebergNativeSuite } val reusedExchanges = collect(cometPlan) { case r: ReusedExchangeExec => r } - val singleShuffleWithReuse = - shuffleExchanges.size == 1 && reusedExchanges.size == 1 - val noShuffles = shuffleExchanges.isEmpty && reusedExchanges.isEmpty assert( - singleShuffleWithReuse || noShuffles, - "Expected fact data read exactly once: either (1 shuffle + 1 ReusedExchange) " + - s"or (0 shuffles), got (${shuffleExchanges.size} shuffles, " + + shuffleExchanges.size == 1 && reusedExchanges.size == 1, + "Expected fact data read exactly once with 1 shuffle + 1 ReusedExchange, " + + s"got (${shuffleExchanges.size} shuffles, " + s"${reusedExchanges.size} reused):\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.q32509_fact") @@ -4672,22 +4639,20 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - val dpExprs = collect(cometPlan) { case s: CometIcebergNativeScanExec => s } - .flatMap(_.runtimeFilters.collect { case d: DynamicPruningExpression => - d.child - }) - val hasSubquery = dpExprs.exists { - case InSubqueryExec(_, _: SubqueryExec, _, _, _, _) => true - case _ => false - } - val hasBroadcast = dpExprs.exists { - case InSubqueryExec(_, _: CometSubqueryBroadcastExec, _, _, _, _) => true - case _ => false - } - assert(!hasSubquery, s"Should not have SubqueryExec DPP:\n${cometPlan.treeString}") - assert(hasBroadcast, s"Should have broadcast DPP:\n${cometPlan.treeString}") + val dpExprs = collect(cometPlan) { case s: CometIcebergNativeScanExec => s } + .flatMap(_.runtimeFilters.collect { case d: DynamicPruningExpression => + d.child + }) + val hasSubquery = dpExprs.exists { + case InSubqueryExec(_, _: SubqueryExec, _, _, _, _) => true + case _ => false + } + val hasBroadcast = dpExprs.exists { + case InSubqueryExec(_, _: CometSubqueryBroadcastExec, _, _, _, _) => true + case _ => false } + assert(!hasSubquery, s"Should not have SubqueryExec DPP:\n${cometPlan.treeString}") + assert(hasBroadcast, s"Should have broadcast DPP:\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.large") } diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala b/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala index 41f596e5eae..2dcb831ccd1 100644 --- a/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala @@ -37,7 +37,7 @@ import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper import org.apache.spark.sql.types.{DoubleType, IntegerType, StringType, StructField, StructType} import org.apache.spark.sql.util.QueryExecutionListener -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark41Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark41Plus private case class WriteSnapshot(snapshotDelta: Long, plans: Seq[SparkPlan]) @@ -710,8 +710,6 @@ class CometIcebergWriteActionSuite test("CoW MERGE with a NOT MATCHED BY SOURCE leg routes through two-op") { assume(icebergAvailable, "Iceberg not available in classpath") - // Iceberg 1.5.x (the Spark 3.4 pairing) rejects the clause in its extensions parser. - assume(isSpark35Plus, "NOT MATCHED BY SOURCE needs Iceberg 1.8+") withIcebergCatalog { warehouseDir => createTable( warehouseDir, @@ -782,10 +780,9 @@ class CometIcebergWriteActionSuite } } - // On Spark 3.5+ the staged CTAS/RTAS operators run their inner append as its own - // `AppendData` QueryExecution, which IcebergWriteStrategy intercepts like any other append. - // Spark 3.4 writes inline inside the exec (no re-planning), so nothing is intercepted there. - test("CTAS and RTAS write through the split operators on Spark 3.5+") { + // The staged CTAS/RTAS operators run their inner append as its own `AppendData` QueryExecution, + // which IcebergWriteStrategy intercepts like any other append. + test("CTAS and RTAS write through the split operators") { assume(icebergAvailable, "Iceberg not available in classpath") withIcebergCatalog { _ => val session = spark @@ -797,15 +794,9 @@ class CometIcebergWriteActionSuite def assertSplitUsage(plans: Seq[SparkPlan], statement: String): Unit = { val (commits, writes) = collectIcebergWriteOps(plans) - if (isSpark35Plus) { - assert( - commits.nonEmpty && writes.nonEmpty, - s"expected the $statement inner append to plan through the split operators: $plans") - } else { - assert( - commits.isEmpty && writes.isEmpty, - s"expected the $statement write to stay inside Spark's staged exec: $plans") - } + assert( + commits.nonEmpty && writes.nonEmpty, + s"expected the $statement inner append to plan through the split operators: $plans") } val ctasPlans = capturePlans { diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala b/spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala index 71da5160f79..913bcb12b5f 100644 --- a/spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala @@ -28,7 +28,6 @@ import org.apache.spark.SparkConf import org.apache.spark.sql.CometTestBase import org.apache.spark.sql.comet.IcebergWriteExec -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus import org.apache.comet.iceberg.IcebergReflection import org.apache.comet.serde.{Compatible, SupportLevel, Unsupported} import org.apache.comet.serde.operator.CometIcebergNativeWrite @@ -142,7 +141,6 @@ class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTes } test("fall-back: format-version=3") { - assume(isSpark35Plus, "V3 tables require Iceberg 1.8.1+ (Spark 3.5 profile)") withDetectionCatalog { dir => createTable(dir, "v3", partitionSpec = "", properties = Some("'format-version'='3'")) assertUnsupportedContains("v3", "format-version=3") diff --git a/spark/src/test/scala/org/apache/comet/CometMathExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometMathExpressionSuite.scala index fdad9e3d3df..a8c145d1036 100644 --- a/spark/src/test/scala/org/apache/comet/CometMathExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometMathExpressionSuite.scala @@ -26,7 +26,6 @@ import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{DataTypes, StructField, StructType} -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus import org.apache.comet.testing.{DataGenOptions, FuzzDataGenerator} class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { @@ -95,7 +94,6 @@ class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelpe } test("width_bucket") { - assume(isSpark35Plus, "width_bucket was added in Spark 3.5") withSQLConf("spark.comet.exec.localTableScan.enabled" -> "true") { spark .createDataFrame( @@ -108,7 +106,6 @@ class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelpe } test("width_bucket - edge cases") { - assume(isSpark35Plus, "width_bucket was added in Spark 3.5") withSQLConf("spark.comet.exec.localTableScan.enabled" -> "true") { spark .createDataFrame(Seq( @@ -125,7 +122,6 @@ class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelpe } test("width_bucket - NaN values") { - assume(isSpark35Plus, "width_bucket was added in Spark 3.5") withSQLConf("spark.comet.exec.localTableScan.enabled" -> "true") { spark .createDataFrame( @@ -137,7 +133,6 @@ class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelpe } test("width_bucket - with range data") { - assume(isSpark35Plus, "width_bucket was added in Spark 3.5") withSQLConf("spark.comet.exec.localTableScan.enabled" -> "true") { spark .range(10) diff --git a/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala b/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala index 3ac221c1076..d3ccc7be7fa 100644 --- a/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala @@ -2512,7 +2512,7 @@ class CometNativeCastSuite extends CometTestBase with AdaptiveSparkPlanHelper { sparkMessage.startsWith( cometMessage.substring(0, math.min(40, cometMessage.length)))) } else { - // for Spark 3.4 we expect to reproduce the error message exactly + // For Spark 3.x we expect to reproduce the error message exactly. assert(cometMessage == sparkMessage) } } diff --git a/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala index 4df944527b4..6948f6c7d42 100644 --- a/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala @@ -796,7 +796,7 @@ class CometTemporalExpressionSuite extends CometTestBase with AdaptiveSparkPlanH val ntzSchema = StructType(Seq(StructField("ts", DataTypes.TimestampNTZType, true))) val ntzDF = FuzzDataGenerator.generateDataFrame(r, spark, ntzSchema, 1000, DataGenOptions()) - // Spark 3.4: unix_micros() does not accept TIMESTAMP_NTZ; baseline matches micros / hour. + // Baseline matches micros / hour. val _spark = spark import _spark.implicits._ val expectedDF = ntzDF diff --git a/spark/src/test/scala/org/apache/comet/CometUuidExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometUuidExpressionSuite.scala index 56029cba282..0a4b164c1a9 100644 --- a/spark/src/test/scala/org/apache/comet/CometUuidExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometUuidExpressionSuite.scala @@ -28,11 +28,10 @@ import org.apache.spark.sql.functions.col * Bit-for-bit comparisons of seeded `uuid` against Spark. * * The SQL `uuid(seed)` form only exists in Spark 4.0+, so `uuid_with_seed.sql` carries a - * `MinSparkVersion: 4.0` marker and is skipped on 3.4 and 3.5. Unseeded `uuid()` cannot be - * compared across engines. That left the Rust golden constants in - * `nondetermenistic_funcs/uuid.rs` as the only guard on those profiles, and their provenance is - * not checkable from the repo -- if they were ever regenerated from the Rust side the test would - * become circular. + * `MinSparkVersion: 4.0` marker and is skipped on 3.5. Unseeded `uuid()` cannot be compared + * across engines. That left the Rust golden constants in `nondetermenistic_funcs/uuid.rs` as the + * only guard on those profiles, and their provenance is not checkable from the repo -- if they + * were ever regenerated from the Rust side the test would become circular. * * `Uuid(Some(seed))` is constructible from Scala on every supported version even where the SQL * form is not, so building the expression directly gives all profiles a real cross-engine diff --git a/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala b/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala index 3b81a76ead5..1abb7223770 100644 --- a/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala +++ b/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala @@ -36,7 +36,7 @@ class SparkErrorConverterSuite extends AnyFunSuite { .getOrElse(fail("Expected CannotReadFile to be converted to a Spark exception")) // `cannotReadFilesError` IS the FAILED_READ_FILE path. Assert on the version-stable message // ("Encountered error while reading file ...") rather than the `FAILED_READ_FILE` literal, - // which only Spark 4.x prepends to getMessage as the error-class tag (3.4/3.5 do not). + // which only Spark 4.x prepends to getMessage as the error-class tag (3.5 does not). assert(ex.getMessage.contains("Encountered error while reading file")) assert(ex.getMessage.contains("part-0.parquet")) } diff --git a/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala b/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala index d6e42b8833d..8632bf57a00 100644 --- a/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala +++ b/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala @@ -84,7 +84,7 @@ case class ExpectError(pattern: String) extends QueryAssertionMode * Optional minimum Spark version required to run this test (e.g. "3.5"). The test is skipped on * older versions. * @param maxSparkVersion - * Optional maximum Spark version this test applies to (e.g. "3.4"). The test is skipped on + * Optional maximum Spark version this test applies to (e.g. "4.0"). The test is skipped on * newer versions. Useful for paired fixtures where each version range has its own expected * error class or output format. */ diff --git a/spark/src/test/scala/org/apache/comet/exec/CometExec3_4PlusSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometExecCompatibilitySuite.scala similarity index 97% rename from spark/src/test/scala/org/apache/comet/exec/CometExec3_4PlusSuite.scala rename to spark/src/test/scala/org/apache/comet/exec/CometExecCompatibilitySuite.scala index d3ecc789309..aa3698f255a 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometExec3_4PlusSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometExecCompatibilitySuite.scala @@ -36,10 +36,8 @@ import org.apache.spark.util.sketch.BloomFilter import org.apache.comet.CometConf import org.apache.comet.CometSparkSessionExtensions.isSpark42Plus -/** - * This test suite contains tests for only Spark 3.4+. - */ -class CometExec3_4PlusSuite extends CometTestBase { +/** Tests for execution APIs that are available in every supported Spark version. */ +class CometExecCompatibilitySuite extends CometTestBase { import testImplicits._ val func_might_contain = new FunctionIdentifier("might_contain") @@ -83,7 +81,6 @@ class CometExec3_4PlusSuite extends CometTestBase { } } - // The syntax is only supported by Spark 3.4+. test("subquery limit: limit with offset should return correct results") { withSQLConf(CometConf.COMET_SHUFFLE_MODE.key -> "jvm") { withTable("t1", "t2") { @@ -135,7 +132,6 @@ class CometExec3_4PlusSuite extends CometTestBase { } } - // Dataset.offset API is not available before Spark 3.4 test("offset") { withSQLConf(CometConf.COMET_SHUFFLE_MODE.key -> "jvm") { checkSparkAnswer(testData.offset(90)) diff --git a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala index f8c325c35c6..026c2e398a1 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala @@ -49,7 +49,7 @@ import org.apache.spark.sql.internal.SQLConf.SESSION_LOCAL_TIMEZONE import org.apache.spark.unsafe.types.UTF8String import org.apache.comet.{CometConf, CometExecIterator, ExtendedExplainInfo} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark41Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark41Plus, isSpark42Plus} import org.apache.comet.serde.Config.ConfigMap import org.apache.comet.testing.{DataGenOptions, ParquetGenerator, SchemaGenOptions} @@ -878,8 +878,8 @@ class CometExecSuite extends CometTestBase { } } - // On 3.5+, CometPlanAdaptiveDynamicPruningFilters converts SABs to - // CometSubqueryBroadcastExec with broadcast reuse. On 3.4, AQE DPP falls back to Spark. + // CometPlanAdaptiveDynamicPruningFilters converts SABs to + // CometSubqueryBroadcastExec with broadcast reuse. test("AQE DPP: BHJ works with CometNativeScanExec") { withTempDir { path => val factPath = s"${path.getAbsolutePath}/fact.parquet" @@ -906,61 +906,54 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) val infos = new ExtendedExplainInfo().generateExtendedInfo(cometPlan) - if (isSpark35Plus) { - // Verify native scan with DPP - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - val dppScans = nativeScans.filter( - _.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) - assert( - dppScans.nonEmpty, - "Expected at least one CometNativeScanExec with DynamicPruningExpression") + // Verify native scan with DPP + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + val dppScans = + nativeScans.filter(_.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) + assert( + dppScans.nonEmpty, + "Expected at least one CometNativeScanExec with DynamicPruningExpression") - // Verify CometSubqueryBroadcastExec with AdaptiveSparkPlanExec child - // (matches Spark's SubqueryBroadcastExec wrapping an ASPE that goes - // through AQE stageCache for broadcast reuse via ReusedExchangeExec) - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } + // Verify CometSubqueryBroadcastExec with AdaptiveSparkPlanExec child + // (matches Spark's SubqueryBroadcastExec wrapping an ASPE that goes + // through AQE stageCache for broadcast reuse via ReusedExchangeExec) + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s + } + assert( + cometSubqueries.nonEmpty, + "Expected CometSubqueryBroadcastExec for broadcast reuse") + cometSubqueries.foreach { csb => assert( - cometSubqueries.nonEmpty, - "Expected CometSubqueryBroadcastExec for broadcast reuse") - cometSubqueries.foreach { csb => - assert( - csb.child.isInstanceOf[AdaptiveSparkPlanExec], - "Expected AdaptiveSparkPlanExec child but got " + - s"${csb.child.getClass.getSimpleName}") - } - - // Verify broadcast reuse: the subquery's ASPE final plan should contain - // ReusedExchangeExec (AQE stageCache matched the join's broadcast) - import org.apache.spark.sql.execution.exchange.ReusedExchangeExec - cometSubqueries.foreach { csb => - val aspe = csb.child.asInstanceOf[AdaptiveSparkPlanExec] - val hasReusedExchange = collect(aspe) { case r: ReusedExchangeExec => - r - }.nonEmpty || collect(aspe) { case b: BroadcastQueryStageExec => - b - }.nonEmpty - assert( - hasReusedExchange, - "DPP subquery's ASPE should contain ReusedExchangeExec or " + - "BroadcastQueryStageExec for broadcast reuse") - } - - // Verify no unconverted SABs remain - assertAqeDppShape(cometPlan) + csb.child.isInstanceOf[AdaptiveSparkPlanExec], + "Expected AdaptiveSparkPlanExec child but got " + + s"${csb.child.getClass.getSimpleName}") + } - // Verify no fallback - assert( - !infos.contains("AQE Dynamic Partition Pruning"), - s"Should not fall back for AQE DPP:\n$infos") - } else { - // 3.4: scan falls back to Spark so Spark handles DPP natively + // Verify broadcast reuse: the subquery's ASPE final plan should contain + // ReusedExchangeExec (AQE stageCache matched the join's broadcast) + import org.apache.spark.sql.execution.exchange.ReusedExchangeExec + cometSubqueries.foreach { csb => + val aspe = csb.child.asInstanceOf[AdaptiveSparkPlanExec] + val hasReusedExchange = collect(aspe) { case r: ReusedExchangeExec => + r + }.nonEmpty || collect(aspe) { case b: BroadcastQueryStageExec => + b + }.nonEmpty assert( - infos.contains("AQE Dynamic Partition Pruning requires Spark 3.5+"), - s"Expected 3.4 AQE DPP fallback message but got:\n$infos") + hasReusedExchange, + "DPP subquery's ASPE should contain ReusedExchangeExec or " + + "BroadcastQueryStageExec for broadcast reuse") } + + // Verify no unconverted SABs remain + assertAqeDppShape(cometPlan) + + // Verify no fallback + assert( + !infos.contains("AQE Dynamic Partition Pruning"), + s"Should not fall back for AQE DPP:\n$infos") } } } @@ -1000,14 +993,12 @@ class CometExecSuite extends CometTestBase { // Verify no CometSubqueryBroadcastExec — Spark handles DPP with its own // SubqueryBroadcastExec since the join is Spark's BroadcastHashJoinExec - if (isSpark35Plus) { - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.isEmpty, - "Should not have CometSubqueryBroadcastExec when Comet BHJ is disabled") + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.isEmpty, + "Should not have CometSubqueryBroadcastExec when Comet BHJ is disabled") } } } @@ -1040,27 +1031,25 @@ class CometExecSuite extends CometTestBase { "on fact_date = dim_date where dim_id > 7") val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - // Verify native scan is used (DPP disabled via TrueLiteral, but scan still native) - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - - // No CometSubqueryBroadcastExec (DPP was disabled), no unconverted SABs - assertAqeDppShape(cometPlan, expectedCometSubqueryBroadcasts = Some(0)) - - // Case 2 of CometPlanAdaptiveDynamicPruningFilters: SMJ with REUSE_BROADCAST_ONLY - // (Spark's default) sets onlyInBroadcast=true on the SAB. With no reusable - // broadcast, the rule replaces the DPP filter with DynamicPruningExpression( - // Literal.TrueLiteral). This distinguishes Case 2 from Case 3 (aggregate - // SubqueryExec), which would appear as a nested SubqueryExec in the filter. - val trueLiteralFilters = nativeScans.flatMap(_.partitionFilters).collect { - case DynamicPruningExpression(Literal.TrueLiteral) => true - } - assert( - trueLiteralFilters.nonEmpty, - "Expected DynamicPruningExpression(TrueLiteral) for onlyInBroadcast=true SMJ, " + - s"got partitionFilters: ${nativeScans.map(_.partitionFilters).mkString("; ")}") + // Verify native scan is used (DPP disabled via TrueLiteral, but scan still native) + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + + // No CometSubqueryBroadcastExec (DPP was disabled), no unconverted SABs + assertAqeDppShape(cometPlan, expectedCometSubqueryBroadcasts = Some(0)) + + // Case 2 of CometPlanAdaptiveDynamicPruningFilters: SMJ with REUSE_BROADCAST_ONLY + // (Spark's default) sets onlyInBroadcast=true on the SAB. With no reusable + // broadcast, the rule replaces the DPP filter with DynamicPruningExpression( + // Literal.TrueLiteral). This distinguishes Case 2 from Case 3 (aggregate + // SubqueryExec), which would appear as a nested SubqueryExec in the filter. + val trueLiteralFilters = nativeScans.flatMap(_.partitionFilters).collect { + case DynamicPruningExpression(Literal.TrueLiteral) => true } + assert( + trueLiteralFilters.nonEmpty, + "Expected DynamicPruningExpression(TrueLiteral) for onlyInBroadcast=true SMJ, " + + s"got partitionFilters: ${nativeScans.map(_.partitionFilters).mkString("; ")}") } } } @@ -1105,17 +1094,15 @@ class CometExecSuite extends CometTestBase { |WHERE s.store_name = '1' AND r.region_name = '2'""".stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - val dppScans = nativeScans.filter( - _.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) - assert(dppScans.nonEmpty, "Expected DPP filters on native scan") + val dppScans = + nativeScans.filter(_.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) + assert(dppScans.nonEmpty, "Expected DPP filters on native scan") - // Verify no unconverted SABs - assertAqeDppShape(cometPlan) - } + // Verify no unconverted SABs + assertAqeDppShape(cometPlan) } } } @@ -1189,22 +1176,20 @@ class CometExecSuite extends CometTestBase { |WHERE d.country = '3'""".stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - val dppScans = nativeScans.filter( - _.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) - assert(dppScans.nonEmpty, "Expected DPP filter on native scan") + val dppScans = + nativeScans.filter(_.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) + assert(dppScans.nonEmpty, "Expected DPP filter on native scan") - // Verify CometSubqueryBroadcastExec is present (not TrueLiteral fallback) - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.nonEmpty, - "Expected CometSubqueryBroadcastExec (DPP should be active, not TrueLiteral)") + // Verify CometSubqueryBroadcastExec is present (not TrueLiteral fallback) + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.nonEmpty, + "Expected CometSubqueryBroadcastExec (DPP should be active, not TrueLiteral)") } } } @@ -1223,21 +1208,19 @@ class CometExecSuite extends CometTestBase { |ON f.store_id = s.store_id WHERE f.date_id <= 1030""".stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - // Verify no unconverted SABs remain - assertAqeDppShape(cometPlan) + // Verify no unconverted SABs remain + assertAqeDppShape(cometPlan) - // If DPP subqueries are present, verify they use CometSubqueryBroadcastExec - // with AdaptiveSparkPlanExec children (ASPE wrapping broadcast for stageCache reuse) - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - cometSubqueries.foreach { csb => - assert( - csb.child.isInstanceOf[AdaptiveSparkPlanExec], - "CometSubqueryBroadcastExec child should be AdaptiveSparkPlanExec, " + - s"got ${csb.child.getClass.getSimpleName}") - } + // If DPP subqueries are present, verify they use CometSubqueryBroadcastExec + // with AdaptiveSparkPlanExec children (ASPE wrapping broadcast for stageCache reuse) + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s + } + cometSubqueries.foreach { csb => + assert( + csb.child.isInstanceOf[AdaptiveSparkPlanExec], + "CometSubqueryBroadcastExec child should be AdaptiveSparkPlanExec, " + + s"got ${csb.child.getClass.getSimpleName}") } } } @@ -1295,17 +1278,15 @@ class CometExecSuite extends CometTestBase { """.stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.nonEmpty, - s"Expected DPP with CometSubqueryBroadcastExec for $dataType key:\n" + - cometPlan.treeString) - - assertAqeDppShape(cometPlan) + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.nonEmpty, + s"Expected DPP with CometSubqueryBroadcastExec for $dataType key:\n" + + cometPlan.treeString) + + assertAqeDppShape(cometPlan) } } } @@ -1326,12 +1307,10 @@ class CometExecSuite extends CometTestBase { |JOIN dim_store s ON f.store_id = s.store_id AND s.country = 'NL'""".stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - assertAqeDppShape(cometPlan) - } + assertAqeDppShape(cometPlan) } } } @@ -1384,24 +1363,22 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val dpExprs = flatMap(cometPlan) { - case s: CometNativeScanExec => - s.partitionFilters.collect { case d: DynamicPruningExpression => d.child } - case _ => Nil - } - val hasSubquery = dpExprs.exists { - case InSubqueryExec(_, _: SubqueryExec, _, _, _, _) => true - case _ => false - } - val hasBroadcast = dpExprs.exists { - case InSubqueryExec(_, _: SubqueryBroadcastExec, _, _, _, _) => true - case InSubqueryExec(_, _: CometSubqueryBroadcastExec, _, _, _, _) => true - case _ => false - } - assert(!hasSubquery, "Should not have SubqueryExec DPP") - assert(hasBroadcast, "Should have broadcast DPP") + val dpExprs = flatMap(cometPlan) { + case s: CometNativeScanExec => + s.partitionFilters.collect { case d: DynamicPruningExpression => d.child } + case _ => Nil + } + val hasSubquery = dpExprs.exists { + case InSubqueryExec(_, _: SubqueryExec, _, _, _, _) => true + case _ => false + } + val hasBroadcast = dpExprs.exists { + case InSubqueryExec(_, _: SubqueryBroadcastExec, _, _, _, _) => true + case InSubqueryExec(_, _: CometSubqueryBroadcastExec, _, _, _, _) => true + case _ => false } + assert(!hasSubquery, "Should not have SubqueryExec DPP") + assert(hasBroadcast, "Should have broadcast DPP") } } } @@ -1426,19 +1403,17 @@ class CometExecSuite extends CometTestBase { """.stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val countBroadcasts = collectWithSubqueries(cometPlan) { - case _: SubqueryBroadcastExec => 1 - case _: CometSubqueryBroadcastExec => 1 - }.sum - val countReused = collectWithSubqueries(cometPlan) { - case ReusedSubqueryExec(_: SubqueryBroadcastExec) => 1 - case ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => 1 - }.sum - - assert(countBroadcasts == 1, s"Expected 1 SubqueryBroadcast, got $countBroadcasts") - assert(countReused == 1, s"Expected 1 ReusedSubquery, got $countReused") - } + val countBroadcasts = collectWithSubqueries(cometPlan) { + case _: SubqueryBroadcastExec => 1 + case _: CometSubqueryBroadcastExec => 1 + }.sum + val countReused = collectWithSubqueries(cometPlan) { + case ReusedSubqueryExec(_: SubqueryBroadcastExec) => 1 + case ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => 1 + }.sum + + assert(countBroadcasts == 1, s"Expected 1 SubqueryBroadcast, got $countBroadcasts") + assert(countReused == 1, s"Expected 1 ReusedSubquery, got $countReused") } } } @@ -1472,17 +1447,15 @@ class CometExecSuite extends CometTestBase { val (sparkPlan, cometPlan) = checkSparkAnswer(sql(query)) - if (isSpark35Plus) { - val sparkProjects = collectWithSubqueries(sparkPlan) { case p: ProjectExec => p } - val cometProjects = collectWithSubqueries(cometPlan) { - case p: ProjectExec => p - case p: CometProjectExec => p - } - assert( - cometProjects.size == sparkProjects.size, - s"Comet project count (${cometProjects.size}) should match " + - s"Spark (${sparkProjects.size})") + val sparkProjects = collectWithSubqueries(sparkPlan) { case p: ProjectExec => p } + val cometProjects = collectWithSubqueries(cometPlan) { + case p: ProjectExec => p + case p: CometProjectExec => p } + assert( + cometProjects.size == sparkProjects.size, + s"Comet project count (${cometProjects.size}) should match " + + s"Spark (${sparkProjects.size})") } } } @@ -1512,9 +1485,7 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) checkAnswer(df, Nil) - if (isSpark35Plus) { - assertAqeDppShape(cometPlan) - } + assertAqeDppShape(cometPlan) } } } @@ -1562,25 +1533,12 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) checkAnswer(df, Row(15, 15) :: Nil) - if (isSpark35Plus) { - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.nonEmpty, - s"Expected CometSubqueryBroadcastExec for DPP.\nPlan:\n${cometPlan.treeString}") - } else { - // On 3.4, fall back to Spark-native DPP: expect a SubqueryBroadcastExec - // (not Comet) indicating Spark's PlanAdaptiveDynamicPruningFilters ran. - val sparkSubqueries = collectWithSubqueries(cometPlan) { - case s: SubqueryBroadcastExec => s - } - assert( - sparkSubqueries.nonEmpty, - "Expected Spark SubqueryBroadcastExec for DPP on 3.4 fallback. " + - "If empty, Spark's rule killed DPP (likely because Comet BHJ was " + - s"not falling back).\nPlan:\n${cometPlan.treeString}") + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.nonEmpty, + s"Expected CometSubqueryBroadcastExec for DPP.\nPlan:\n${cometPlan.treeString}") } } } @@ -1664,12 +1622,10 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) checkAnswer(df, Row(15, 15) :: Nil) - // SABs should have been unwrapped by CometPlanAdaptiveDynamicPruningFilters on 3.5+. - if (isSpark35Plus) { - assertAqeDppShape(cometPlan) - } + // SABs should have been unwrapped by CometPlanAdaptiveDynamicPruningFilters. + assertAqeDppShape(cometPlan) - if (isSpark35Plus && !isSpark41Plus) { + if (!isSpark41Plus) { // 3.5 - 4.0: CometPlanAdaptiveDynamicPruningFilters rewrites the SAB into // CometSubqueryBroadcastExec with the join's CometBroadcastExchange for native // broadcast reuse. @@ -1680,13 +1636,9 @@ class CometExecSuite extends CometTestBase { cometSubqueries.nonEmpty, s"V2 scan should have CometSubqueryBroadcastExec for DPP:\n$cometPlan") } else { - // 3.4 and 4.1+: DPP runs as Spark-native SubqueryBroadcastExec. - // - 3.4: CometSpark34AqeDppFallbackRule keeps the BHJ build broadcast Spark-native - // so Spark's PlanAdaptiveDynamicPruningFilters can create SubqueryBroadcastExec - // and AQE stageCache can dedupe with the DPP subquery's broadcast. - // - 4.1+: Partial/final aggregate shuffle is elided, which removes Comet's entry - // point for this query, so CometPlanAdaptiveDynamicPruningFilters falls into - // its Spark-native branch. + // On 4.1+, DPP runs as Spark-native SubqueryBroadcastExec. Partial/final aggregate + // shuffle is elided, which removes Comet's entry point for this query, so + // CometPlanAdaptiveDynamicPruningFilters falls into its Spark-native branch. val sparkSubqueries = collectWithSubqueries(cometPlan) { case s: SubqueryBroadcastExec => s } @@ -1777,99 +1729,53 @@ class CometExecSuite extends CometTestBase { """.stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - // Regression check: without the ReusedSubqueryExec unwrap in extractSABData, - // one CSAB survives the rule and trips CometSubqueryAdaptiveBroadcastExec.doExecute - // at runtime. assertAqeDppShape verifies no CSABs remain in the final plan. - assertAqeDppShape(cometPlan) + // Regression check: without the ReusedSubqueryExec unwrap in extractSABData, + // one CSAB survives the rule and trips CometSubqueryAdaptiveBroadcastExec.doExecute + // at runtime. assertAqeDppShape verifies no CSABs remain in the final plan. + assertAqeDppShape(cometPlan) - // Subquery reuse: exactly one canonical CometSubqueryBroadcastExec, plus at - // least one ReusedSubqueryExec(CometSubqueryBroadcastExec) pointer for the - // second fact scan. Without this dedup, both fact scans would evaluate the - // DPP subquery independently. - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.size == 1, - "Expected exactly 1 CometSubqueryBroadcastExec (shared between fact scans), " + - s"got ${cometSubqueries.size}:\n${cometPlan.treeString}") - val reusedCsbs = collectWithSubqueries(cometPlan) { - case r @ ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => r - } - assert( - reusedCsbs.nonEmpty, - "Expected at least one ReusedSubqueryExec(CometSubqueryBroadcastExec) " + - s"for the second fact scan's DPP filter:\n${cometPlan.treeString}") - - // Broadcast reuse via AQE stageCache: the DPP subquery's ASPE and the main - // BHJ should share the same underlying CometBroadcastExchange. Without this, - // we'd build two identical broadcasts of the dim. - val dppBroadcast = cometSubqueries.head.child match { - case aspe: AdaptiveSparkPlanExec => - val bqs = collectFirst(aspe) { case b: BroadcastQueryStageExec => b } - assert( - bqs.isDefined, - "Expected BroadcastQueryStageExec inside DPP subquery's ASPE:\n" + - cometPlan.treeString) - bqs.get.broadcast - case other => - fail( - s"Unexpected CometSubqueryBroadcastExec child: ${other.getClass.getSimpleName}") - } - val hasReuse = find(cometPlan) { - case ReusedExchangeExec(_, e) => e eq dppBroadcast - case b: BroadcastExchangeLike => b eq dppBroadcast - case _ => false - }.isDefined - assert( - hasReuse, - "DPP subquery's broadcast should be reused by the main BHJ " + - s"(via AQE stageCache):\n${cometPlan.treeString}") - } else { - // Spark 3.4: injectQueryStageOptimizerRule is unavailable, so - // CometPlanAdaptiveDynamicPruningFilters can't run. V1 fact scans are rejected - // to Spark by CometScanRule.transformV1Scan, and CometSpark34AqeDppFallbackRule - // tags the BHJ's build-side BroadcastExchange so Spark's own - // PlanAdaptiveDynamicPruningFilters handles DPP natively. Expected shape - // mirrors the 3.5+ assertions but with Spark-native node types. - val sparkSubqueries = collectWithSubqueries(cometPlan) { - case s: SubqueryBroadcastExec => s - } - assert( - sparkSubqueries.size == 1, - "Expected exactly 1 SubqueryBroadcastExec on 3.4 (Spark-native DPP, " + - s"shared between fact scans), got ${sparkSubqueries.size}. If 0, " + - "CometSpark34AqeDppFallbackRule didn't keep the BHJ Spark-native and " + - s"Spark's rule killed DPP:\n${cometPlan.treeString}") - val reusedSparkSubqueries = collectWithSubqueries(cometPlan) { - case r @ ReusedSubqueryExec(_: SubqueryBroadcastExec) => r - } - assert( - reusedSparkSubqueries.nonEmpty, - "Expected at least one ReusedSubqueryExec(SubqueryBroadcastExec) on 3.4 " + - s"for the second fact scan's DPP filter:\n${cometPlan.treeString}") - val dppBroadcast = sparkSubqueries.head.child match { - case aspe: AdaptiveSparkPlanExec => - val bqs = collectFirst(aspe) { case b: BroadcastQueryStageExec => b } - assert( - bqs.isDefined, - "Expected BroadcastQueryStageExec inside DPP subquery's ASPE:\n" + - cometPlan.treeString) - bqs.get.broadcast - case other => - fail(s"Unexpected SubqueryBroadcastExec child: ${other.getClass.getSimpleName}") - } - val hasReuse = find(cometPlan) { - case ReusedExchangeExec(_, e) => e eq dppBroadcast - case b: BroadcastExchangeLike => b eq dppBroadcast - case _ => false - }.isDefined - assert( - hasReuse, - "DPP subquery's broadcast should be reused by the main BHJ on 3.4:\n" + - cometPlan.treeString) + // Subquery reuse: exactly one canonical CometSubqueryBroadcastExec, plus at + // least one ReusedSubqueryExec(CometSubqueryBroadcastExec) pointer for the + // second fact scan. Without this dedup, both fact scans would evaluate the + // DPP subquery independently. + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s + } + assert( + cometSubqueries.size == 1, + "Expected exactly 1 CometSubqueryBroadcastExec (shared between fact scans), " + + s"got ${cometSubqueries.size}:\n${cometPlan.treeString}") + val reusedCsbs = collectWithSubqueries(cometPlan) { + case r @ ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => r + } + assert( + reusedCsbs.nonEmpty, + "Expected at least one ReusedSubqueryExec(CometSubqueryBroadcastExec) " + + s"for the second fact scan's DPP filter:\n${cometPlan.treeString}") + + // Broadcast reuse via AQE stageCache: the DPP subquery's ASPE and the main + // BHJ should share the same underlying CometBroadcastExchange. Without this, + // we'd build two identical broadcasts of the dim. + val dppBroadcast = cometSubqueries.head.child match { + case aspe: AdaptiveSparkPlanExec => + val bqs = collectFirst(aspe) { case b: BroadcastQueryStageExec => b } + assert( + bqs.isDefined, + "Expected BroadcastQueryStageExec inside DPP subquery's ASPE:\n" + + cometPlan.treeString) + bqs.get.broadcast + case other => + fail(s"Unexpected CometSubqueryBroadcastExec child: ${other.getClass.getSimpleName}") } + val hasReuse = find(cometPlan) { + case ReusedExchangeExec(_, e) => e eq dppBroadcast + case b: BroadcastExchangeLike => b eq dppBroadcast + case _ => false + }.isDefined + assert( + hasReuse, + "DPP subquery's broadcast should be reused by the main BHJ " + + s"(via AQE stageCache):\n${cometPlan.treeString}") } } } @@ -2061,8 +1967,6 @@ class CometExecSuite extends CometTestBase { } test("subquery execution under CometTakeOrderedAndProjectExec should not fail") { - assume(isSpark35Plus, "SPARK-45584 is fixed in Spark 3.5+") - withTable("t1") { sql(""" |CREATE TABLE t1 USING PARQUET @@ -2963,11 +2867,10 @@ class CometExecSuite extends CometTestBase { .map(_ => (Random.nextInt(), Random.nextInt() % 5)), "tbl") { - (if (isSpark35Plus) Seq("tinyint", "short", "int", "long", "string") else Seq("long")) - .foreach { input_type => - val df = sql(f"SELECT bloom_filter_agg(cast(_2 as $input_type)) FROM tbl") - checkSparkAnswerAndOperator(df) - } + Seq("tinyint", "short", "int", "long", "string").foreach { input_type => + val df = sql(f"SELECT bloom_filter_agg(cast(_2 as $input_type)) FROM tbl") + checkSparkAnswerAndOperator(df) + } } spark.sessionState.functionRegistry.dropFunction(funcId_bloom_filter_agg) @@ -3741,8 +3644,8 @@ class CometExecSuite extends CometTestBase { df.select("*").groupBy("key", "value").count(), includeClasses = Seq(classOf[CometSparkToColumnarExec])) - // Verify that the BatchScanExec nodes supported columnar output when requested for Spark 3.4+. - // Earlier versions support columnar output for fewer type. + // Verify that the BatchScanExec nodes support columnar output when requested. + // Spark 3.5 supports columnar output for fewer types. val leaves = df.queryExecution.executedPlan.collectLeaves() if (parquetVectorized) { assert(leaves.forall(_.supportsColumnar)) diff --git a/spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala index 2da0a4c4d84..3cd4a4c1f86 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala @@ -294,12 +294,10 @@ class CometJoinSuite extends CometTestBase { sql("SELECT /*+ SHUFFLE_HASH(tbl_a) */ * FROM tbl_a FULL JOIN tbl_b ON tbl_a._2 = tbl_b._1") checkSparkAnswerAndOperator(df3) - // TODO: Spark 3.4 returns SortMergeJoin for this query even with SHUFFLE_HASH hint. - // Left join with build left and right join with build right in hash join is only supported - // in Spark 3.5 or above. See SPARK-36612. - // // Left join: build left - // sql("SELECT /*+ SHUFFLE_HASH(tbl_a) */ * FROM tbl_a LEFT JOIN tbl_b ON tbl_a._2 = tbl_b._1") + val dfLeftBuildLeft = + sql("SELECT /*+ SHUFFLE_HASH(tbl_a) */ * FROM tbl_a LEFT JOIN tbl_b ON tbl_a._2 = tbl_b._1") + checkSparkAnswerAndOperator(dfLeftBuildLeft) // Inner join: build right val df4 = diff --git a/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala b/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala index a1ae1af1d1c..145d0033ecf 100644 --- a/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala +++ b/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala @@ -36,7 +36,6 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.StructType import org.apache.comet.CometConf -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus import org.apache.comet.testing.{DataGenOptions, FuzzDataGenerator, SchemaGenOptions} class CometParquetWriterSuite extends CometTestBase { @@ -217,7 +216,6 @@ class CometParquetWriterSuite extends CometTestBase { } test("parquet write with unsupported compression codec falls back to Spark") { - assume(isSpark35Plus, "lz4_raw was added in Spark 3.5") withTempPath { dir => val outputPath = new File(dir, "output.parquet").getAbsolutePath val df = spark.range(0, 100).selectExpr("id", "cast(id as string) as name") diff --git a/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala b/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala index 5444a89fa36..88b0738b894 100644 --- a/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala +++ b/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala @@ -34,7 +34,7 @@ import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec, ShuffleEx import org.apache.spark.sql.types.{DataTypes, StructField, StructType} import org.apache.comet.{CometConf, CometExplainInfo} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark42Plus} import org.apache.comet.testing.{DataGenOptions, FuzzDataGenerator} /** @@ -380,7 +380,6 @@ class CometExecRuleSuite extends CometTestBase { } test("CometExecRule should not allow try_sum mixed execution") { - assume(isSpark35Plus, "try_sum was added in Spark 3.5") withTempView("test_data") { createTestDataFrame.createOrReplaceTempView("test_data") val sparkPlan = @@ -477,7 +476,7 @@ class CometExecRuleSuite extends CometTestBase { withTempView("test_data") { createTestDataFrame.createOrReplaceTempView("test_data") - // Cast to bigint: Spark 3.4's bloom_filter_agg only accepts a long-typed first + // Cast to bigint so the test uses a type supported by bloom_filter_agg. // argument; later versions widened it to any integral type. val sparkPlan = createSparkPlan(spark, "SELECT bloom_filter_agg(CAST(id AS BIGINT)) FROM test_data") @@ -516,7 +515,7 @@ class CometExecRuleSuite extends CometTestBase { withTempView("test_data") { createTestDataFrame.createOrReplaceTempView("test_data") - // Cast to bigint: Spark 3.4's bloom_filter_agg only accepts a long-typed first + // Cast to bigint so the test uses a type supported by bloom_filter_agg. // argument; later versions widened it to any integral type. val sparkPlan = createSparkPlan(spark, "SELECT bloom_filter_agg(CAST(id AS BIGINT)) FROM test_data") @@ -639,7 +638,7 @@ class CometExecRuleSuite extends CometTestBase { } test("CometExecRule should not convert hash aggregate when grouping key contains map type") { - // Spark 3.4/3.5 reject `array>` as a grouping key in the analyzer (not orderable), + // Spark 3.5 rejects `array>` as a grouping key in the analyzer (not orderable), // so the plan never reaches CometExecRule on those versions. The guard we're exercising // (containsMapType) only matters on Spark 4.0+, which permits the GROUP BY to be analyzed. assume(isSpark40Plus) diff --git a/spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala b/spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala index f59ffa7d632..3c9c8dc90bc 100644 --- a/spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala +++ b/spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala @@ -439,7 +439,7 @@ abstract class CometTestBase // differs across architectures, so a native kernel and Spark can return NaNs that print // identically but compare unequal. Spark fixed this in `compare` itself for 4.1.2 and 4.2.0 // ("in some hardware NaN can be represented with different bits, so first check for it"), but - // 3.4, 3.5, 4.0 and 4.1.1 still compare raw bits. Canonicalize NaN here so every supported + // 3.5, 4.0 and 4.1.1 still compare raw bits. Canonicalize NaN here so every supported // version asserts NaN-ness without asserting the payload. Signed zero is deliberately left // alone: distinguishing it is the reason the raw-bit comparison exists. case f: java.lang.Float => if (f.isNaN) Float.NaN else f.floatValue diff --git a/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala b/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala index 8f309f5cd27..94dff4839d2 100644 --- a/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala +++ b/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala @@ -37,7 +37,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.test.TestSparkSession import org.apache.comet.{CometConf, ExtendedExplainInfo} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark41Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark41Plus, isSpark42Plus} /** * Similar to [[org.apache.spark.sql.PlanStabilitySuite]], checks that TPC-DS Comet plans don't @@ -337,8 +337,7 @@ private object CometPlanStabilitySuite { Seq( isSpark42Plus -> s"approved-plans-${variant}-spark4_2", isSpark41Plus -> s"approved-plans-${variant}-spark4_1", - isSpark40Plus -> s"approved-plans-${variant}-spark4_0", - isSpark35Plus -> s"approved-plans-${variant}-spark3_5") + isSpark40Plus -> s"approved-plans-${variant}-spark4_0") .dropWhile(!_._1) .map(_._2) :+ s"approved-plans-${variant}" } diff --git a/spark/src/test/spark-3.4/org/apache/comet/shims/ShimCometTPCHQuerySuite.scala b/spark/src/test/spark-3.4/org/apache/comet/shims/ShimCometTPCHQuerySuite.scala deleted file mode 100644 index f19f03ff2fd..00000000000 --- a/spark/src/test/spark-3.4/org/apache/comet/shims/ShimCometTPCHQuerySuite.scala +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.shims - -import org.apache.spark.sql.{SparkSession, SQLQueryTestHelper} - -trait ShimCometTPCHQuerySuite extends SQLQueryTestHelper { - protected def getNormalizedQueryExecutionResult( - session: SparkSession, - sql: String): (String, Seq[String]) = { - getNormalizedResult(session, sql) - } -} diff --git a/spark/src/test/spark-3.4/org/apache/spark/sql/ShimCometTestBase.scala b/spark/src/test/spark-3.4/org/apache/spark/sql/ShimCometTestBase.scala deleted file mode 100644 index ce56eeb4364..00000000000 --- a/spark/src/test/spark-3.4/org/apache/spark/sql/ShimCometTestBase.scala +++ /dev/null @@ -1,61 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.spark.sql - -import org.apache.spark.SparkConf -import org.apache.spark.sql.catalyst.expressions.{Expression, MakeDecimal} -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan - -trait ShimCometTestBase { - type SparkSessionType = SparkSession - - def createSparkSessionWithExtensions(conf: SparkConf): SparkSessionType = { - SparkSession - .builder() - .config(conf) - .master("local[1]") - .withExtensions(new org.apache.comet.CometSparkSessionExtensions) - .getOrCreate() - } - - def datasetOfRows(spark: SparkSession, plan: LogicalPlan): DataFrame = { - Dataset.ofRows(spark, plan) - } - - def getColumnFromExpression(expr: Expression): Column = { - new Column(expr) - } - - def extractLogicalPlan(df: DataFrame): LogicalPlan = { - df.logicalPlan - } - - def createMakeDecimalColumn(child: Expression, precision: Int, scale: Int): Column = { - new Column(MakeDecimal(child, precision, scale)) - } - - def createMakeDecimalColumn( - child: Expression, - precision: Int, - scale: Int, - nullOnOverflow: Boolean): Column = { - new Column(MakeDecimal(child, precision, scale, nullOnOverflow)) - } -}