Skip to content

Incorrect AVG result from empty native partial buffers #5418

Description

@sunchao

Describe the bug

An empty native scalar AVG partial currently exports (sum = NULL, count = 0). Spark initializes AVG buffers to (sum = 0, count = 0) and adds partial sums without coalescing nulls. In the supported native-partial / Spark-final path for non-decimal AVG, an empty partial can therefore turn a nonempty query result into NULL.

The same scalar-state distinction matters for decimal AVG: a legitimate empty buffer must be distinguishable from an overflow buffer. A fix must keep the final result of an empty AVG as NULL and preserve decimal overflow behavior.

Steps to reproduce

With Comet installed and enabled in a fresh local Spark session (for example, local[4]), create four separate Parquet scan partitions and filter out all rows in three of them:

import tempfile

for key, value in {
    "spark.sql.adaptive.enabled": "false",
    "spark.sql.files.maxPartitionBytes": "1048576",
    "spark.sql.parquet.filterPushdown": "false",
    "spark.comet.scan.enabled": "false",
    "spark.comet.convert.parquet.enabled": "true",
    "spark.comet.shuffle.enabled": "false",
    "spark.comet.testing.aggregate.finalMode.enabled": "false",
}.items():
    spark.conf.set(key, value)

with tempfile.TemporaryDirectory() as tmp:
    path = f"{tmp}/data"
    (spark.range(8, numPartitions=4)
        .selectExpr("id", "cast(id + 1 as int) quantity")
        .write.parquet(path))
    spark.read.parquet(path).createOrReplaceTempView("avg_partial_probe")
    spark.sql("SELECT avg(quantity) FROM avg_partial_probe WHERE id = 1").show()

The final-mode testing switch isolates the native-partial / Spark-final boundary. The query has a native partial aggregate and a Spark final aggregate, but returns NULL instead of 2.0.

Expected behavior

The result is 2.0, matching Spark. If the filter removes every row, the final result remains NULL.

Additional context

Reproduced against Apache Comet main at 2699f59b71788e17a2714910e166a3f83deed937 using Spark 4.0.4. This is a partial-buffer representation problem, not a request to enable mixed execution for decimal AVG.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions