Describe the bug
An empty native scalar AVG partial currently exports (sum = NULL, count = 0). Spark initializes AVG buffers to (sum = 0, count = 0) and adds partial sums without coalescing nulls. In the supported native-partial / Spark-final path for non-decimal AVG, an empty partial can therefore turn a nonempty query result into NULL.
The same scalar-state distinction matters for decimal AVG: a legitimate empty buffer must be distinguishable from an overflow buffer. A fix must keep the final result of an empty AVG as NULL and preserve decimal overflow behavior.
Steps to reproduce
With Comet installed and enabled in a fresh local Spark session (for example, local[4]), create four separate Parquet scan partitions and filter out all rows in three of them:
import tempfile
for key, value in {
"spark.sql.adaptive.enabled": "false",
"spark.sql.files.maxPartitionBytes": "1048576",
"spark.sql.parquet.filterPushdown": "false",
"spark.comet.scan.enabled": "false",
"spark.comet.convert.parquet.enabled": "true",
"spark.comet.shuffle.enabled": "false",
"spark.comet.testing.aggregate.finalMode.enabled": "false",
}.items():
spark.conf.set(key, value)
with tempfile.TemporaryDirectory() as tmp:
path = f"{tmp}/data"
(spark.range(8, numPartitions=4)
.selectExpr("id", "cast(id + 1 as int) quantity")
.write.parquet(path))
spark.read.parquet(path).createOrReplaceTempView("avg_partial_probe")
spark.sql("SELECT avg(quantity) FROM avg_partial_probe WHERE id = 1").show()
The final-mode testing switch isolates the native-partial / Spark-final boundary. The query has a native partial aggregate and a Spark final aggregate, but returns NULL instead of 2.0.
Expected behavior
The result is 2.0, matching Spark. If the filter removes every row, the final result remains NULL.
Additional context
Reproduced against Apache Comet main at 2699f59b71788e17a2714910e166a3f83deed937 using Spark 4.0.4. This is a partial-buffer representation problem, not a request to enable mixed execution for decimal AVG.
Describe the bug
An empty native scalar AVG partial currently exports
(sum = NULL, count = 0). Spark initializes AVG buffers to(sum = 0, count = 0)and adds partial sums without coalescing nulls. In the supported native-partial / Spark-final path for non-decimal AVG, an empty partial can therefore turn a nonempty query result intoNULL.The same scalar-state distinction matters for decimal AVG: a legitimate empty buffer must be distinguishable from an overflow buffer. A fix must keep the final result of an empty AVG as
NULLand preserve decimal overflow behavior.Steps to reproduce
With Comet installed and enabled in a fresh local Spark session (for example,
local[4]), create four separate Parquet scan partitions and filter out all rows in three of them:The final-mode testing switch isolates the native-partial / Spark-final boundary. The query has a native partial aggregate and a Spark final aggregate, but returns
NULLinstead of2.0.Expected behavior
The result is
2.0, matching Spark. If the filter removes every row, the final result remainsNULL.Additional context
Reproduced against Apache Comet
mainat2699f59b71788e17a2714910e166a3f83deed937using Spark 4.0.4. This is a partial-buffer representation problem, not a request to enable mixed execution for decimal AVG.