What is the problem the feature request solves?
Comet currently falls back when an ordinary native Parquet write has a Spark VariantType column, even when the child already produces a whole Variant value. This prevents native scan-to-write copies such as reading SELECT v FROM parquet_table and writing the result back to Parquet.
Spark 4.1.3 defines the ordinary, unshredded Parquet representation as an annotated group with required binary value and metadata children (schema conversion, value writing). The existing arrow-rs 58.4 writer already maps an Arrow Struct Field carrying the Variant extension marker to the Parquet Variant logical annotation (extension mapping).
This issue is limited to whole-value, unshredded output. #3983 continues to track shredded Variant reader/writer support.
Describe the potential solution
- Allow direct, top-level Variant fields only at the ordinary native Parquet writer boundary when the native child already provides canonical
[value, metadata] storage.
- Preserve
ARROW:extension:name=arrow.parquet.variant, field name/nullability, Binary child types, and child order in the writer schema.
- Reuse the existing protobuf-to-Arrow Field path and
parquet::arrow::ArrowWriter; do not add another Variant dependency or encoder.
- Write a Parquet
VARIANT(1) annotated group containing exactly value and metadata.
Verify Spark and Comet round trips for objects, arrays/scalars, Variant JSON null, SQL NULL, nullable parents, multiple Variant columns, and fields before/after Variant. Assert the plan contains the native writer, the Parquet footer has the Variant annotation, and Spark reads the result as VariantType. Preserve Spark 3.x behavior.
Additional context
Depends on the whole-value scan/Field identity work in #5407. Variant-valued native expression producers from #5425 may later feed this writer but are not required for the scan-to-write case.
Shredded/typed_value output, nested Variant, Variant-producing expressions, Iceberg writes, casts, C2R, shuffle, and spill remain separate.
What is the problem the feature request solves?
Comet currently falls back when an ordinary native Parquet write has a Spark
VariantTypecolumn, even when the child already produces a whole Variant value. This prevents native scan-to-write copies such as readingSELECT v FROM parquet_tableand writing the result back to Parquet.Spark 4.1.3 defines the ordinary, unshredded Parquet representation as an annotated group with required binary
valueandmetadatachildren (schema conversion, value writing). The existing arrow-rs 58.4 writer already maps an Arrow Struct Field carrying the Variant extension marker to the Parquet Variant logical annotation (extension mapping).This issue is limited to whole-value, unshredded output. #3983 continues to track shredded Variant reader/writer support.
Describe the potential solution
[value, metadata]storage.ARROW:extension:name=arrow.parquet.variant, field name/nullability, Binary child types, and child order in the writer schema.parquet::arrow::ArrowWriter; do not add another Variant dependency or encoder.VARIANT(1)annotated group containing exactlyvalueandmetadata.Verify Spark and Comet round trips for objects, arrays/scalars, Variant JSON null, SQL NULL, nullable parents, multiple Variant columns, and fields before/after Variant. Assert the plan contains the native writer, the Parquet footer has the Variant annotation, and Spark reads the result as
VariantType. Preserve Spark 3.x behavior.Additional context
Depends on the whole-value scan/Field identity work in #5407. Variant-valued native expression producers from #5425 may later feed this writer but are not required for the scan-to-write case.
Shredded/
typed_valueoutput, nested Variant, Variant-producing expressions, Iceberg writes, casts, C2R, shuffle, and spill remain separate.