What is the problem the feature request solves?
Comet keeps native columnar-to-row conversion on Spark fallback when a schema contains VariantType. This prevents a native scan or projection that carries Variant from feeding row-based Spark operators through Comet's native C2R path.
Spark does not encode Variant in UnsafeRow as an ordinary nested Struct. VariantVal defines a dedicated payload, and UnsafeWriter.write(VariantVal) writes 4-byte value length + value bytes + metadata bytes.
Comet's current supportsSchema does not admit Variant, while the native C2R initialization path currently derives only the physical Arrow datatype. Treating the [value, metadata] storage as a normal Struct would produce the wrong UnsafeRow representation.
Describe the potential solution
- Preserve explicit Variant identity when initializing native C2R.
- Add a top-level Variant writer that consumes Comet's canonical
[value: Binary, metadata: Binary] array and emits Spark's exact UnsafeRow Variant payload.
- Preserve the distinction between SQL NULL and Variant JSON null, and validate malformed child/null combinations rather than panicking.
- Keep Variant out of generic Struct handling.
Focused parity tests should cover objects, arrays/scalars, Variant JSON null, SQL NULL, nullable parents, empty batches, and fields before/after Variant. The resulting row should expose Spark VariantType and round-trip through UnsafeRow.getVariant with the same value and metadata bytes.
Additional context
Related work: #5407 establishes the native columnar [value, metadata] representation; #5425 tracks Variant-valued native expression output. This issue is only C2R transport.
Nested Variant, shuffle/spill, Python transport, writes, and general Variant expressions remain separate. Spark 3.x behavior must remain unchanged.
What is the problem the feature request solves?
Comet keeps native columnar-to-row conversion on Spark fallback when a schema contains
VariantType. This prevents a native scan or projection that carries Variant from feeding row-based Spark operators through Comet's native C2R path.Spark does not encode Variant in
UnsafeRowas an ordinary nested Struct.VariantValdefines a dedicated payload, andUnsafeWriter.write(VariantVal)writes4-byte value length + value bytes + metadata bytes.Comet's current
supportsSchemadoes not admit Variant, while the native C2R initialization path currently derives only the physical Arrow datatype. Treating the[value, metadata]storage as a normal Struct would produce the wrong UnsafeRow representation.Describe the potential solution
[value: Binary, metadata: Binary]array and emits Spark's exact UnsafeRow Variant payload.Focused parity tests should cover objects, arrays/scalars, Variant JSON null, SQL NULL, nullable parents, empty batches, and fields before/after Variant. The resulting row should expose Spark
VariantTypeand round-trip throughUnsafeRow.getVariantwith the same value and metadata bytes.Additional context
Related work: #5407 establishes the native columnar
[value, metadata]representation; #5425 tracks Variant-valued native expression output. This issue is only C2R transport.Nested Variant, shuffle/spill, Python transport, writes, and general Variant expressions remain separate. Spark 3.x behavior must remain unchanged.