[SPARK-57462][PYTHON][SQL] Add PySpark support for nanosecond-precision timestamp types

[stevomitric]

What changes were proposed in this pull request?

Exposes the nanosecond-capable timestamp types TIMESTAMP_NTZ(p) / TIMESTAMP_LTZ(p) (p in [7, 9]) in PySpark. Before this change python/pyspark/sql/types.py defined only the microsecond singletons, so any DataFrame whose schema contained one of these types failed on the Python side even though the JVM and the Spark Connect protocol already supported them.

• python/pyspark/sql/types.py: adds TimestampNTZNanosType(precision) and TimestampLTZNanosType(precision) (plus the shared AnyTimestampNanoType base, which is not exported), with precision validation, simpleString / jsonValue / __repr__, and toInternal / fromInternal. Registers the parameterized JSON type names, mirroring DataType.parseDataType in sql/api: precision 6 maps to the standard microsecond type, 7-9 to the nanosecond types, and any other precision is rejected. Also registers the two types in _acceptable_types (used by _make_type_verifier, so createDataFrame accepts datetime.datetime values) and in _get_jvm_type_name (so printSchema() renders timestamp_ntz(9) rather than a name derived from the class).
• python/pyspark/sql/connect/types.py: converts the two types to and from the Connect DataType proto in both directions, treating an omitted precision as 9 per types.proto.
• sql/api/.../types/ops/TimestampNanosTypeApiOps.scala: implements the Types Framework Python-interop hooks (needConversionInPython, makeFromJava) so values round-trip over Py4J. Without these, EvaluatePython.makeFromJava fell through to its catch-all and silently produced NULL for every nanosecond column.
• sql/core/.../EvaluatePython.scala: adds the reverse direction, converting the internal TimestampNanosVal to epoch microseconds. Without it the value reached the pickler as a raw TimestampNanosVal, which has no registered pickler.
• python/pyspark/errors/error-conditions.json: adds INVALID_TIMESTAMP_PRECISION, worded to match the JVM error condition of the same name.

The external Python value is datetime.datetime, which is microsecond-resolution, so the Py4J protocol carries epoch microseconds and sub-microsecond digits are truncated at the Python boundary in both directions. This mirrors the shipped TimeType behaviour and is the microsecond-only Python/UDF limitation already documented by SPARK-57808; the stored value keeps full precision. Type inference is unchanged: a bare datetime.datetime still infers microsecond TimestampType, and the nanosecond types are reachable only through an explicit schema.

Arrow and pandas value conversion (toPandas, createDataFrame from pandas, and therefore the Spark Connect data path) is deliberately not included here and remains follow-up work; to_arrow_type continues to reject these types.

Why are the changes needed?

Without Python type classes, a TIMESTAMP(9) column cannot be read or written from PySpark at all: proto_schema_to_pyspark_data_type raised UNSUPPORTED_OPERATION for the Connect schema, and reading df.schema failed because the parameterized JSON type name had no Python parser. This is the last missing client for the umbrella SPARK-56822.

Does this PR introduce any user-facing change?

Yes. TimestampNTZNanosType and TimestampLTZNanosType are new public types in pyspark.sql.types. They are only reachable via an explicit schema or a nanosecond-typed query result, and the server keeps them behind the spark.sql.timestampNanosTypes.enabled preview flag, so no existing behaviour changes.

How was this patch tested?

New tests in python/pyspark/sql/tests/test_types.py:

• DataTypeTests: precision validation (7-9 accepted; -1/0/5/6/10 rejected with INVALID_TIMESTAMP_PRECISION), string representations including printSchema() rendering, equality / hashing / pickling, JSON parsing across precisions including the 6 -> microsecond mapping and the rejected precisions, nested array/map/struct JSON round-trip, and toInternal / fromInternal agreement with the microsecond types.
• DataTypeVerificationTests: accepted and rejected values for both types.
• TypesTestsMixin.test_timestamp_nanos_type: DDL parse agreement with the JVM, plus a createDataFrame / collect round-trip with nulls, and a check that a value stored at nanosecond precision still renders 9 fractional digits server-side while truncating to microseconds when collected as a datetime.

New test in python/pyspark/sql/tests/connect/test_connect_plan.py: DataType proto round-trip for both types across precisions, nested, and with precision omitted.

Was this patch authored or co-authored using generative AI tooling?

Co-authored: Claude Opus 5

[uros-b]

Suggested change

	.. versionadded:: 5.0.0
	.. versionadded:: 4.4.0

[uros-b]

Suggested change

	.. versionadded:: 5.0.0
	.. versionadded:: 4.4.0

[uros-b]

Suggested change

	.. versionadded:: 5.0.0
	.. versionadded:: 4.4.0

[uros-b]

The Python-side TIMESTAMP_NANOS_PYTHON_MAP_KEY fills {"type": self.schema.simpleString()} (the whole DataFrame schema) while the message reads "Cannot convert a map with <type> keys...", so it renders e.g. "a map with struct<m:map<timestamp_ntz(9),int>> keys"; the JVM twin (EvaluatePython.toJava) passes just mt.keyType.sql. Pass the offending key type (or reword) for an accurate, consistent message.