Automate supported Spark versions and publish a Spark image matrix

What feature you would like to be added?

Define a supported Apache Spark version policy, automatically propose updates when new supported Spark patch releases become available, and publish a documented matrix of Spark runtime images for each supported Spark version and architecture.

This issue supersedes the stale-closed #1982 by adding automated version discovery, lifecycle management, verification, and matrix publication rather than relying on manual image builds.

Why is this needed?

Spark Operator releases should not force users onto one Spark version or require every user to maintain custom Spark runtime images. Manual version tracking causes supported images and documentation to drift behind Apache Spark releases. Users also need a clear compatibility matrix showing which Spark Operator releases are tested with which Spark versions and which image tags are available.

Describe the solution you would like

Establish a support policy, for example latest patch releases from the currently supported Spark minor lines.
Keep supported Spark versions in one machine-readable source of truth.
Add scheduled automation that detects new Apache Spark releases and opens reviewed pull requests updating that source, build inputs, tests, and generated documentation.
Build and publish Spark runtime images for every supported Spark version across linux/amd64 and linux/arm64.
Use unambiguous immutable tags containing Spark version and, where needed, Hadoop/Java variants.
Publish a generated compatibility matrix covering Spark Operator version, Spark version, image tag, architecture, Java/Hadoop variant, and support status.
Exercise each supported image in CI with representative SparkApplication tests before publication.
Define deprecation and removal behavior for Spark versions leaving the support window.
Generate provenance, checksums, or signatures according to Kubeflow release standards.

Describe alternatives you have considered

Publish only one Spark image per Spark Operator release. This couples unrelated release cycles and forces upgrades.
Require users to build all Spark images themselves. This creates duplicated work and inconsistent security maintenance.
Maintain multiple images manually. This does not prevent version or documentation drift.

Additional context

Related stale issue: #1982

The repository already contains spark-docker/ and e2e support for loading a Spark image, providing a foundation for matrix builds and validation.

This work should coordinate with release automation tracked by #3063.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

I am thinking the project should generally support:

The default current stable Spark line at the time of operator release
Selected upstream LTS or migration-critical Spark lines as stated in here. (3.5.x falls under "migration-critical Spark lines".)
At most three Spark minor lines concurrently. This means 3.5.x, 4.1.x, and 4.2.x as of now.

This however, adds maintenance burden. We now have to test the operator for all support spark versions. Given spark-submit's stability and Spark's strong compatibility inside a major line, the burden may not be too bad. We probably don't need to test for version compatibility every PR. Weekly or per release tests might be enough.

@nabuskey

@nabuskey in the above example, does that mean we are supporting all version under 3.5 series or only the LTS version. the proplem with LTS version is that its never a fixed one, often bugs tend to shift it and I guess that's what you meant up by maintainers overhead. But happy to support this on ongoing basis as it will make spark version migrations for users very easy. In general. all kubeflow components support multiple versions of kubernetes so we can also support multiple versions of saprk and ci-checks can be added for the same.

That was my thought around the matrix. It's mostly that we have to commit to running the e2e tests 4x- at least until 3.5 is dropped. The scope of the operator hasn't changed I think since the introduction of pod templates?

Yep that's what I meant by maintenance overhead. We need to add testing and promise to providing fixes for version specific bugs.

As of right now, we must support Spark 3.5.x because Spark is still in the transition period and they intend to support it until the end of next year.
Distinguishing between LTS and non LTS is tricky imo. The Spark project does define LTS but I am not sure if this is something we want to define and support in our repository.

@nabuskey

Thanks @nabuskey — the three-rule framing works for me, and I agree with @vjanelle that the real cost here is e2e runtime, not the build. Two things push toward deciding this sooner rather than later:

Spark 4.3.0 is imminent. Per the versioning policy, 4.3.0 had its code freeze and branch cut on Aug 1, with RC voting late August. So within weeks the candidate set is 3.5.x, 4.1.x, 4.2.x and 4.3.x — four lines against a stated cap of three. We need a rule that resolves this automatically rather than a list we hand-edit each quarter.

Our current default isn't in the proposed set. The repo points at apache/spark:4.0.4 in the Makefile, all 14 examples/*.yaml, and config/samples/ (and 3.5.0 in test/e2e/sparkapplication_test.go). 4.0.x isn't among the three lines above, so adopting this policy drops 4.0 as the default on day one. That's user-visible and shouldn't arrive as an incidental generator diff.

Current upstream state: 3.5.9, 4.1.3, 4.0.4, 4.2.0.

Proposed design

1. A source of truth that transcribes upstream rather than defining policy. I am reluctant to own an LTS definition as its a shifting goalpost and I think we can sidestep it: we don't classify anything, we just record upstream's maintenance dates as opaque fields with a source link and apply a selection rule on top.

# Transcribed from https://spark.apache.org/versioning-policy.html
# spark-operator does not define LTS. It records upstream dates and applies a rule.
policy:
  max_concurrent_lines: 3
  rule: >-
    Latest patch of each line whose upstream maintenance window is open,
    preferring the newest stable line and any line with an extended window.

lines:
  - minor: "3.5"
    latest_patch: "3.5.9"
    upstream_maintained_until: "2027-11"   # extended migration window
    upstream_note: "Security fixes only."
    status: supported
  - minor: "4.1"
    latest_patch: "4.1.3"
    status: undecided                      # see question 2 below
    upstream_note: "Feature release, 6-month window — first line to age out."
  - minor: "4.2"
    latest_patch: "4.2.0"
    status: supported
    default: true

A validator fails CI when a window lapses or the list exceeds max_concurrent_lines — mechanical checks, no judgment encoded.

2. Generate everything downstream of it. README matrix, SPARK_IMAGE, example manifests, e2e defaults, and the published compatibility matrix all render from this file, with a drift check that fails on an uncommitted diff — same pattern as make generate and make detect-crds-drift. Today the README matrix stops at v2.3.x while VERSION reads v2.5.0-rc.0, which is exactly the drift this closes.

3. Tiered test cadence, with no added per-PR cost. This is the answer to @vjanelle's concern. Today's e2e matrix is 4 k8s versions × 2 deploy methods = 8 jobs per PR; multiplying by Spark versions gives 24–32, which isn't acceptable. But the multiplication isn't necessary:

Per PR: unchanged — 8 jobs, default Spark line only.
Weekly + workflow_dispatch before releases: Spark version × one k8s version × one deploy method — 3–4 jobs.

Worth noting: the operator has exactly three Spark-version gates in the whole codebase — sparkapplication_validator.go, sparkconnect_validator.go, and two call sites in submission.go — and every one is CompareSemanticVersion(..., "3.0.0") guarding pod templates. @vjanelle's point that the scope hasn't changed since pod templates is literally true in the source, which is why per-PR compat testing looks hard to justify.

4. Image publishing comes later. spark-docker/ is currently orphaned — nothing in CI builds it and its ARG SPARK_IMAGE still says spark:3.5.5. The multi-arch build can reuse the per-arch-digest + imagetools create pattern from release-latest-images.yaml.

Decisions needed

Latest patch per line, or any patch within the line? Suggestion: test and default to the latest patch, but don't block older ones — we have no patch-level logic anyway.
Does 4.1.x stay in the supported set? I've left it undecided above rather than deciding for you. It's a 6-month-window feature release, so it's the first line to age out — but @nabuskey named it explicitly, so I'd rather have the call made here than have a generator quietly drop it.
Three concurrent lines or four? @nabuskey said at most three, @vjanelle mentioned running e2e 4x. Needs resolving before 4.3.0 ships, since that's when the cap actually binds.
Does 4.0.x get a deprecation window? It's the current default. Release note plus one-release grace period, or a clean cut?

Proposed PR sequence

Source of truth + generator + drift check + policy doc
Multi-arch spark-docker/ build and publish
Scheduled workflow opening a PR that bumps only spark-versions.yaml
Tiered e2e matrix
Provenance and signing

Happy to take this on starting with (1) once there's a call on the questions above.

@nabuskey

Hi @nabuskey, @vikas-saxena02! Following up on the image discussion from yesterday's call, I think this issue is probably the right place to continue the discussion rather than opening a separate one.

I've been thinking about the proposal to build and publish our own Spark runtime images, particularly from the security and long-term maintenance perspective.

My initial thought was that owning the image could give the project more control over rebuild cadence, verification, and supply-chain metadata. At the same time, it would also mean taking on an ongoing maintenance commitment: rebuilding, scanning, testing, supporting architectures/Java variants, handling CVEs, and keeping the image aligned with upstream Spark.

One finding that changed how I think about this slightly is the distinction Apache itself makes between the two Spark images:

apache/spark: reviewed and published by the Apache Spark community, but according to apache/spark-docker, a version is generally built and pushed once at release time.
Docker Official spark: uses the same Spark Dockerfile sources, but is maintained through the Docker Official Images process and is actively rebuilt for updates/security fixes.

Ref: https://github.com/apache/spark-docker#about-images

So I think the comparison may be broader than simply:

apache/spark vs a Kubeflow-owned image

There seems to be a useful third baseline:

apache/spark vs Docker Official spark vs a Kubeflow-built image.

From a security perspective, I also think it would help to separate the type of vulnerability we are trying to address.

If the vulnerability is in an OS package or JRE, a Kubeflow-controlled rebuild cadence could improve remediation without Kubeflow having to discover or implement the vulnerability fix itself.

If the vulnerability is inside Spark or one of its bundled JARs, however, rebuilding the same Spark distribution may not solve it. At that point, we would either need to wait for/upgrade to an upstream fix or start carrying downstream patches, which creates a different maintenance problem.

For that reason, my current preference would be to keep any Kubeflow image delta from upstream as small and explicit as possible: potentially own the build/release/security process without gradually creating a Kubeflow-specific Spark distribution.

Vikas, could you clarify what ownership boundary you have in mind for building our own images? (Asking this to better understand the direction myself)

For example, were you thinking primarily about:

building a Kubeflow derivative from an existing upstream Spark image,
building our own runtime image around official Spark release artifacts/base/JRE, or
eventually owning Spark/JAR-level patches as well?

Also, were there particular CVEs or classes of vulnerabilities behind the proposal? Knowing whether the main concern is OS/JRE-level or Spark/dependency-level would help establish what owning the image is expected to solve.

Small experiment before the next call

Before the next meeting, I am considering putting together a small controlled comparison around Spark 4.0.4:

                 Spark 4.0.4
                     |
          +----------+-----------+
          |          |           |
          v          v           v
   apache/spark    spark     kubeflow-proto
          |          |           |
          +----------+-----------+
                     |
                     v
               Same test suite
                     |
        +------------+-------------+
        |            |             |
     Security     Startup       Compatibility
      analysis     measurements       tests

For the first prototype, I would intentionally keep kubeflow-proto close to the upstream Spark 4.0.4 runtime rather than introducing Spark/JAR-level patches or unrelated optimizations. The initial question I want to measure is what we gain or lose by owning the build and image lifecycle itself.

Initial matrix:

Measure	`apache/spark`	`spark` official	Kubeflow prototype
Critical CVEs	?	?	?
High CVEs	?	?	?
Fixable CVEs	?	?	?
Compressed/uncompressed size	?	?	?
Cold image pull	?	?	?
Warm driver startup	?	?	?
Executor readiness	?	?	?
SparkPi	pass/fail	pass/fail	pass/fail
PySpark	pass/fail	pass/fail	pass/fail
Spark Operator e2e	pass/fail	pass/fail	pass/fail
amd64 runtime	pass/fail	pass/fail	pass/fail
arm64 build/runtime	pass/fail	pass/fail	pass/fail

I would keep the Kubernetes side controlled as much as possible: same Spark version, SparkApplication, Kubernetes environment, resource settings, and test suite, changing primarily the runtime image.

For startup measurements, I would also separate cold and warm runs so that registry/network pull time is not confused with Spark/container startup behavior, and repeat the timings rather than relying on a single run.

Beyond the summary matrix, I am thinking of recording:

exact image digests and build/test date
total/fixable vulnerability counts by severity
which component/layer introduces important vulnerabilities
whether findings originate from OS, JRE, Spark/JVM dependencies, or Python packages
base OS, JRE, Python, and package inventory
compressed/uncompressed image size
cold pull and warm pod/driver startup
driver/executor readiness
SparkPi/PySpark compatibility
existing Spark Operator e2e compatibility
amd64 behavior
multi-arch build and, if practical, native arm64 validation
SBOM/provenance/signing feasibility

The goal would not be to argue abstractly that owning an image is "better". I would rather put some evidence behind the trade-off before the project commits to another artifact and lifecycle that maintainers need to support.

Would this experiment be useful before the next call?

And are there Spark Operator-specific requirements that you think should be mandatory in the prototype/benchmark, for example, entrypoint behavior, UID/security context, Java variants, Python/R requirements, Hadoop/cloud connectors, or specific cases from the existing e2e suite?

One long-term maintenance reference I found relevant is Meta's recent write-up on escaping their WebRTC fork:

https://engineering.fb.com/2026/04/09/developer-tools/escaping-the-fork-how-meta-modernized-webrtc-across-50-use-cases/

It is obviously a very different system and scale, but I think the maintenance principle is relevant: keep upstream as the source of truth, keep downstream modifications small and explicit, continuously validate against upstream changes, and avoid turning useful customization into a permanently divergent fork.

If this experiment looks useful, I am happy to take a first pass before the next meeting and post the methodology/results here.

@thisis-Shitanshu

Hi @nabuskey, @vikas-saxena02! Following up on the image discussion from yesterday's call, I think this issue is probably the right place to continue the discussion rather than opening a separate one.

I've been thinking about the proposal to build and publish our own Spark runtime images, particularly from the security and long-term maintenance perspective.

My initial thought was that owning the image could give the project more control over rebuild cadence, verification, and supply-chain metadata. At the same time, it would also mean taking on an ongoing maintenance commitment: rebuilding, scanning, testing, supporting architectures/Java variants, handling CVEs, and keeping the image aligned with upstream Spark.

One finding that changed how I think about this slightly is the distinction Apache itself makes between the two Spark images:

apache/spark: reviewed and published by the Apache Spark community, but according to apache/spark-docker, a version is generally built and pushed once at release time.

Docker Official spark: uses the same Spark Dockerfile sources, but is maintained through the Docker Official Images process and is actively rebuilt for updates/security fixes.

Ref: https://github.com/apache/spark-docker#about-images

So I think the comparison may be broader than simply:

apache/spark vs a Kubeflow-owned image

There seems to be a useful third baseline:

apache/spark vs Docker Official spark vs a Kubeflow-built image.

From a security perspective, I also think it would help to separate the type of vulnerability we are trying to address.

If the vulnerability is in an OS package or JRE, a Kubeflow-controlled rebuild cadence could improve remediation without Kubeflow having to discover or implement the vulnerability fix itself.

If the vulnerability is inside Spark or one of its bundled JARs, however, rebuilding the same Spark distribution may not solve it. At that point, we would either need to wait for/upgrade to an upstream fix or start carrying downstream patches, which creates a different maintenance problem.

For that reason, my current preference would be to keep any Kubeflow image delta from upstream as small and explicit as possible: potentially own the build/release/security process without gradually creating a Kubeflow-specific Spark distribution.

Vikas, could you clarify what ownership boundary you have in mind for building our own images? (Asking this to better understand the direction myself)

For example, were you thinking primarily about:

building a Kubeflow derivative from an existing upstream Spark image,

building our own runtime image around official Spark release artifacts/base/JRE, or

eventually owning Spark/JAR-level patches as well?

Also, were there particular CVEs or classes of vulnerabilities behind the proposal? Knowing whether the main concern is OS/JRE-level or Spark/dependency-level would help establish what owning the image is expected to solve.

Small experiment before the next call

Before the next meeting, I am considering putting together a small controlled comparison around Spark 4.0.4:
                 Spark 4.0.4
                     |
          +----------+-----------+
          |          |           |
          v          v           v
   apache/spark    spark     kubeflow-proto
          |          |           |
          +----------+-----------+
                     |
                     v
               Same test suite
                     |
        +------------+-------------+
        |            |             |
     Security     Startup       Compatibility
      analysis     measurements       tests
For the first prototype, I would intentionally keep kubeflow-proto close to the upstream Spark 4.0.4 runtime rather than introducing Spark/JAR-level patches or unrelated optimizations. The initial question I want to measure is what we gain or lose by owning the build and image lifecycle itself.

Initial matrix:

Measure apache/spark spark official Kubeflow prototype
Critical CVEs ? ? ?
High CVEs ? ? ?
Fixable CVEs ? ? ?
Compressed/uncompressed size ? ? ?
Cold image pull ? ? ?
Warm driver startup ? ? ?
Executor readiness ? ? ?
SparkPi pass/fail pass/fail pass/fail
PySpark pass/fail pass/fail pass/fail
Spark Operator e2e pass/fail pass/fail pass/fail
amd64 runtime pass/fail pass/fail pass/fail
arm64 build/runtime pass/fail pass/fail pass/fail
I would keep the Kubernetes side controlled as much as possible: same Spark version, SparkApplication, Kubernetes environment, resource settings, and test suite, changing primarily the runtime image.

For startup measurements, I would also separate cold and warm runs so that registry/network pull time is not confused with Spark/container startup behavior, and repeat the timings rather than relying on a single run.

Beyond the summary matrix, I am thinking of recording:

exact image digests and build/test date

total/fixable vulnerability counts by severity

which component/layer introduces important vulnerabilities

whether findings originate from OS, JRE, Spark/JVM dependencies, or Python packages

base OS, JRE, Python, and package inventory

compressed/uncompressed image size

cold pull and warm pod/driver startup

driver/executor readiness

SparkPi/PySpark compatibility

existing Spark Operator e2e compatibility

amd64 behavior

multi-arch build and, if practical, native arm64 validation

SBOM/provenance/signing feasibility

The goal would not be to argue abstractly that owning an image is "better". I would rather put some evidence behind the trade-off before the project commits to another artifact and lifecycle that maintainers need to support.

Would this experiment be useful before the next call?

And are there Spark Operator-specific requirements that you think should be mandatory in the prototype/benchmark, for example, entrypoint behavior, UID/security context, Java variants, Python/R requirements, Hadoop/cloud connectors, or specific cases from the existing e2e suite?

One long-term maintenance reference I found relevant is Meta's recent write-up on escaping their WebRTC fork:

https://engineering.fb.com/2026/04/09/developer-tools/escaping-the-fork-how-meta-modernized-webrtc-across-50-use-cases/

It is obviously a very different system and scale, but I think the maintenance principle is relevant: keep upstream as the source of truth, keep downstream modifications small and explicit, continuously validate against upstream changes, and avoid turning useful customization into a permanently divergent fork.

If this experiment looks useful, I am happy to take a first pass before the next meeting and post the methodology/results here.

@thisis-Shitanshu from my perspective, when i scanned the lastest version of the image about a month ago (which is no longer the latest and the current one has quite a few of them fixed), there are following categories of security issues which i am summarising in the table below:

#	Category	What it covers	Ecosystem	Can be fixed by spark-operator team	Introduced by upstream image
1	OS / distro packages	Ubuntu jammy packages installed via `apt` (`gnupg2`, `wget`, `libc6`, `krb5-user`, `libnss3`, `gosu`, …) and their transitive deps	Ubuntu (dpkg)	`tini` only (added in our Dockerfile)	Ubuntu base + all Spark-installed apt packages
2	Go toolchain, binary, deps	Modules in `go.mod`/`go.sum` and the Go toolchain that compiles `/usr/bin/spark-operator`	Go	Entirely ours — bump `go.mod`, bump builder image	Nothing
3	Spark + bundled JARs	Spark's own code plus the ~250 third-party JARs in `/opt/spark/jars` (jackson-databind, netty, avro, commons-*, derby, zookeeper, Hive, kubernetes-client/okhttp)	Maven	Only by overriding individual JARs downstream	All ~250 JARs, incl. Spark's own code
4	JDK / Java runtime	Temurin JDK at `/opt/java/openjdk`, on OpenJDK's quarterly CPU cadence	none standard	Nothing, while we inherit the Spark base	Full JDK via `eclipse-temurin:<n>-jammy`
5	Python runtime + packages	`python3`, `pip`, `setuptools` and pip-installed packages	PyPI	Yes — by switching to a scala-only base tag	python3 + pip via the unsuffixed (python3) tag
6	Bundled JS / frontend	Minified JS/CSS shipped as resources inside `spark-core` under `org/apache/spark/ui/static/` (jQuery, Bootstrap, DataTables, d3)	npm	Nothing, short of stripping JAR resources	UI assets inside `spark-core`
7	Image hardening	Non-CVE findings: setuid binaries, unnecessary tooling, group-writable paths, mutable base tag	n/a	Digest pinning; our own Dockerfile lines	setuid binaries, extra tooling, writable paths

Three notes on the table:

Category 3 is the largest by volume and the noisiest — hadoop-client-api and hadoop-client-runtime are shaded fat JARs, so
scanners attribute CVEs to relocated classes that aren't reachable.
Category 4 has no standard ecosystem because the JDK is an unpacked tarball, not a dpkg package. Package-manager-based scanners may miss it entirely, so an absence of findings here is not evidence of absence.
Category 6 is distinct from 3 despite living inside a JAR: different ecosystem, different advisory sources, and versions frozen at whatever Spark last bumped.

PS:dont focus much on this section, the `Summary` section is important

Summary

Only category 2 is ours today. Go dependencies are fully under our control and already covered by source-level scanning. Every othercategory arrives pre-installed from the upstream Spark image.

Category 5 doesn't need a custom base. The unsuffixed Spark tag is the python3 variant — apache/spark:4.0.2 and
apache/spark:4.0.2-python3 share digest 17dcbf3ceb10. Spark also publishes a scala-only variant, so this category could be dropped by pinning that tag instead. Nothing in the operator pod executes Python the binary is Go, the entrypoint is bash, and PySpark runs in driver/executor pods using the user's spec.image. But again, we are just carrying extra vulnerabilities for something the operator doesn't need.. Much of what we inherit is inert. The operator shells out to spark-submit, which needs a Java runtime, bash, and a subset of the Spark distribution. Alongside that we carry:

python3 and pip — never executed in this pod
Spark Web UI JavaScript — the operator serves no Spark UI
a full JDK rather than a JRE — the base tag is
eclipse-temurin:<n>-jammy, not -jre-jammy, so javac and the
jdk.* tool modules ship with a process that never compiles
Spark runtime tooling — wget, gnupg2, net-tools, gosu,
libnss-wrapper, present for running Spark and for build-time GPG
verification

The question for this issue: stay on the upstream Spark image, or build our own base.

Staying upstream means categories 1, 3, 4, 6 and most of 7 remain outside our control — we inherit whatever the ASF ships and wait for the next Spark release to fix it, including for components we never load. Given Kubeflow is a CNCF graduated project, so I think we need to go minimal on security issues specially hig and critical ones.

Building our own base would put categories 5, 6 and 7 in our hands, along with much of 1 and a meaningful share of 3 and 4. It would also mean owning a rebuild cadence, CVE patching, multi-arch builds, and divergence from the ASF's tested Spark distribution — real ongoing cost that currently sits upstream.

Worth deciding deliberately rather than by default.

cc: @vjanelle @juliusvonkohout @nabuskey @vara-bonthu

Navigation Menu

Repository navigation

Automate supported Spark versions and publish a Spark image matrix #3075

What feature you would like to be added?

Why is this needed?

Describe the solution you would like

Describe alternatives you have considered

Additional context

Love this feature?

Proposed design

Decisions needed

Proposed PR sequence

Small experiment before the next call

Small experiment before the next call

PS:dont focus much on this section, the `Summary` section is important

Summary

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Automate supported Spark versions and publish a Spark image matrix #3075

Description

What feature you would like to be added?

Why is this needed?

Describe the solution you would like

Describe alternatives you have considered

Additional context

Love this feature?

Activity

nabuskey commented on Aug 7, 2026

vikas-saxena02 commented on Aug 8, 2026

vjanelle commented on Aug 8, 2026

nabuskey commented on Aug 9, 2026

vikas-saxena02 commented on Aug 10, 2026

Proposed design

Decisions needed

Proposed PR sequence

thisis-Shitanshu commented on Aug 22, 2026

Small experiment before the next call

vikas-saxena02 commented on Sep 1, 2026

Small experiment before the next call

PS:dont focus much on this section, the Summary section is important

Summary

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

PS:dont focus much on this section, the `Summary` section is important