Correct outdated information in documentation by nabuskey · Pull Request #3044 · kubeflow/spark-operator

nabuskey · 2026-07-14T23:42:50Z

Purpose of this PR

The documentation needs a lot of love. This corrects some major issues with the documentation site.

Change Category

Bugfix (non-breaking change which fixes an issue)
Feature (non-breaking change which adds functionality)
Breaking change (fix or feature that could affect existing functionality)
Documentation update

Rationale

Checklist

I have conducted a self-review of my own code.
I have updated documentation accordingly.
I have added tests that prove my changes are effective or that my feature works.
Existing unit tests pass locally with my changes.

Additional Notes

Signed-off-by: Manabu McCloskey <manabu.mccloskey@gmail.com>

google-oss-prow · 2026-07-14T23:42:57Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from nabuskey. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot

Pull request overview

Updates documentation to match the current Spark Operator API and Helm chart.

Changes:

Corrects SparkApplication image, dependency, and monitoring guidance.
Fixes the ScheduledSparkApplication time-zone field.
Modernizes Helm configuration, metrics, CRD upgrade, and webhook instructions.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
`docs/website/user-guide/writing-sparkapplication.md`	Updates application specification guidance.
`docs/website/user-guide/using-sparkapplication.md`	Corrects image and init-container documentation.
`docs/website/user-guide/running-sparkapplication-on-schedule.md`	Corrects the `timeZone` field name.
`docs/website/getting-started/index.md`	Revises installation and configuration guidance.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

RobuRishabh · 2026-07-15T14:25:31Z

+| --- | --- | --- |
+| `controller.workers` | Concurrent controller workers | `10` |
+| `controller.uiService.enable` | Create Spark UI Services | `true` |
+| `spark.jobNamespaces` | Explicit namespaces containing Spark workloads | `[default]` |


here this section still says spark.jobNamespaces defaults to [] and that omitting it limits the operator to its own namespace. need to rewrite: default watches default; include "" (or empty list so --namespaces is omitted) for cluster-wide; selector adds more namespaces. Also update the SA/RBAC sentences to match.

RobuRishabh

@nabuskey, the docs looks solid from my side. Some minor nits

RobuRishabh · 2026-07-15T14:25:31Z

+| --- | --- | --- |
+| `controller.workers` | Concurrent controller workers | `10` |
+| `controller.uiService.enable` | Create Spark UI Services | `true` |
+| `spark.jobNamespaces` | Explicit namespaces containing Spark workloads | `[default]` |


here this section still says spark.jobNamespaces defaults to [] and that omitting it limits the operator to its own namespace. need to rewrite: default watches default; include "" (or empty list so --namespaces is omitted) for cluster-wide; selector adds more namespaces. Also update the SA/RBAC sentences to match.

RobuRishabh · 2026-07-15T14:27:56Z

+### CRD upgrades

-To upgrade the operator, e.g., to use a newer version container image with a new tag, run the following command with updated parameters for the Helm release:
+Helm does not automatically upgrade those CRDs. For upgrades that include CRD changes, enable the chart's pre-upgrade hook:


“those CRDs” has no antecedent. we can write something like Helm installs CRDs on first install but does not upgrade them on helm upgrade. For upgrades that include CRD changes, enable …........<continue>.....

RobuRishabh · 2026-07-15T14:42:13Z

+Configure these components with:

-The operator enables cache resynchronization so periodically the informers used by the operator will re-list existing objects it manages and re-trigger resource events. The resynchronization interval in seconds can be configured using the flag `-resync-interval`, with a default value of 30 seconds.
+| Helm value | Purpose | Default |


The table lists 11 values but the chart has 500+ lines of configuration. Consider adding a note like "See values.yaml for the full list of configuration options" or linking to the chart's generated docs, so users know this is a curated subset.

RobuRishabh · 2026-07-15T14:48:19Z

@@ -4,7 +4,7 @@ As with all other Kubernetes API objects, a `SparkApplication` needs the `apiVer

 A `SparkApplication` also needs a [`.spec` section](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status). This section contains fields for specifying various aspects of an application including its type (`Scala`, `Java`, `Python`, or `R`), deployment mode (`cluster` or `client`), main application resource URI (e.g., the URI of the application jar), main class, arguments, etc. Node selectors are also supported via the optional field `.spec.nodeSelector`.


This whole same section is also present in https://github.com/kubeflow/spark-operator/blob/master/docs/website/user-guide/using-sparkapplication.md#:~:text=A%20SparkApplication%20also,spec.nodeSelector. using-sparkapplication.md file, its a duplicate

Not touching this yet. The documentation needs a lot of changes. I'd like to make small changes like this PR.

Signed-off-by: Manabu McCloskey <manabu.mccloskey@gmail.com>

RobuRishabh · 2026-07-17T03:22:25Z


 If you installed the operator using the Helm chart and overrode the `spark.jobNamespaces` to some other, pre-existing namespace, the Helm chart will create the necessary service account and RBAC in the specified namespace.

 The Spark Operator uses the Spark Job Namespace to identify and filter relevant events for the `SparkApplication` CRD. If you specify a namespace for Spark Jobs, and then submit a SparkApplication resource to another namespace, the Spark Operator will filter out the event, and the resource will not get deployed. If you don't specify a namespace, the Spark Operator will see only `SparkApplication` events for the Spark Operator namespace.


If you don't specify a namespace, the Spark Operator will see only SparkApplication events for the Spark Operator namespace.

This line still says that omitting namespaces limits the operator to the Spark Operator namespace. an empty spark.jobNamespaces (or "") makes the controller cluster-wide (--namespaces unset / empty). Need to rewrite to match the controller flag and the empty-list RBAC sentence.

github-actions · 2026-10-15T04:13:16Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

correct outdated information in documentation

e39fd20

Signed-off-by: Manabu McCloskey <manabu.mccloskey@gmail.com>

Copilot AI review requested due to automatic review settings July 14, 2026 23:42

google-oss-prow Bot requested review from ImpSy and RobuRishabh July 14, 2026 23:42

google-oss-prow Bot added the size/L label Jul 14, 2026

Copilot started reviewing on behalf of nabuskey July 14, 2026 23:43

View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

RobuRishabh reviewed Jul 15, 2026

View reviewed changes

clarify job namespaces and helm values

0ac7b6f

Signed-off-by: Manabu McCloskey <manabu.mccloskey@gmail.com>

RobuRishabh reviewed Jul 17, 2026

View reviewed changes

github-actions Bot added the lifecycle/stale label Oct 15, 2026

		@@ -4,7 +4,7 @@ As with all other Kubernetes API objects, a `SparkApplication` needs the `apiVer

		A `SparkApplication` also needs a [`.spec` section](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status). This section contains fields for specifying various aspects of an application including its type (`Scala`, `Java`, `Python`, or `R`), deployment mode (`cluster` or `client`), main application resource URI (e.g., the URI of the application jar), main class, arguments, etc. Node selectors are also supported via the optional field `.spec.nodeSelector`.


		If you installed the operator using the Helm chart and overrode the `spark.jobNamespaces` to some other, pre-existing namespace, the Helm chart will create the necessary service account and RBAC in the specified namespace.

		The Spark Operator uses the Spark Job Namespace to identify and filter relevant events for the `SparkApplication` CRD. If you specify a namespace for Spark Jobs, and then submit a SparkApplication resource to another namespace, the Spark Operator will filter out the event, and the resource will not get deployed. If you don't specify a namespace, the Spark Operator will see only `SparkApplication` events for the Spark Operator namespace.

Conversation

nabuskey commented Jul 14, 2026

Purpose of this PR

Change Category

Rationale

Checklist

Additional Notes

Uh oh!

google-oss-prow Bot commented Jul 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

RobuRishabh Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RobuRishabh left a comment

Choose a reason for hiding this comment

Uh oh!

RobuRishabh Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

RobuRishabh Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RobuRishabh Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

RobuRishabh Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

nabuskey Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

RobuRishabh Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants