feat: Add preinstalled mode to e2e test suite by shruthis4 · Pull Request #3045 · kubeflow/spark-operator

shruthis4 · 2026-07-15T13:32:38Z

Summary

Enable e2e tests to run against clusters where the Spark operator is already deployed, without requiring tests to manage operator installation and teardown.

This supports testing workflows on managed Kubernetes clusters (EKS, GKE, RHOAI) where operators are pre-deployed and long-lived.

Changes

Core Features

PREINSTALLED mode: Set PREINSTALLED=true to skip operator install/uninstall
Runtime feature detection: Test suite detects enabled features by inspecting deployment args
- Driver PodDisruptionBudget support (--enable-driver-pdb)
- Namespace selector configuration (--namespace-selector=)
RBAC validation: Verify Spark job RBAC exists before running tests
Configurable deployment location: Override namespace and deployment name for non-standard installations

Environment Variables

Variable	Values	Default	Description
`PREINSTALLED`	`true`, `false`	`false`	Skip operator install/uninstall when true
`DEPLOY_METHOD`	`helm`, `kustomize`	`helm`	Determines webhook names and RBAC scheme
`OPERATOR_NAMESPACE`	any namespace	`spark-operator`	Where to find the operator deployment
`OPERATOR_DEPLOYMENT_NAME`	deployment name	`spark-operator-controller`	Controller deployment name

Test Behavior

Feature detection:

Tests requiring driver PDB skip when --enable-driver-pdb not detected
Tests requiring namespace filtering skip when --namespace-selector not configured
Suite fails early with clear instructions if Spark job RBAC is missing

RBAC requirements:
Preinstalled mode requires complete Spark job RBAC in the default namespace:

ServiceAccount: spark-operator-spark
Role: spark-operator-spark (Helm) or spark-role (Kustomize)
RoleBinding: spark-operator-spark (Helm) or spark-role-binding (Kustomize)

Usage

Preinstalled Mode (Managed Clusters)

# Operator installed via Helm
DEPLOY_METHOD=helm PREINSTALLED=true make e2e-test-preinstalled

# Operator installed via Kustomize
DEPLOY_METHOD=kustomize PREINSTALLED=true make e2e-test-preinstalled

# Custom deployment location
OPERATOR_NAMESPACE=my-namespace \
OPERATOR_DEPLOYMENT_NAME=my-spark-controller \
DEPLOY_METHOD=helm \
PREINSTALLED=true \
make e2e-test-preinstalled

Standard Mode (Unchanged)

# Default: Helm deployment, manages full lifecycle
make e2e-test

# Kustomize deployment
DEPLOY_METHOD=kustomize make e2e-test

Prerequisites for Preinstalled Mode

Operator deployed and running in target cluster

Spark job RBAC exists in default namespace:

# Apply if missing
kubectl apply -k config/spark-rbac -n default

# Or ensure Helm chart has spark.rbac.create=true (default)

KUBECONFIG points to target cluster
DEPLOY_METHOD matches how operator was deployed

Benefits

Faster iteration: Deploy operator once, iterate on tests without reinstalling
CI/CD flexibility: Separate operator deployment from test execution
Pre-production testing: Validate against long-running operator instances
Managed cluster support: Test on EKS, GKE, AKS, RHOAI where operators are pre-deployed

Backward Compatibility

All existing workflows continue to work unchanged:

make e2e-test still manages full operator lifecycle
DEPLOY_METHOD=kustomize make e2e-test still works
Default behavior is non-preinstalled mode

Change Category

Feature (non-breaking change which adds functionality)

Checklist

I have conducted a self-review of my own code.
I have added tests that prove my changes are effective or that my feature works.
Existing unit tests pass locally with my changes.

google-oss-prow · 2026-07-15T13:32:44Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chenyi015 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot

Pull request overview

Adds an e2e mode for testing clusters with an existing Spark operator deployment.

Changes:

Parses PREINSTALLED to skip operator lifecycle management.
Configures webhook names by deployment method.
Applies Spark application RBAC during preinstalled runs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
`test/e2e/suite_test.go`	Implements preinstalled-mode setup and teardown behavior.
`Makefile`	Passes preinstalled configuration to the e2e suite.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

-e2e-test: envtest kind-load-image kind-load-spark-image ## Run the e2e tests against a Kind k8s instance that is spun up.
-	@echo "Running e2e tests (deploy_method=$(DEPLOY_METHOD))..."
-	DEPLOY_METHOD=$(DEPLOY_METHOD) IMAGE_TAG=$(IMAGE_TAG) KUBECONFIG=$(KIND_KUBE_CONFIG) go test ./test/e2e/ -v -ginkgo.v -timeout 30m
+e2e-test: envtest kind-load-image kind-load-spark-image ## Run the e2e tests against a Kind k8s instance. DEPLOY_METHOD: helm (default) or kustomize. PREINSTALLED=true to skip install.


+	export DEPLOY_METHOD="$(DEPLOY_METHOD)" && \
+	export PREINSTALLED="$(PREINSTALLED)" && \
+	export IMAGE_TAG="$(IMAGE_TAG)" && \
+	export KUBECONFIG="$(KIND_KUBE_CONFIG)" && \


+		By("Applying Spark job RBAC to default namespace (preinstalled mode)")
+		repoRoot := filepath.Join("..", "..")
+		sparkRBACDir := filepath.Join(repoRoot, "config", "spark-rbac")
+		rbacCmd := exec.Command("kubectl", "apply", "-k", sparkRBACDir, "-n", "default", "--server-side", "--force-conflicts")


-		installViaHelm()
-	case "kustomize":
-		installViaKustomize()
+	if !preinstalled {


Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

+		// Check if Spark RBAC already exists (may have been created by Helm or Kustomize)
+		By("Checking if Spark RBAC already exists")
+		sa := &corev1.ServiceAccount{}
+		err = k8sClient.Get(context.TODO(),
+			types.NamespacedName{Name: "spark-operator-spark", Namespace: "default"}, sa)
+
+		if err == nil {
+			By("Spark ServiceAccount already exists, skipping RBAC application")
+			// RBAC is already configured by the preinstalled operator
+		} else if errors.IsNotFound(err) {
+			By("Spark ServiceAccount not found, applying RBAC to default namespace")
+			repoRoot := filepath.Join("..", "..")
+			sparkRBACDir := filepath.Join(repoRoot, "config", "spark-rbac")
+			rbacCmd := exec.Command("kubectl", "apply", "-k", sparkRBACDir, "-n", "default")
+			rbacCmd.Stdout = GinkgoWriter
+			rbacCmd.Stderr = GinkgoWriter
+			Expect(rbacCmd.Run()).NotTo(HaveOccurred(), "Failed to apply Spark job RBAC")
+			sparkRBACApplied = true
+		} else {
+			Fail(fmt.Sprintf("Failed to check for Spark ServiceAccount: %v", err))
+		}


+	for _, container := range deployment.Spec.Template.Spec.Containers {
+		for _, arg := range container.Args {
+			if arg == "--enable-driver-pdb=true" {
+				driverPDBEnabled = true
+				break
+			}
+		}
+	}


Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

+			if !driverPDBEnabled {
+				Skip("Driver PDB feature not enabled in operator (--enable-driver-pdb=true required)")


+		if err == nil {
+			By("Spark ServiceAccount already exists, skipping RBAC application")
+			// RBAC is already configured by the preinstalled operator
+		} else if apierrors.IsNotFound(err) {


+			rbacDelCmd := exec.Command("kubectl", "delete", "-k", sparkRBACDir, "-n", "default", "--ignore-not-found", "--timeout=60s")
+			rbacDelCmd.Stdout = GinkgoWriter
+			rbacDelCmd.Stderr = GinkgoWriter
+			_ = rbacDelCmd.Run()


-		installViaHelm()
-	case "kustomize":
-		installViaKustomize()
+	if !preinstalled {


+	deployment := &appsv1.Deployment{}
+	err = k8sClient.Get(context.TODO(),
+		types.NamespacedName{Name: "spark-operator-controller", Namespace: ReleaseNamespace},
+		deployment)
+	Expect(err).NotTo(HaveOccurred(), "Failed to get controller deployment")


+.PHONY: e2e-test-preinstalled
+e2e-test-preinstalled: envtest ## Run the e2e tests against a preinstalled operator. DEPLOY_METHOD: helm (default) or kustomize.
+	@echo "Running e2e tests against preinstalled operator (deploy_method=$(DEPLOY_METHOD))..."
+	export DEPLOY_METHOD="$(DEPLOY_METHOD)" && \
+	export PREINSTALLED="true" && \
+	go test ./test/e2e/ -v -ginkgo.v -timeout 30m


Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

+			if strings.HasPrefix(arg, "--job-namespace-selector=") {
+				jobNamespaceSelectorConfigured = true
+			}


+			By("Cleaning up test-created Spark RBAC from default namespace")
+			repoRoot := filepath.Join("..", "..")
+			sparkRBACDir := filepath.Join(repoRoot, "config", "spark-rbac")
+			rbacDelCmd := exec.Command("kubectl", "delete", "-k", sparkRBACDir, "-n", "default", "--ignore-not-found", "--timeout=60s")


Enable e2e tests to run against clusters where the Spark operator is already deployed (EKS, GKE, RHOAI managed clusters). - Add PREINSTALLED env var to skip operator install/uninstall - Add OPERATOR_NAMESPACE and OPERATOR_DEPLOYMENT_NAME env vars for custom deployments - Detect driver PDB and namespace selector features at runtime from deployment args - Validate Spark RBAC exists based on deployment method (Helm vs Kustomize) - Skip feature-specific tests when features unavailable - Prevent nil panic in pdb_test.go AfterEach when test skipped Environment variables: - PREINSTALLED: Set to 'true' to skip operator lifecycle management - DEPLOY_METHOD: 'helm' or 'kustomize' (default: helm) - determines webhook and RBAC naming - OPERATOR_NAMESPACE: Namespace where operator is deployed (default: spark-operator) - OPERATOR_DEPLOYMENT_NAME: Controller deployment name (default: spark-operator-controller) Requirements for preinstalled mode: - Operator must be deployed and running - Spark job RBAC must exist in 'default' namespace (ServiceAccount, Role, RoleBinding) - Set DEPLOY_METHOD to match how operator was deployed Signed-off-by: Shruthi Sankepelly <shruthi.s4@gmail.com>

RobuRishabh · 2026-07-17T04:40:17Z

+		allMissing := true
+		sa := &corev1.ServiceAccount{}
+		if err := k8sClient.Get(context.TODO(),
+			types.NamespacedName{Name: "spark-operator-spark", Namespace: "default"}, sa); err == nil {
+			allMissing = false // SA exists
+		}
+
+		role := &rbacv1.Role{}
+		if err := k8sClient.Get(context.TODO(),
+			types.NamespacedName{Name: roleName, Namespace: "default"}, role); err == nil {
+			allMissing = false // Role exists
+		}
+
+		roleBinding := &rbacv1.RoleBinding{}
+		if err := k8sClient.Get(context.TODO(),
+			types.NamespacedName{Name: roleBindingName, Namespace: "default"}, roleBinding); err == nil {
+			allMissing = false // RoleBinding exists
+		}
+
+		if allMissing {
+			Fail(fmt.Sprintf("Spark job RBAC is missing in preinstalled mode.\n"+
+				"The e2e suite requires ServiceAccount, Role, and RoleBinding for Spark applications in the 'default' namespace.\n"+
+				"Please apply RBAC before running tests:\n"+
+				"  kubectl apply -k config/spark-rbac -n default\n"+
+				"Or if using Helm, ensure spark.rbac.create=true (default)"))
+		} else {
+			By(fmt.Sprintf("Spark RBAC exists for %s deployment", deployMethod))
+		}
 	}



The allMissing predicate is inverted relative to the comment (“ALL are missing”). Setting allMissing = false when any of SA/Role/RoleBinding exists means a lone ServiceAccount satisfies the check. Helm can create SA without Role/RoleBinding (spark.serviceAccount.create vs spark.rbac.create). Please require all three to exist (and surface unexpected Get errors), e.g. fail unless saOK && roleOK && roleBindingOK.

RobuRishabh · 2026-07-17T04:42:46Z

+			if !jobNamespaceSelectorConfigured {
+				Skip("Namespace filtering test requires operator configured with --job-namespace-selector flag")


Nit: Skip text says --job-namespace-selector, but the controller flag is --namespace-selector (cmd/operator/controller/start.go). need to align the message with the real flag

RobuRishabh · 2026-07-17T05:12:38Z

 	switch deployMethod {
 	case "helm":
 		mutatingWebhookName = "spark-operator-webhook"
 		validatingWebhookName = "spark-operator-webhook"


Webhook names are still hardcoded to spark-operator-webhook for Helm. The Helm chart derives these from {{ fullname }}-webhook, which changes with fullnameOverride or a non-default release name. So a preinstalled operator with fullnameOverride: my-operator will time out at the "Waiting for webhooks" step even when OPERATOR_NAMESPACE and OPERATOR_DEPLOYMENT_NAME are correctly set.

Worth either adding MUTATING_WEBHOOK_NAME/VALIDATING_WEBHOOK_NAME overrides (consistent with the other OPERATOR_* vars), or documenting that preinstalled Helm mode currently requires the default release name.

RobuRishabh · 2026-07-17T05:12:50Z

+		types.NamespacedName{Name: operatorDeploymentName, Namespace: operatorNamespace},
+		deployment)
+	if err != nil {
+		Fail(fmt.Sprintf("Failed to get controller deployment %s/%s: %v\nIn preinstalled mode, set OPERATOR_NAMESPACE and OPERATOR_DEPLOYMENT_NAME to match your deployment",


The feature detection runs unconditionally (both preinstalled and non-preinstalled), but the Fail() message only mentions preinstalled mode. If this fails in standard mode (e.g., the Helm install partially failed), the error message will mislead. Consider branching the message:

hint := "Check that the operator deployment completed successfully" if preinstalled { hint = "In preinstalled mode, set OPERATOR_NAMESPACE and OPERATOR_DEPLOYMENT_NAME to match your deployment" }

RobuRishabh · 2026-07-17T05:12:58Z

 	By("Waiting for the webhooks to be ready")
 	mutatingWebhookKey := types.NamespacedName{Name: mutatingWebhookName}
 	validatingWebhookKey := types.NamespacedName{Name: validatingWebhookName}
 	Expect(waitForMutatingWebhookReady(context.Background(), mutatingWebhookKey)).NotTo(HaveOccurred())
 	Expect(waitForValidatingWebhookReady(context.Background(), validatingWebhookKey)).NotTo(HaveOccurred())


same as https://github.com/kubeflow/spark-operator/pull/3045/changes#r3600513159

RobuRishabh · 2026-07-17T05:14:52Z

+e2e-test: envtest kind-load-image kind-load-spark-image ## Run the e2e tests against a Kind k8s instance. DEPLOY_METHOD: helm (default) or kustomize.
 	@echo "Running e2e tests (deploy_method=$(DEPLOY_METHOD))..."
-	DEPLOY_METHOD=$(DEPLOY_METHOD) IMAGE_TAG=$(IMAGE_TAG) KUBECONFIG=$(KIND_KUBE_CONFIG) go test ./test/e2e/ -v -ginkgo.v -timeout 30m
+	export DEPLOY_METHOD="$(DEPLOY_METHOD)" && \
+	export IMAGE_TAG="$(IMAGE_TAG)" && \
+	export KUBECONFIG="$(KIND_KUBE_CONFIG)" && \


The existing e2e-test target was changed from inline env-var style (VAR=val cmd) to export VAR && cmd. Both work, but the original inline form is more idiomatic for Makefiles and scopes the vars to just the go test process. This change isn't needed for the PR and adds noise to the diff.

Copilot AI review requested due to automatic review settings July 15, 2026 13:32

google-oss-prow Bot requested review from ImpSy and RobuRishabh July 15, 2026 13:32

google-oss-prow Bot added the size/M label Jul 15, 2026

shruthis4 force-pushed the feat/e2e-preinstalled-mode branch from 72aff1a to e61e5fc Compare July 15, 2026 13:33

Copilot started reviewing on behalf of shruthis4 July 15, 2026 13:35 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

shruthis4 force-pushed the feat/e2e-preinstalled-mode branch 2 times, most recently from cdfa7ba to ec2675e Compare July 15, 2026 15:20

google-oss-prow Bot added size/L and removed size/M labels Jul 15, 2026

shruthis4 requested a review from Copilot July 15, 2026 15:21

Copilot started reviewing on behalf of shruthis4 July 15, 2026 15:21 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

shruthis4 force-pushed the feat/e2e-preinstalled-mode branch from ec2675e to 4e125ab Compare July 15, 2026 15:58

shruthis4 requested a review from Copilot July 15, 2026 15:59

Copilot started reviewing on behalf of shruthis4 July 15, 2026 15:59 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

shruthis4 force-pushed the feat/e2e-preinstalled-mode branch from 4e125ab to 9646ac9 Compare July 15, 2026 16:37

shruthis4 requested a review from Copilot July 15, 2026 16:38

Copilot started reviewing on behalf of shruthis4 July 15, 2026 16:38 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

shruthis4 force-pushed the feat/e2e-preinstalled-mode branch from 9646ac9 to dfe8571 Compare July 15, 2026 17:05

RobuRishabh suggested changes Jul 17, 2026

View reviewed changes

google-oss-prow Bot assigned RobuRishabh Jul 17, 2026

Navigation Menu

Uh oh!

feat: Add preinstalled mode to e2e test suite - #3045

feat: Add preinstalled mode to e2e test suite#3045
shruthis4 wants to merge 1 commit into
kubeflow:masterfrom
shruthis4:feat/e2e-preinstalled-mode

shruthis4 commented Jul 15, 2026 •

edited

Loading

Uh oh!

google-oss-prow Bot commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

RobuRishabh Jul 17, 2026

Uh oh!

RobuRishabh Jul 17, 2026

Uh oh!

RobuRishabh Jul 17, 2026

Uh oh!

RobuRishabh Jul 17, 2026

Uh oh!

RobuRishabh Jul 17, 2026

Uh oh!

RobuRishabh Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		if !driverPDBEnabled {
		Skip("Driver PDB feature not enabled in operator (--enable-driver-pdb=true required)")

		if !jobNamespaceSelectorConfigured {
		Skip("Namespace filtering test requires operator configured with --job-namespace-selector flag")

Uh oh!

Conversation

shruthis4 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Core Features

Environment Variables

Test Behavior

Usage

Preinstalled Mode (Managed Clusters)

Standard Mode (Unchanged)

Prerequisites for Preinstalled Mode

Benefits

Backward Compatibility

Change Category

Checklist

Uh oh!

google-oss-prow Bot commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

RobuRishabh Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

RobuRishabh Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

RobuRishabh Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

RobuRishabh Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

RobuRishabh Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

RobuRishabh Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shruthis4 commented Jul 15, 2026 •

edited

Loading