[DUKartik]

Purpose of this PR

This PR adds support for configuring physical Kubernetes CPU requests and limits for SparkConnect through the SparkConnect CRD.

Currently, SparkConnect exposes the cores field, which configures Spark driver/executor cores. However, cores does not provide independent control over the physical Kubernetes CPU request and limit for the corresponding pods.

Proposed changes:

• Add optional coreRequest and coreLimit fields to SparkPodSpec for SparkConnect server and executor pods.
• Map coreRequest and coreLimit to the corresponding Spark Kubernetes CPU request/limit configuration for the driver and executors.
• Add CPU quantity validation and unit tests for CPU configuration generation and validation.
• Regenerate the CRD, OpenAPI, Swagger, and Python API artifacts.
• Update the SparkConnect example to demonstrate the new CPU request/limit fields.

Example

executor:
  instances: 2
  cores: 1
  coreRequest: "1000m"
  coreLimit: "1"
  memory: 512m

The new fields are independent of cores:

• cores → spark.driver.cores / spark.executor.cores
• coreRequest → physical Kubernetes CPU request
• coreLimit → physical Kubernetes CPU limit

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

SparkConnect currently does not expose physical Kubernetes CPU request/limit configuration through its CRD.

This change provides independent control over Spark's core configuration and the Kubernetes CPU resources assigned to SparkConnect driver and executor pods, while preserving the existing behavior of the cores field.

The new fields are optional, so existing SparkConnect resources remain backward compatible.

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

The implementation maps the new fields as follows:

• server.coreRequest → spark.kubernetes.driver.request.cores
• server.coreLimit → spark.kubernetes.driver.limit.cores
• executor.coreRequest → spark.kubernetes.executor.request.cores
• executor.coreLimit → spark.kubernetes.executor.limit.cores

CPU quantity validation follows the repository's existing Kubernetes quantity validation conventions.

Generated API artifacts were regenerated and verified, including the CRD, OpenAPI/Swagger specification, and Python API models.

The SparkConnect example was updated to document usage of coreRequest and coreLimit.

Relevant SparkConnect controller and webhook tests pass locally, go vet passes, CRD drift checks pass, generated artifacts are up to date, and git diff --check passes.

Closes #2957

[github-actions]

🎉 Welcome to the Kubeflow Spark Operator! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
• Our team will review your PR soon!

Join the community:

• Slack: Join our #kubeflow-spark-on-kubernetes Slack channel.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[DUKartik]

Hi @tariq-hasan, I’ve implemented the upstream SparkConnect CRD support for #2957 and opened this PR.

The changes add coreRequest and coreLimit for SparkConnect server/executor CPU resources, while keeping them independent of the existing cores field.

Since you pointed me to the upstream issue, I’d appreciate your review when you have a chance. Thanks!

[tariq-hasan]

Hi @DUKartik! Thank you for raising the PR. I have added some review comments.

[tariq-hasan]

/ok-to-test
/retest

[DUKartik]

Thanks @tariq-hassan for pointing this out.

I updated the implementation based on your feedback:

• server.coreRequest and server.coreLimit are now applied directly to the Kubernetes CPU request/limit of the operator-created SparkConnect server pod.
• Removed the spark.kubernetes.driver.request.cores and spark.kubernetes.driver.limit.cores mappings for the server.
• Kept executor.coreRequest and executor.coreLimit mapped to spark.kubernetes.executor.request.cores and spark.kubernetes.executor.limit.cores, since the executor pods are created by Spark.
• Added reconciler tests covering CPU request/limit handling, Cores independence, invalid quantities, and template resources.
• Regenerated the affected API/CRD/OpenAPI/Python API artifacts.

The focused SparkConnect/webhook tests, go vet, gofmt, and git diff --check all pass locally.

[tariq-hasan]

Hi @DUKartik! I have added a few comments.

[vikas-saxena02]

@DUKartik are you able to address the review comments on this one

[vikas-saxena02]

/retest

[DUKartik]

Hi @tariq-hasan,
I noticed the Prow bot removed the LGTM label after I pushed the empty commit to retry CI. Since it didn't change the implementation,
Could you please re-apply /lgtm if the current revision still looks good to you?

Sorry for the extra notification, and thanks again for your guidance!

[tariq-hasan]

/ok-to-test
/retest

[tariq-hasan]

/ok-to-test
/retest

[tariq-hasan]

If you rebase the PR on master I think it should help trigger the CI. One of the tests is failing currently.

[DUKartik]

Thanks @tariq-hasan! I looked into the CI failure a bit further.
It seems the Minikube setup is hitting GitHub's anonymous API rate limit (403), possibly because inputs.github-token is empty for this workflow. Using secrets.GITHUB_TOKEN might help address this.
Since the branch already appears up to date with master, I'm not sure rebasing would help with this particular failure.
Would you prefer to include this small workflow change here or handle it separately?

Thanks again for your guidance!

[vikas-saxena02]

@DUKartik this is a known issue best is to execute a /retest ehen you run into this issue it will rerun failed tests. As the developer it always works for me. i am going to try this for you but it m,ay not work so feel free to do it once you are back online

[vikas-saxena02]

/retest

[vikas-saxena02]

@DUKartik seems like /retest only works from developer himself. can you please do that

[DUKartik]

/retest

[DUKartik]

Thanks a lot, @vikas-saxena02, for your help! /retest didn't seem to work for me, but the latest merge commit retriggered CI, and all checks have now passed.
Really appreciate you looking into this!

[DUKartik]

Thanks @tariq-hasan! The latest merge commit retriggered CI, and all checks have now passed.
I really appreciate your patience and guidance throughout this process.
Could you please re-apply /lgtm if the current revision looks good to you?
Thanks again!

[tariq-hasan]

Thanks for the contribution!
/lgtm

[vikas-saxena02]

@nabuskey can you please merge it so taht #3112 is unblocked

[DUKartik]

Hi @nabuskey ,
could you please take a look at this PR when you get a chance? If everything looks good to you, would you mind merging it? Thanks for your time and help!

[DUKartik]

Hi @nabuskey ,
I've addressed the review comments and updated the validation logic, tests, and documentation. Could you please re-review the latest changes? If everything looks good, I'll create a merge commit from main.
Thanks for the feedback!

[nabuskey]

/approve

Thank you very much!

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nabuskey

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [nabuskey]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[DUKartik]

Hi @Tariq and @nabuskey,
thank you both for your valuable feedback and guidance.
I really appreciate your time and support!