Repository navigation
Add AddConnectJar option for SparkClient instead of hardcoding Spark Connect JARs #491
Description
What you would like to be added?
Stop always injecting the Spark Connect Maven JAR in build_spark_connect_cr() (kubeflow/spark/backends/kubernetes/utils.py). Today the CR builder unconditionally sets spark.jars to a Maven spark-connect URL and merges user jars:
connect_jar_url = (
f"https://repo1.maven.org/maven2/org/apache/spark/"
f"spark-connect_{constants.SPARK_CONNECT_PACKAGE_SCALA_VERSION}/{spark_version}/"
f"spark-connect_{constants.SPARK_CONNECT_PACKAGE_SCALA_VERSION}-{spark_version}.jar"
)
base_conf: dict[str, str] = {
"spark.jars": connect_jar_url,
"spark.connect.grpc.binding.address": "0.0.0.0",
}
if spark_conf:
existing_jars = spark_conf.get("spark.jars", "").strip()
if existing_jars:
base_conf["spark.jars"] = f"{connect_jar_url},{existing_jars}"Proposed change
build_spark_connect_cr()— only set required defaults (e.g.spark.connect.grpc.binding.address: 0.0.0.0). Pass throughspark_confas-is; do not setspark.jarsunless the user provides it.- AddConnectJar option — new callable option in
kubeflow.spark.types.options(exported fromkubeflow.spark), following the same pattern asName,Tolerations, etc.:
- Applied via
SparkClient.connect(options=[AddConnectJar()]) - When
enabled=True(default), adds the Maven Spark Connect JAR URL tospec.spark_conf["spark.jars"] - If
spark.jarsalready exists, prepend the Connect JAR (Connect first, then user jars) AddConnectJar(enabled=False)is a no-op- Kubernetes backend only; clear error for incompatible backends
- Tests & docs
- Unit tests for
AddConnectJar(set, prepend, disabled, incompatible backend) - Update
build_spark_connect_crtests sospark.jarsis absent by default - Docstring example on
SparkClient.connect()for opt-in usage
Example
@dataclass
class AddConnectJar:
"""Explicitly add Spark Connect JAR URL to spark_conf["spark.jars"].
Use this option when your image does not already include the Spark Connect JAR.
If spark.jars already exists, Connect JAR is prepended to preserve user jars.
Supported backends:
- Kubernetes
Args:
enabled: Whether to add the Spark Connect JAR URL. Defaults to True.
"""
enabled: bool = True
def __call__(
self, spark_connect: models.SparkV1alpha1SparkConnect, backend: RuntimeBackend
) -> None:
"""Apply Connect JAR injection to the SparkConnect model."""
from kubeflow.spark.backends.kubernetes import constants
from kubeflow.spark.backends.kubernetes.backend import KubernetesBackend
if not isinstance(backend, KubernetesBackend):
raise ValueError(
f"AddConnectJar option is not compatible with {type(backend).__name__}. "
f"Supported backends: KubernetesBackend"
)
if not self.enabled:
return
spark_version = spark_connect.spec.spark_version or constants.DEFAULT_SPARK_VERSION
connect_jar_url = (
f"https://repo1.maven.org/maven2/org/apache/spark/"
f"spark-connect_{constants.SPARK_CONNECT_PACKAGE_SCALA_VERSION}/{spark_version}/"
f"spark-connect_{constants.SPARK_CONNECT_PACKAGE_SCALA_VERSION}-{spark_version}.jar"
)
if spark_connect.spec.spark_conf is None:
spark_connect.spec.spark_conf = {}
existing_jars = spark_connect.spec.spark_conf.get("spark.jars", "").strip()
spark_connect.spec.spark_conf["spark.jars"] = (
f"{connect_jar_url},{existing_jars}" if existing_jars else connect_jar_url
)from kubeflow.spark import AddConnectJar, SparkClient
client = SparkClient()
spark = client.connect(
options=[AddConnectJar()],
spark_conf={"spark.sql.adaptive.enabled": "true"},
)
# Custom image that already bundles Connect — no Maven download
spark = client.connect() # no AddConnectJar
# Extra jars + Connect
spark = client.connect(
options=[AddConnectJar()],
spark_conf={"spark.jars": "local:///opt/spark/jars/my-plugin.jar"},
)Why is this needed?
Hardcoding spark.jars is wrong for many deployments
- Official Spark images (e.g.
apache/spark) often already include the Connect JAR. Forcing a Maven URL causes redundant downloads, Ivy/cache issues, and startup failures when the cluster has no outbound access to Maven Central. - Some teams ship Connect in a custom image or internal artifact repo; unconditional injection overrides that model.
- Pulling JARs at runtime conflicts with air-gapped / restricted networks and with reproducible, image-based workflows.
Opt-in matches KEP-107 extensibility
- Other SparkConnect settings use the options pattern (
Name, tolerations, etc.). Connect JAR injection belongs there too, not as hidden logic in the CR builder. - Users who need the Maven fallback (minimal images without Connect) can pass
AddConnectJar()explicitly.
Clearer separation of concerns
build_spark_connect_cr()should build the CR from explicit user input + safe defaults (binding address), not opinionated dependency management.- JAR lifecycle stays under user control unless they opt in.
Love this feature?
Give it a 👍 We prioritize the features with most 👍
Activity
kliushenkov commentedon May 15, 2026
/area spark
Sumitha007 commentedon Jul 13, 2026
Hi! I'd like to work on this feature. I'll investigate the current implementation and prepare a PR.
tariq-hasan commentedon Aug 29, 2026
/remove-label needs-triage
tariq-hasan commentedon Aug 29, 2026
Hi @kliushenkov, thanks for raising the issue and @Sumitha007, for your interest in the implementation!
I think for this use case we'd need to think more carefully about the options API surface we'd want to support and borrow from the precedent set by the Trainer SDK.
The options API surface for Trainer SDK seems to cover identity and metadata (
Name,Labels,Annotations) as well as execution policy (ActiveDeadlineSeconds,TrainerCommand,TrainerArgs) and this second portion derives from the CRD schema. Comparing against Spark SDK we cover identity and metadata (Name,Labels,Annotations) as well as placement logic (NodeSelector,Toleration). There's also aRuntimePatchesescape hatch on the Trainer side but no equivalent for Spark.In light of this I don't think an
AddConnectJaroption fits the bill in terms of overall design coherence. Dynamic allocation support that's being covered as part of #489 does fit this hierarchy in a clean way - given it's already supported on the CRD schema and this is a tuning knob that power users would want to leverage.At the same time though this is another form of justification for why adding first-class support for dependencies in the
SparkConnectAPI may make sense, mirroring what already exists forSparkApplication. The motivating use case for theDependenciesfield on theSparkConnectCRD though relates to #470 and we may want to subsume this spark-connect jar download discussion under that topic.To support this capability therefore I think we'd need to find the right abstraction that makes sense before we commit to an implementation and most likely #470 and kubeflow/spark-operator#2959 are what will get us there and cover the gap in this issue.
@andreyvelich @Shekharrajak @Goku2099 @RobuRishabh @nabuskey Would appreciate any further thoughts on this issue.
Goku2099 commentedon Aug 30, 2026
Agree with the sequencing.
AddConnectJaras a bespoke option would just be a workaround for the actual gap (noDependenciesfield onSparkConnectSpec), and folding it into #470 avoids adding one-off options that don't derive from the CRD schema.nabuskey commentedon Sep 2, 2026
Implementing kubeflow/spark-operator#2959 makes sense to me.
tariq-hasan commentedon Oct 8, 2026
/milestone v0.6
Rajadi16 commentedon Oct 8, 2026
Hi @Sumitha007 @tariq-hasan, I see this is now in the v0.6 milestone. @Sumitha007, are you still working on it? If you haven't started, I'd like to take it. I plan to add an AddConnectJar-style option to the SparkClient options instead of the hardcoded Connect JARs, with unit tests. Happy to align on the API shape first if you'd prefer.
tariq-hasan commentedon Oct 9, 2026
Hi @Rajadi16! Thank you for the interest. We do not want to implement this as a
AddConnectJaroption. We'd want to do this as aDependenciesoption that can land in the SDK after kubeflow/spark-operator#3120 is completed.