fix(spark): Extracting spark_version from driver image

[Akshat-Raj]

What this PR does / why we need it:
Currently, the only caller of build_spark_connect_cr does not use the spark version from the driver.image. Which means that it was defaulting to the DEFAULT_SPARK_VERSION in every case.

This PR fixes this conflict by:

1. Extracting the spark version using regex from the driver image.
2. Passing the extracted version to build_spark_connect_cr inside _create_session.

If a user doesn't provide an image, or uses a custom tag without a semantic version (e.g., latest), it falls back to returning None, preserving the existing default behavior.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes #780

Checklist:

• Docs included if any changes are user facing
  (Note: No doc changes needed as this does not make any user facing changes.)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kramaranya for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment