Restore test coverage reporting to Kubeflow standards #3076

What happened?

• I have searched the open/closed issues and my issue is not listed.

Spark Operator runs make unit-test in CI and generates cover.out plus cover.html, but CI does not publish the coverage profile or report coverage status. The files disappear with the runner, so maintainers cannot track coverage trends or detect regressions from pull requests.

CI also does not enforce minimum coverage for newly added or changed code. New behavior can therefore merge without meaningful test coverage even when repository-wide coverage remains unchanged or increases because of unrelated packages.

Kubeflow Trainer publishes its Go coverage profile to Coveralls from its test workflow. Spark Operator should restore equivalent Kubeflow project coverage reporting, establish an agreed repository baseline, and enforce an agreed minimum coverage threshold for new code.

The old #57 tracked improving unit test coverage and was closed in 2018. This issue tracks the current CI reporting and enforcement gap.

Reproduction Code

1. Open .github/workflows/integration.yaml.
2. Observe that the build-spark-operator job runs make unit-test.
3. Observe that no later step uploads cover.out to Coveralls or another project-standard coverage service.
4. Open a pull request that adds untested code or reduces coverage; CI provides no coverage delta, new-code coverage result, or failing coverage status.

Expected behavior

• CI publishes cover.out using the coverage service and workflow pattern adopted by Kubeflow projects.
• Pull requests show repository coverage, coverage delta, and coverage for newly added or changed code.
• CI enforces an agreed minimum coverage percentage for new code and fails pull requests that do not meet it.
• Repository documentation displays current coverage status and documents the enforced threshold where appropriate.
• Maintainers define an initial repository baseline and a policy that prevents unintentional overall coverage regressions without blocking incremental test improvements.
• Coverage enforcement supports explicit, reviewed exclusions for generated or otherwise non-testable code rather than silently lowering the threshold.
• Coverage collection remains compatible with envtest-backed packages and excludes only packages intentionally omitted today, such as e2e and drift tests.

Actual behavior

make unit-test generates local coverage files, but CI neither preserves nor reports them. Coverage regressions and insufficient coverage of new code are invisible during review.

Environment & Versions

• Kubernetes Version: N/A
• Spark Operator Version: master
• Apache Spark Version: N/A

Additional context

Kubeflow Trainer reference workflow: https://github.com/kubeflow/trainer/blob/master/.github/workflows/test-go.yaml

Current Spark Operator workflow: https://github.com/kubeflow/spark-operator/blob/master/.github/workflows/integration.yaml

Current coverage generation: https://github.com/kubeflow/spark-operator/blob/master/Makefile

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍