Skip to content

Test spark lite heavy#1348

Open
Chandanydv1234 wants to merge 2 commits into
zinggAI:mainfrom
Chandanydv1234:test-spark-lite-heavy
Open

Test spark lite heavy#1348
Chandanydv1234 wants to merge 2 commits into
zinggAI:mainfrom
Chandanydv1234:test-spark-lite-heavy

Conversation

@Chandanydv1234

Copy link
Copy Markdown
Contributor

Closes #1275

What & why

Our Spark JUnits all shared one SparkSessionProvider that read
zingg.properties, which sets the driver/executor to 8g. Only the end-to-end
tests actually need that — the rest were paying for 8g for no reason. This
splits the provider into a base plus two memory profiles so unit tests can run
light.

Changes

  • Added SparkSessionProviderBase (abstract) that holds all the session
    setup; subclasses just say which properties file to load via
    getPropertiesFile().
  • SparkSessionProviderLite loads a new zingg-lite.properties (1g),
    SparkSessionProviderHeavy loads zingg-heavy.properties (8g, the old file
    renamed).
  • Added TestSparkBaseLite / TestSparkBaseHeavy extensions.
  • Moved the 18 unit tests onto TestSparkBaseLite, and the two integration
    executor tests (TestSparkExecutorsCompound, TestSparkExecutorsSingle)
    onto TestSparkBaseHeavy.
  • Removed the old SparkSessionProvider and TestSparkBase.
  • Set reuseForks=false in spark/core's surefire config. A Spark session is a
    singleton per JVM (getOrCreate hands back the first one built), so without a
    fresh JVM per test class the first class to run would fix the memory for the
    rest — this keeps the 1g and 8g profiles independent.

Tests

Ran locally with mvn -pl spark/core -am test. The provider logs
Spark driver memory: X GB on startup, so the numbers below are what each
session actually came up with:

  • test-compile for spark/core + upstream modules — compiles clean.
  • TestSparkFrame (Lite) — 1 GB — 32 run, 0 failures, 2 skipped.
  • TestSparkExecutorsCompound (Heavy) — 8 GB — 1 run, 0 failures.
  • TestSparkExecutorsSingle (Heavy) — 8 GB — 1 run, 0 failures.
  • Sanity-checked the original code behaves the same, so no regression.

One open question

The issue says "only the integration test" for Heavy, but I put both
TestSparkExecutorsCompound and TestSparkExecutorsSingle on Heavy since both
run full pipelines off configSparkIntTest.json. Happy to move Single back to
Lite if you'd rather stick to the letter of the issue.

@Chandanydv1234
Chandanydv1234 force-pushed the test-spark-lite-heavy branch from 890cdc1 to d96e9f8 Compare July 22, 2026 08:57

@sonalgoyal sonalgoyal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of a static getInstance method, can you have a static attribute named isSessionInitialised just like https://www.baeldung.com/java-junit5-run-before-all-classes ?

dropped getInstance from the providers and used the isSessionInitialised
flag in the extension to build the session once. made initializeSession
public. memory stays the same, lite 1g and heavy 8g.
@Chandanydv1234

Copy link
Copy Markdown
Contributor Author

instead of a static getInstance method, can you have a static attribute named isSessionInitialised just like https://www.baeldung.com/java-junit5-run-before-all-classes ?

Done as suggested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

configurable value of zingg properties in spark session provider in junits

3 participants