ci: smoke-test the computing-unit cluster (master boot + worker join)

[mengw15]

What changes were proposed in this PR?

Nothing in CI verified that an amber worker node can join a running cluster. The standalone computing-unit-master boot check (#6377) forms only a 1-node self-seeded cluster, texera-web doesn't touch clustering, and the packaged computing-unit-worker launcher (bin/computing-unit-worker) is never started at all. A regression in the worker's join path — its Pekko remoting/serialization, seed-node config, or startActorWorker itself — would break the deployed worker while build, unit tests, and the boot checks all stay green.

The worker can't reuse smoke-boot.sh: ComputingUnitWorker.main only calls AmberRuntime.startActorWorker, which runs no HTTP server and binds pekko.remote.artery.canonical.port = 0 (a random ephemeral port), so there is no fixed port to probe. Booted standalone it just retries its seed and stays up, so a "did not crash" check proves nothing — the worker's real health signal is cluster membership.

• Add .github/scripts/smoke-cluster.sh — a self-contained harness that boots a computing-unit-master (non-cluster mode self-seeds pekko://Amber@localhost:2552 and serves Dropwizard on :8085) and a no-arg computing-unit-worker (defaults its seed to localhost:2552 and skips the getNodeIpAddress → checkip.amazonaws.com lookup that only fires with --serverAddr, so no external network), then asserts the worker actually JOINS — waiting for ClusterListener's Now we have 2 nodes in the cluster readiness line on the master (a deliberate signal, not a crash-scan; contrast smoke-boot.sh false positive: jOOQ "tip of the day" text matches the crash regex and fails boot checks at random #6332). It fails fast if either JVM dies and reaps both on exit.
• Replace the standalone master boot step with this single cluster check. smoke-cluster.sh boots the master, waits for it to listen on :8085, and verifies it self-joins its 1-node cluster before launching the worker — so a texera-web fails to start after Hadoop 3.4.3 upgrade: NoClassDefFoundError from pjfanning jersey-json #6204-class linkage crash on master startup is still caught (reported as master exited before listening). The separate smoke-boot.sh master step (ci: add a computing-unit-master boot smoke test in amber-integration #6377) is a strict subset of this, so it's removed rather than run redundantly. Wired into amber-integration after the texera-web boot, reusing the amber dist already built there. ubuntu-latest only.
• Add .github/scripts/test_smoke_cluster.sh — drives the harness end-to-end with fake launchers (6 cases), asserting the specific verdict message for each outcome so that deleting a fast-fail branch (and letting a case pass only via the slow timeout) turns the test red. The happy path ties the join signal to the worker actually running, so it can't pass if the worker is never launched. Discovered and run by the infra job.

Any related issues, documentation, discussions?

Closes #6523, raised from the #6377 review (suggested by @Yicong-Huang). Supersedes the standalone computing-unit-master boot step from #6377 (this cluster check boots and verifies the master too); complements the texera-web boot (#6319). Smoke rationale from #6220.

How was this PR tested?

• amber-integration (ubuntu-latest) runs the new step in this PR: the master and worker boot, the worker joins, and the master logs 2 nodes in the cluster.
• test_smoke_cluster.sh passes 6/6 locally on bash 3.2.57 (macOS) and bash 5; mutation-tested — deleting the worker-alive fast-fail, the master-alive fast-fail, or the worker launch each turns the test red; shellcheck clean; deterministic across repeated runs with no leftover processes.
• Verified single-host + no external egress by reading AmberRuntime.startActorMaster/startActorWorker: the non-cluster master self-seeds localhost:2552, and a no-arg worker joins it while skipping the checkip.amazonaws.com lookup.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (claude-opus-4-8)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• No candidates found from git blame history.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.65%. Comparing base (0467c76) to head (6d76dcb).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6719      +/-   ##
============================================
- Coverage     74.65%   74.65%   -0.01%     
  Complexity     3438     3438              
============================================
  Files          1160     1160              
  Lines         45793    45793              
  Branches       5069     5069              
============================================
- Hits          34189    34188       -1     
  Misses         9953     9953              
- Partials       1651     1652       +1     

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.00% <ø> (ø)
agent-service	76.76% <ø> (ø)
amber	66.81% <ø> (ø)		Carriedforward from 0467c76
computing-unit-managing-service	18.00% <ø> (ø)
config-service	66.66% <ø> (ø)
file-service	66.80% <ø> (ø)
frontend	78.56% <ø> (-0.01%)	⬇️
notebook-migration-service	78.94% <ø> (ø)
pyamber	91.99% <ø> (ø)
workflow-compiling-service	55.14% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

✅ No material benchmark regressions detected

🟢 2 better · 🔴 0 worse · ⚪ 13 noise (<±5%) · 0 without baseline

Compared against main 0467c76 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🟢	bs=10 sw=10 sl=64	565	0.345	17,357/23,616/23,616 us	🟢 -5.4% / 🔴 +49.3%
⚪	bs=100 sw=10 sl=64	1,196	0.73	82,393/114,640/114,640 us	⚪ within ±5% / 🟢 +18.5%
⚪	bs=1000 sw=10 sl=64	1,436	0.877	698,670/730,691/730,691 us	⚪ within ±5% / 🟢 +37.6%

Baseline details

Latest main 0467c76 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	565 tuples/sec	571 tuples/sec	769.76 tuples/sec	-1.1%	-26.6%
bs=10 sw=10 sl=64	MB/s	0.345 MB/s	0.349 MB/s	0.47 MB/s	-1.1%	-26.6%
bs=10 sw=10 sl=64	p50	17,357 us	17,605 us	12,663 us	-1.4%	+37.1%
bs=10 sw=10 sl=64	p95	23,616 us	24,965 us	15,813 us	-5.4%	+49.3%
bs=10 sw=10 sl=64	p99	23,616 us	24,965 us	19,001 us	-5.4%	+24.3%
bs=100 sw=10 sl=64	throughput	1,196 tuples/sec	1,215 tuples/sec	1,009 tuples/sec	-1.6%	+18.5%
bs=100 sw=10 sl=64	MB/s	0.73 MB/s	0.741 MB/s	0.616 MB/s	-1.5%	+18.5%
bs=100 sw=10 sl=64	p50	82,393 us	79,347 us	99,444 us	+3.8%	-17.1%
bs=100 sw=10 sl=64	p95	114,640 us	115,020 us	105,012 us	-0.3%	+9.2%
bs=100 sw=10 sl=64	p99	114,640 us	115,020 us	116,467 us	-0.3%	-1.6%
bs=1000 sw=10 sl=64	throughput	1,436 tuples/sec	1,441 tuples/sec	1,044 tuples/sec	-0.3%	+37.5%
bs=1000 sw=10 sl=64	MB/s	0.877 MB/s	0.879 MB/s	0.637 MB/s	-0.2%	+37.6%
bs=1000 sw=10 sl=64	p50	698,670 us	699,151 us	964,034 us	-0.1%	-27.5%
bs=1000 sw=10 sl=64	p95	730,691 us	739,197 us	1,007,497 us	-1.2%	-27.5%
bs=1000 sw=10 sl=64	p99	730,691 us	739,197 us	1,040,936 us	-1.2%	-29.8%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,354.00,200,128000,565,0.345,17356.82,23616.18,23616.18
1,100,10,64,20,1672.70,2000,1280000,1196,0.730,82393.19,114640.19,114640.19
2,1000,10,64,20,13923.97,20000,12800000,1436,0.877,698670.09,730690.60,730690.60