Skip to content

Fix flake in multi-thread handoff test#517

Open
Jawnnypoo wants to merge 1 commit into
masterfrom
Jawnnypoo/fix-multi-thread-handoff-flake
Open

Fix flake in multi-thread handoff test#517
Jawnnypoo wants to merge 1 commit into
masterfrom
Jawnnypoo/fix-multi-thread-handoff-flake

Conversation

@Jawnnypoo

Copy link
Copy Markdown
Member

Summary

The test formula multi-thread handoff to executing thread was flaky (~10-30% failure rate). It submitted work to two single-thread executors back-to-back and assumed thread-a would always win the SynchronizedUpdateQueue takeOver CAS — but there was no synchronization forcing that ordering, so sometimes thread-b won the race instead:

expected: [(50, thread-a), (10, thread-a), (10, thread-b)]
but was : [(10, thread-b), (50, thread-b), (10, thread-b)]

This adds an awaitTakeOver option to MultiThreadRobot.thread() that blocks until the named thread has actually claimed the queue. Detection uses an InspectoronStateChanged fires synchronously inside the listener, which itself runs inside takeOver, so observing the state change with pendingEvent.threadName == name proves that thread owns threadRunning. Once thread-a holds the queue, thread-b's submission is guaranteed to be enqueued behind it.

No production code changed — only the test helper and the one assertion that depended on the racy ordering.

Test plan

  • Ran the previously-flaky test 20 consecutive times — all pass
  • Ran the two other MultiThreadRobot-based tests (formula multi-threaded events fired at the same time, formula multi-threaded input after termination) — still pass

🤖 Generated with Claude Code

The test submitted work to two single-thread executors back-to-back and
assumed thread-a would always win the `SynchronizedUpdateQueue` takeOver
CAS. There was no synchronization forcing that ordering, so on ~10-30%
of runs thread-b's executor started up first, took over the queue, and
processed all three events itself — producing:

    expected: [(50, thread-a), (10, thread-a), (10, thread-b)]
    but was : [(10, thread-b), (50, thread-b), (10, thread-b)]

Add an `awaitTakeOver` option to `MultiThreadRobot.thread()` that blocks
until the named thread has actually claimed the queue. Detection uses an
Inspector — `onStateChanged` fires synchronously inside the listener,
which itself runs inside `takeOver`, so observing the state change with
`pendingEvent.threadName == name` proves that thread owns `threadRunning`.
Once thread-a holds the queue, thread-b's submission is guaranteed to be
enqueued behind it, restoring the deterministic ordering the assertion
expects.

Verified by running the test 20 consecutive times — all pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@carrotkite

Copy link
Copy Markdown

JaCoCo Code Coverage 93.83% ✅

Class Covered Meta Status

Generated by 🚫 Danger

@carrotkite

Copy link
Copy Markdown

📊 Benchmark Comparison Report

Summary

  • Regressions: 0 ✅
  • Improvements: 22 🎉
  • Unchanged: 3

🎉 Performance Improvements

Benchmark Baseline Current Change
com.instacart.formula.benchmarks.TransitionQueueBenchmark.measure10000 0.495 ± 0.002 us/op 0.323 ± 0.001 us/op -34.8% 🟢
com.instacart.formula.benchmarks.TransitionQueueBenchmark.measure100 0.485 ± 0.013 us/op 0.322 ± 0.004 us/op -33.7% 🟢
com.instacart.formula.benchmarks.TransitionQueueBenchmark.measure1 0.485 ± 0.006 us/op 0.329 ± 0.006 us/op -32.3% 🟢
com.instacart.formula.benchmarks.ChildrenInitializationBenchmark.initializeNewChildren (childrenCount=1) 0.834 ± 0.047 us/op 0.604 ± 0.052 us/op -27.6% 🟢
com.instacart.formula.benchmarks.ActionInitializationBenchmark.initializeNewActions (actionCount=1) 0.84 ± 0.03 us/op 0.619 ± 0.041 us/op -26.3% 🟢
com.instacart.formula.benchmarks.GlobalEffectQueueBenchmark.measure100Effects 10.742 ± 0.239 us/op 8.235 ± 0.346 us/op -23.3% 🟢
com.instacart.formula.benchmarks.ActionCountBenchmark.stateChanges (actionCount=1) 15.397 ± 0.088 us/op 11.836 ± 0.081 us/op -23.1% 🟢
com.instacart.formula.benchmarks.TransitionBenchmark.transitions (depth=0) 15.228 ± 0.05 us/op 11.754 ± 0.08 us/op -22.8% 🟢
com.instacart.formula.benchmarks.CallbackOverheadBenchmark.transitions (callbackCount=10) 15.775 ± 0.146 us/op 12.215 ± 0.076 us/op -22.6% 🟢
com.instacart.formula.benchmarks.ActionCountBenchmark.stateChanges (actionCount=25) 15.715 ± 0.111 us/op 12.173 ± 0.134 us/op -22.5% 🟢
com.instacart.formula.benchmarks.TransitionBenchmark.transitions (depth=10) 15.535 ± 0.069 us/op 12.07 ± 0.111 us/op -22.3% 🟢
com.instacart.formula.benchmarks.ChildrenCountBenchmark.stateChanges (childrenCount=25) 15.86 ± 0.068 us/op 12.358 ± 0.117 us/op -22.1% 🟢
com.instacart.formula.benchmarks.CallbackInitializationBenchmark.initializeNewCallbacks (callbackCount=10) 2.007 ± 0.226 us/op 1.57 ± 0.312 us/op -21.8% 🟢
com.instacart.formula.benchmarks.ChildrenCountBenchmark.stateChanges (childrenCount=1) 15.27 ± 0.113 us/op 11.95 ± 0.226 us/op -21.7% 🟢
com.instacart.formula.benchmarks.TransitionBenchmark.transitions (depth=20) 15.973 ± 0.063 us/op 12.566 ± 0.075 us/op -21.3% 🟢
com.instacart.formula.benchmarks.CallbackInitializationBenchmark.initializeNewCallbacks (callbackCount=50) 7.264 ± 1.829 us/op 5.737 ± 0.783 us/op -21.0% 🟢
com.instacart.formula.benchmarks.ActionCountBenchmark.stateChanges (actionCount=100) 16.796 ± 0.118 us/op 13.269 ± 0.16 us/op -21.0% 🟢
com.instacart.formula.benchmarks.CallbackOverheadBenchmark.transitions (callbackCount=50) 17.644 ± 0.812 us/op 14.042 ± 0.699 us/op -20.4% 🟢
com.instacart.formula.benchmarks.ChildrenCountBenchmark.stateChanges (childrenCount=100) 17.502 ± 0.131 us/op 13.959 ± 0.055 us/op -20.2% 🟢
com.instacart.formula.benchmarks.ActionInitializationBenchmark.initializeNewActions (actionCount=25) 3.819 ± 0.102 us/op 3.207 ± 0.052 us/op -16.0% 🟢
com.instacart.formula.benchmarks.ChildrenInitializationBenchmark.initializeNewChildren (childrenCount=25) 3.93 ± 0.063 us/op 3.417 ± 0.131 us/op -13.0% 🟢
com.instacart.formula.benchmarks.ActionInitializationBenchmark.initializeNewActions (actionCount=100) 12.403 ± 0.108 us/op 11.155 ± 0.142 us/op -10.1% 🟢
No significant changes (3 benchmarks)
Benchmark Baseline Current Change
com.instacart.formula.benchmarks.ChildrenInitializationBenchmark.initializeNewChildren (childrenCount=100) 12.999 ± 0.527 us/op 12.32 ± 0.12 us/op -5.2%
com.instacart.formula.benchmarks.GlobalEffectQueueBenchmark.measure10Effects 0.766 ± 0.026 us/op 0.698 ± 0.007 us/op -8.9%
com.instacart.formula.benchmarks.GlobalEffectQueueBenchmark.measure1Effect 0.059 ± 0.002 us/op 0.056 ± 0.001 us/op -4.2%

Regressions: ±10% with non-overlapping confidence intervals. Improvements: ±10% change only.

Generated by 🚫 Danger

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants