You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We replayed the full BalatroBench CC0 dataset (241 runs, 18k+ actions) on a
vanilla-faithful Balatro simulator as a differential check. Two balatrobot-level
defects we found and filed upstream reach published results:
cash_out race (fix!(lua.endpoints): cash_out fires before round-eval rows commit — rewards under-collected, sometimes $0 balatrobot#231).bot.py calls cash_out the moment
it sees ROUND_EVAL; the endpoint fires G.FUNCS.cash_out before the
round-eval rows finish committing, so affected cash-outs collect one blind
reward short ($3/$4/$5), or $0 when the payout is only the reward row. In our
replay, of the runs with money divergence, 61 had live short (the race
direction) and 13 the opposite (unexplained, possibly a second mechanism).
Several dataset runs die in the shop on purchases they could have afforded
with the missing rewards.
won-flag cross-run leak (fix!(lua): G.GAME state survives menu+start — a previous run's won flag marks later losses as wins balatrobot#232). The executor starts its
Balatro instances once and cycles the whole task list through them; each run
begins with menu + start, and G.GAME.won survives that path. Since bot.py records finish_reason = "won" whenever gamestate["won"] is
truthy, a loss that occurs on an instance after that instance has produced
any win can be recorded — and published — as a win. We observed the
leaked flag 16 times in our own long-lived-instance campaigns; the exposure
here grows with tasks-per-instance.
A note for dataset consumers
Independent of the above: in gamestates.jsonl, ~1% of tool calls are
valid-but-unexecutable on live; they consume a response without emitting a
gamestate, which shifts every later index. Anyone aligning gamestates[i] with
action i needs to re-align at those events (we detected it when a "divergence"
decoded exactly to the following step's rearrange arguments).
Suggested Fixes
Pick up the balatrobot fixes when they land.
Affected wins are detectable after the fact: a recorded win whose final
gamestate never beat the ante-8 boss (chips/ante inspection) is the leak.
Consider re-scoring or flagging affected runs in the published data.
Meanwhile, for benchmark-grade integrity: one game process per run (avoids all
cross-run G.GAME leaks), and delay cash_out until the eval rows commit.
Happy to share our replay tooling/results for cross-checking.
Description
We replayed the full BalatroBench CC0 dataset (241 runs, 18k+ actions) on a
vanilla-faithful Balatro simulator as a differential check. Two balatrobot-level
defects we found and filed upstream reach published results:
cash_out race (fix!(lua.endpoints): cash_out fires before round-eval rows commit — rewards under-collected, sometimes $0 balatrobot#231).
bot.pycallscash_outthe momentit sees
ROUND_EVAL; the endpoint firesG.FUNCS.cash_outbefore theround-eval rows finish committing, so affected cash-outs collect one blind
reward short ($3/$4/$5), or $0 when the payout is only the reward row. In our
replay, of the runs with money divergence, 61 had live short (the race
direction) and 13 the opposite (unexplained, possibly a second mechanism).
Several dataset runs die in the shop on purchases they could have afforded
with the missing rewards.
won-flag cross-run leak (fix!(lua): G.GAME state survives menu+start — a previous run's won flag marks later losses as wins balatrobot#232). The executor starts its
Balatro instances once and cycles the whole task list through them; each run
begins with
menu+start, andG.GAME.wonsurvives that path. Sincebot.pyrecordsfinish_reason = "won"whenevergamestate["won"]istruthy, a loss that occurs on an instance after that instance has produced
any win can be recorded — and published — as a win. We observed the
leaked flag 16 times in our own long-lived-instance campaigns; the exposure
here grows with tasks-per-instance.
A note for dataset consumers
Independent of the above: in
gamestates.jsonl, ~1% of tool calls arevalid-but-unexecutable on live; they consume a response without emitting a
gamestate, which shifts every later index. Anyone aligning
gamestates[i]withaction
ineeds to re-align at those events (we detected it when a "divergence"decoded exactly to the following step's
rearrangearguments).Suggested Fixes
gamestate never beat the ante-8 boss (chips/ante inspection) is the leak.
Consider re-scoring or flagging affected runs in the published data.
cross-run G.GAME leaks), and delay
cash_outuntil the eval rows commit.Happy to share our replay tooling/results for cross-checking.