Skip to content

bug: leaderboard runs affected by two balatrobot defects — economy short-collected, losses recorded as wins #85

Description

@Khetnen

Description

We replayed the full BalatroBench CC0 dataset (241 runs, 18k+ actions) on a
vanilla-faithful Balatro simulator as a differential check. Two balatrobot-level
defects we found and filed upstream reach published results:

  1. cash_out race (fix!(lua.endpoints): cash_out fires before round-eval rows commit — rewards under-collected, sometimes $0 balatrobot#231). bot.py calls cash_out the moment
    it sees ROUND_EVAL; the endpoint fires G.FUNCS.cash_out before the
    round-eval rows finish committing, so affected cash-outs collect one blind
    reward short ($3/$4/$5), or $0 when the payout is only the reward row. In our
    replay, of the runs with money divergence, 61 had live short (the race
    direction) and 13 the opposite (unexplained, possibly a second mechanism).
    Several dataset runs die in the shop on purchases they could have afforded
    with the missing rewards.

  2. won-flag cross-run leak (fix!(lua): G.GAME state survives menu+start — a previous run's won flag marks later losses as wins balatrobot#232). The executor starts its
    Balatro instances once and cycles the whole task list through them; each run
    begins with menu + start, and G.GAME.won survives that path. Since
    bot.py records finish_reason = "won" whenever gamestate["won"] is
    truthy, a loss that occurs on an instance after that instance has produced
    any win can be recorded — and published — as a win. We observed the
    leaked flag 16 times in our own long-lived-instance campaigns; the exposure
    here grows with tasks-per-instance.

A note for dataset consumers

Independent of the above: in gamestates.jsonl, ~1% of tool calls are
valid-but-unexecutable on live; they consume a response without emitting a
gamestate, which shifts every later index. Anyone aligning gamestates[i] with
action i needs to re-align at those events (we detected it when a "divergence"
decoded exactly to the following step's rearrange arguments).

Suggested Fixes

  • Pick up the balatrobot fixes when they land.
  • Affected wins are detectable after the fact: a recorded win whose final
    gamestate never beat the ante-8 boss (chips/ante inspection) is the leak.
    Consider re-scoring or flagging affected runs in the published data.
  • Meanwhile, for benchmark-grade integrity: one game process per run (avoids all
    cross-run G.GAME leaks), and delay cash_out until the eval rows commit.

Happy to share our replay tooling/results for cross-checking.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions