1. Provider variance is folded into model variance
The response logs in the CC0 dataset show extra_body with
provider: {sort: "throughput"}, which scattered generations across upstreams —
we saw Amazon Bedrock, Google, SiliconFlow, AtlasCloud, and DeepInfra serving
runs of the same model. Upstreams differ in quantization and sampling defaults,
so part of the leaderboard's run-to-run variance is provider variance, not
model variance.
Suggestions, in increasing order of effort:
- Surface the served provider (already present in
responses.jsonl) in the
leaderboard/run metadata.
- Pin
provider.order / provider.only per model for benchmark runs.
2. Prompt caching is off — roughly a 3× cost saving available
The ~42k-char invariant part 0 (game manual + strategy) is re-sent on every one
of ~200 calls per game (cache_control is unset on all three content parts) —
about 10.5k tokens of identical input per call, which matches the measured 14k
input tokens/call. The three-part message split is already exactly the right
shape for caching: adding cache_control to part 0 (Anthropic-style caching,
which OpenRouter passes through) cuts per-game cost roughly 3× for
caching-capable models ($5.60/game → under $2 at Sonnet-class pricing), with
no behavioral change.
3. Minor: seeded runs are not perfectly reproducible across game launches
Some SMODS-rebuilt pools are iterated in per-launch order before being fed to
the seeded RNG — we confirmed this for To Do List's target hand and Orbital
Tag's hand choice (same seed + identical action trace → different outcome after
a game relaunch, stable within one launch). Worth a caveat wherever the
benchmark claims seed-level determinism.
1. Provider variance is folded into model variance
The response logs in the CC0 dataset show
extra_bodywithprovider: {sort: "throughput"}, which scattered generations across upstreams —we saw Amazon Bedrock, Google, SiliconFlow, AtlasCloud, and DeepInfra serving
runs of the same model. Upstreams differ in quantization and sampling defaults,
so part of the leaderboard's run-to-run variance is provider variance, not
model variance.
Suggestions, in increasing order of effort:
responses.jsonl) in theleaderboard/run metadata.
provider.order/provider.onlyper model for benchmark runs.2. Prompt caching is off — roughly a 3× cost saving available
The ~42k-char invariant part 0 (game manual + strategy) is re-sent on every one
of ~200 calls per game (
cache_controlis unset on all three content parts) —about 10.5k tokens of identical input per call, which matches the measured
14k$5.60/game → under $2 at Sonnet-class pricing), withinput tokens/call. The three-part message split is already exactly the right
shape for caching: adding
cache_controlto part 0 (Anthropic-style caching,which OpenRouter passes through) cuts per-game cost roughly 3× for
caching-capable models (
no behavioral change.
3. Minor: seeded runs are not perfectly reproducible across game launches
Some SMODS-rebuilt pools are iterated in per-launch order before being fed to
the seeded RNG — we confirmed this for To Do List's target hand and Orbital
Tag's hand choice (same seed + identical action trace → different outcome after
a game relaunch, stable within one launch). Worth a caveat wherever the
benchmark claims seed-level determinism.