fix(dax): stop publishing failed installs and truncated disk sizes - #387
Open
tode-rl wants to merge 1 commit into
Open
fix(dax): stop publishing failed installs and truncated disk sizes#387tode-rl wants to merge 1 commit into
tode-rl wants to merge 1 commit into
Conversation
Two defects in dax-benchmark.sh that put wrong numbers in the published results rather than failing loudly. **A failed `bun install` is recorded as a successful phase.** `install_dependencies` ends with `git diff --exit-code`, which exits 0 whenever the lockfile is untouched — masking the status of the `bun install` before it. `set -e` does not catch this either, because `phase()` runs its callee under `set +e`. Observed on a 1 GiB sandbox: `bun install` is SIGKILLed by the OOM killer at t=27s, node_modules never exists, and the run still emits `BENCH_PHASE install 4248` and proceeds into typecheck, which then fails with a confusing "Cannot run turbo.json" because the toolchain was never installed. Any provider whose install dies without dirtying the lockfile publishes the same silently-wrong timing. `bun install || return` propagates it. **Workspace size above 2 GiB is truncated to 3 bytes.** `disk()` used awk's `print`, and mawk switches to `%.6g` once `$1 * 1024` exceeds INT_MAX, emitting `3.07106e+09`; the consumer's `parseInt()` then reads that as `3`. This is in the current results: every provider on a mawk image (e2b, miosa, mosaic, runloop, sail, superserve, upstash) reports `diskAfterInstall: 3`, rendered as "0.0 MB" on the benchmark pages, while gawk/busybox images report the true ~3.1 GB for the identical workload. `printf "%.0f"` formats it exactly on every awk. Verified against a full run: 38,182 directories, 276,525 files, `du -skx` = 3,239,731,200 bytes, now reported verbatim. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VXBre1j9JaGUwwzXta1B3E
tode-rl
force-pushed
the
fix/dax-install-masking-and-du-overflow
branch
from
September 2, 2026 20:04
096c806 to
01bdbb5
Compare
tode-rl
marked this pull request as ready for review
September 2, 2026 20:41
Contributor License AgreementAll contributors are covered by a CLA. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects in
benchmarks/scripts/dax-benchmark.shthat put wrong numbers into the published results rather than failing loudly. Both are currently visible on the live benchmark pages.1. A failed
bun installis recorded as a successful phaseA function returns its last command's status, so
git diff --exit-code— which exits 0 whenever the lockfile is untouched — masks whateverbun installdid.set -edoes not catch it either, becausephase()runs its callee underset +e.Observed on a 1 GiB sandbox:
node_modulesnever exists, yet the run reports a fast successful install and proceeds into typecheck, which then fails with a confusingCannot run "…/turbo.json"— because the toolchain was never installed. Any provider whose install dies without dirtying the lockfile publishes the same silently-wrong timing.Fixed with
bun install || return.2. Workspace size above 2 GiB is truncated to
3bytesmawk (the default
awkon Debian/Ubuntu images) switches to%.6gonce$1 * 1024exceedsINT_MAX, emitting3.07106e+09. The consumer'sparseInt()then reads that as3.This is in the results today. For the identical workload:
diskAfterInstall: 3— e2b, miosa, mosaic, runloop, sail, superserve, upstash (mawk images), rendered as 0.0 MB on the benchmark pagesdiskAfterInstall: ~3.1e9— arker, daytona, tenki, tensorlake, vercel (gawk/busybox images)It splits by
awkimplementation, not by anything about the provider.printf "%.0f"formats the value exactly on every awk.Verification
Ground truth from a complete run of the pinned workload (opencode @
08fb473): 38,182 directories, 276,525 files,du -skx= 3,239,731,200 bytes — now reported verbatim instead of as3.Not included
While investigating I found a third, separate issue worth raising on its own: on some providers
dureports only directory inodes because their filesystem leavesst_blocksunpopulated for regular files, sodiskAfter*understates real usage by 20-150x (isorun/blaxel report directory-count x 512; archil/beam/createos/declaw/namespace report directory-count x 4096). That is not fixable with a formatting change — it would needdu --apparent-size, which changes what the metric means — so I have left it out of this PR.🤖 Generated with Claude Code
https://claude.ai/code/session_01VXBre1j9JaGUwwzXta1B3E