Skip to content

Add versioned benchmark configuration and recoverable Studio runs - #6

Merged
cgmattos merged 21 commits into
mainfrom
ailabs/config-repository
Sep 11, 2026
Merged

Add versioned benchmark configuration and recoverable Studio runs#6
cgmattos merged 21 commits into
mainfrom
ailabs/config-repository

Conversation

@cgmattos

Copy link
Copy Markdown
Contributor

Configured benchmark runs now resolve a frozen configuration revision, retain complete report evidence, and can pause or explicitly resume without replaying finalized attempts. Settings provides the versioned configuration editor; interrupted work keeps its original identity, evidence and spending ceiling.

Changes

  • Add Git-backed configuration storage, validation, conflict-safe saves to main, and read-only snapshots when repository access is unavailable.
  • Add configured Pause/Resume with a recovery preview, exclusive execution ownership, persisted cancellation, and billing/version/approval checks before continuation.
  • Read reports from retained versioned attempt evidence and update Activity from configured result events, including recovered attempts.
  • Include the Gmail front-door translation correction and operational recovery/deployment records. AutomationBench inputs and historical verdicts remain unchanged.
  • Integrate current main (0d7a127), preserving its updated run page. Remove its stale test assertion for the already removed Actions column.

Validation

  • Integrated Python regression group: 131 passed; the sole stale run-page assertion was corrected, then all 9 run-page tests passed.
  • Real browser pause/drain/resume: 4 free attempts in 2 segments, matching uninterrupted execution and preserving earlier results.
  • Configuration editor browser checks passed on desktop and 390px, including validation, conflict handling and draft retention.
  • Hosted controls/report verification: 19 deployed source hashes matched the reviewed upload; 28 historical job/event/result files across 12 jobs remained byte-identical.
  • No paid benchmark was launched. CI runs tests and corpus validation, with no deployment step.

Remaining prerequisites

Live Git saves still require the restricted PAT to gain repository access; hosted configuration stays read-only meanwhile. The pre-existing hosted world differs from the frozen tier-simple world, so version checks continue to block another launch until the matching upstream runtime is restored. The current-main integration in this PR has not been redeployed.

Detailed evidence: monarch-benchmark/docs/rounds/2026-09-11-studio-run-controls.md.

@cgmattos
cgmattos merged commit aa53ffb into main Sep 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant