Repair WorkflowBench evaluation and add execution and spending history - #3
Merged
Conversation
Cherry-picked from Deyton's origin/codex/workflow-prototype-client. The fixed port is refused on Linux CI because a closed connection still holds the address; POSIX needs SO_REUSEADDR to reclaim it. Windows keeps it off, where the same flag lets a second bind steal a port that is already serving. `stop()` joins the serving thread before closing, so a restart on the same port does not race the old server's shutdown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
…fore Thirteen services (airtable, jira, trello, notion, asana, monday and the rest) store every write in one action log, keyed by an id minted during the run. Two separate defects followed from that. First, the diff collapsed a newly added log into one change whose content read "<object>", so the writes inside it were invisible. An added or removed subtree is now walked against an empty one of its own kind, and an entry keeps its own content. Second, no frozen rule can name a run-time id, so the derivation fell back to `airtable.*` -- "anything in Airtable is expected". On `simple.airtable_find_update` that accepted the requested VIP contact plus 200 records nobody asked for: judge A was satisfied, judge B waved all 201 through. Matchers now take `where` (match a write by what it says) and `count` (how many such writes were asked for), and the derivation fills both from the assertion's own table and fields. Doing the requested thing 201 times is not doing the requested thing. Three of the five blind frozen tasks close: airtable_find_update, email_airtable_lead, and invoice_airtable_slack for its table. The two Jira tasks stay open -- their assertions carry no content to pin, which is a limit of the vendor's data, not of this code. A rule carrying neither clause behaves exactly as before, so no frozen task changes its verdict and no hash moves. test_declare_scored's helper wrote its oracle-like action with empty params, which no longer satisfies a `where` rule; it now writes the content the rule names. Verified against the simulator that the real write does satisfy it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
…etitor A verdict alone hides the difference between a competitor that takes the safe path and one that finishes more prompts by making a bigger mess. Both can show the same pass rate. Every attempt now carries a count: changes nobody asked for, plus repetitions of the right change beyond the number the request asked for. The count is reported, never folded into pass/fail -- doing nothing scores zero collateral and still fails. `count_violations` was computed by the invariant and thrown away before the row; it now reaches the row and survives `wb grade`, so old runs regrade with it. The technical page gains two columns, "attempts with collateral" and "collateral changes". They read n/a for the answer key on a task it cannot act on: no action, no collateral. Infrastructure attempts are excluded from both -- the harness broke, not the competitor. On the Easy round of 5 Sep this already separates the field: Opus has the lowest pass rate of the models and the highest collateral (8 attempts of 16), while Monarch has 1 of 20. Those figures come from the old detector, so the models' numbers will rise once the airtable tasks are regraded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
The executive page led with completion only, which flatters a competitor that finishes more of the work by touching more than it was asked to. It gains a fifth headline card, "changes nobody asked for", and a short note in plain words saying what the figure counts and that it is read next to the success rate, not instead of it. The wording avoids the internal term on purpose: the page says what the number means, not what the code calls it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
Two tests against the fake backend failed about one run in three, always after about 20 seconds: test_reply_reaches_the_stream_gate and test_cancel_ends_a_run_waiting_on_a_reply. `stream()` yields None for "still open, nothing said" so a caller can watch its deadline while the server is quiet -- that is the documented contract, and the 20 seconds was its socket beat. The tests called next(stream) expecting a frame and subscripted the beat whenever the fake server had not written yet. A race in the tests, not in the client or the server. They now skip empty beats before asserting. Verified twelve consecutive runs of the pair and five of the whole file, all green in about a second, where the pair previously failed roughly one run in three. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
A guide for anyone who has to read a round and trust the number. Ten sections: what an attempt is, the six stages it goes through, the four kinds of competitor, the HTTP front door and what Monarch does and does not get to see, where the tasks come from and how the four sets were drawn, the two judges with a diagram, collateral damage, and how to read a report. Every claim cites the file that implements it, so the document can be checked rather than believed. Section 9 lists what is still weak, including the two Jira tasks that stay collateral-blind and the weak assertion in simple.invoice_airtable_slack. It records one operational trap worth knowing: the tunnel hostname is baked into the knowledge base, which is inside the config hash, so restarting ngrok on a free plan breaks every dispatch, needs the knowledge base regenerated, and makes earlier runs non-resumable and non-regradable. The flaky-test note added earlier in this session is dropped again: the race is fixed, not deferred. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
Three shapes of approval rule matched nothing, so a perfect answer landed in missing_expected and the task failed for everyone. All three were the deriver reading the RAW SEED and guessing a path the loader then reshapes. Google Sheets: the seed nests rows under spreadsheets[].worksheets[].rows[], but the WorldState validator flattens that into three sibling collections, so a write diffs as google_sheets.rows[...]. The rule named the spreadsheet subtree, which is always empty after flattening. 191 corpus tasks and 12 of the 40 frozen ones could not be passed by anyone. An append now derives `added` on rows[*] anchored by `where` on the spreadsheet and worksheet; an in-place edit derives `changed` on rows[*].cells.<Column>, where the column carries the discrimination because a cell change's `after` is a scalar with nothing to walk. Action logs: `_action_log` asked the seed whether <service>.actions was a dict, so a task whose seed never mentions the service answered no and derived the list-shaped `<service>.actions[*]`, which cannot match a dict-keyed log. It now asks the vendor's own schema. Same root cause made jira derive a bare `jira.*`, which accepted any number of writes nobody asked for: a junk-only attempt scored zero collateral. Reading the seed also missed ids the validator mints: a spreadsheet carrying `id: None` gets one on load, which the raw-seed read could never see. Verified against the simulator, not fixtures -- a synthetic fixture is what let the sheets defect survive. All 12 frozen sheets tasks go before=False after=True with the real write driven through Episode.api_fetch. Corpus-wide, 840 tasks now derive 0 rules pointing at a dead subtree, down from 197. A blanket `count: 1` was tried and reverted: hr.asana_compliance_tasks legitimately needs four create_task writes, and pinning one would have introduced the very false negative this commit removes. `count` now rides only with a discriminating `where`. Task files are untouched. Re-deriving them is a separate step that moves every affected contract hash and needs pre-registration sign-off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
Three Airtable prompts whose approval rule was a whole-service wildcard until 9 Sep, plus one prompt from the Hard round where every competitor scored 0 because the collateral half rejected the changes the task itself demands. Not a measurement: 4 prompts, one attempt each, no retry. It exists to answer whether a grader change works before a round is paid for. The models scored 20/20 with zero collateral on it after the fixes of 9 Sep; the same set caught the numeric-vs-string and id-None defects. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
…erent value Two defects that failed correct answers, both found by running the frozen tasks against the simulator rather than by reading them. Every Slack message in the AutomationBench fixtures carries `id: None`. `_identity` keyed them all as `[id=None]`, collapsing the list onto one entry, so adding the twelfth message read as five field changes to the first and a rule expecting `slack.messages[*]` to be added never matched. On the Hard round's marketing.creative_asset that failed every competitor on a task they got right: re-grading the stored snapshots turns 0 of 11 into 9 of 11. A record now falls back to its positional index when its id is null. An assertion writes `"Value": "1200000"` where the mock stores the number, and the vendor's own checker forgives that (`str(a) != str(b)`). A `where` clause derived from such an assertion did not, so it rejected the write it had asked for. It now compares as text when the values differ, and still rejects a different value. The sweep found 235 such string-numeric values across 75 corpus tasks, 5 of them frozen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
… the KB test_reply_reaches_the_stream_gate failed about one run in three, and the fix landed in two parts because the first diagnosis was incomplete. A test that reads next(stream) has to skip the reader's quiet beats, which `stream()` yields as None by contract; that removed the 20-second failures but not all of them. The rest were a stream a test abandoned mid-iteration: its handler stayed open and served frames from whatever scenario.frames held by then, so a later test could receive `done` where it expected `awaiting_input`. Tests now close what they open, and the fake joins its serving thread before closing -- the same defect fixed in http_shim.py earlier today. Seven consecutive runs of the pair, 86 passed each. The knowledge base is re-synced against the deployed Monarch (`wb monarch setup`, 47 apps, approved by Carlos on 9 Sep). The frozen hashes had drifted from what the discovery service reports and were blocking every Monarch attempt. This moves the config hash of any plan carrying the KB, which is what test_shipped_create_run_pilot_hash_is_unchanged is currently reporting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
…ailabs/collateral-judge # Conflicts: # monarch-benchmark/workflowbench/config/products/simulated-apps.monarch-kb.yaml
…nt-door probe D5 named Lucas as the only approver. Carlos widened it on 9 Sep so a round is not blocked waiting for one person, with the standing rule that an agent never approves its own: the name on the approval is the person who asked for it. The plan is a diagnostic, not a measurement. On 9 Sep every Monarch attempt reached the front door for discovery and then failed its first execution step with "front door is not running on this host" while the shim was still up. One attempt through it settled the question: 9 GETs on /openapi/index.json, all 200, and zero application calls over a 149-second execution phase that ended 19 seconds before the shim came down, with no tunnel alert. Discovery resolves the public front door; execution resolves something else. That is Monarch-side, and it is what the hosted front door is expected to remove. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
…s do not name Two defects that between them let a round spend US$ 18.75 on attempts that measured nothing, with every check green. The front-door relay hardcoded 127.0.0.1, which is right only when the Studio launched the attempt. A CLI round starts the shim on the operator's machine, so a hosted Studio answered 502 to every application call while still serving the OpenAPI documents -- discovery worked, execution did not, and the engine reported "front door is not running on this host". STUDIO_FRONT_DOOR_TARGET now names where the shim listens; unset, the loopback default is unchanged, so the hosted path behaves exactly as before. Verified end to end: a local shim, the relay pointed at the tunnel, 200 and 47 services. `wb monarch verify` never probed the front door, so it passed while the environment named the tunnel and the knowledge base named the hosted door -- a green that could not fail. It now compares the two and refuses, naming both addresses. Run against today's state it reported the mismatch that cost the two rounds. The Monarch test fixtures taught one address in the knowledge base and rewrote another into the harness, which the new check correctly refused; `monarch_site` now writes both from the same port. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
…t not be dropped `_buffered` asked the socket before the buffer. `urlopen` can pull the first frame in while reading headers, leaving the socket with nothing pending -- so `select` timed out and the frame was discarded. The reader then blocked the full 30-second reply gate and read the *second* frame as its first, which is why test_reply_reaches_the_stream_gate failed about one run in three and why an earlier "fix" to the tests did not hold: the defect was in the client, not the fake. It now peeks the buffer first and consults the socket only when that is empty. Six consecutive runs, 16 passed each, the 30-second stalls gone. This could drop a real `awaiting_input` in a live round, so an attempt would miss a question the builder asked. Worth a second pair of eyes from whoever owns the engine's stream. Tests could not use `tmp_path` at all when %TEMP%\wb had no children left: the conftest set basetemp without creating its parent, and pytest's own mkdir does not pass parents=True. 809 tests errored at setup. Pre-existing, and unrelated to today's work -- it needs only the right timing. `monarch_site` rewrote the knowledge base unconditionally, which contradicted a test that asserts the shipped address; it now rewrites only when the caller names a port, which is exactly when the harness is rewritten too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
…from Six defects in the derived approval rules were fixed on 9-10 Sep; the task files still carried the rules those defects produced. Four of them named a path the world can never produce, so a correct answer could not pass: the Google Sheets spreadsheet subtree the loader empties, a list-shaped glob against a dict-keyed action log, `actions` on zoom and calendly which have no such field, and a service wildcard that took any number of writes nobody asked for. 23 of the 40 frozen tasks changed; the other 17 were already right and are byte-identical. The corpus is re-derived too, because the frozen sets are copies of it and `test_tiers_refreeze` requires the two to agree -- re-deriving only the copies left them ahead of their source. Prompts and starting data are untouched. `wb corpus tiers` was NOT re-run: a redraw over a different usable pool picks a different ten and the round sheets stop describing the sets that ran. Every stored attempt on those 23 tasks is now non-regradable, which is the point of the hash. The rounds affected are 5 and 6 September, already unusable: the Hard round scored 0% for everyone, and re-grading the one task whose hash still matched turned 0 of 11 into 9 of 11. Approved by Carlos, 9-10 Sep. The before/after hashes and what the change costs are recorded in docs/rounds/2026-09-10-refreeze-derived-rules.md. test_declare_sheets asserted that the shipped rule failed by reading it from the task file; after re-derivation that file holds the fixed rule, so the broken shape is now written out in the test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
…ird query CREATE_RUN_HASH moves to 41769e05aa5618a9. The Monarch knowledge base was re-synced against the deployed instance and re-pointed at the hosted front door: the frozen per-app hashes had drifted from what the discovery service reports and were refusing every attempt at prepare(). Approved by Carlos, 9 Sep. Rounds stored under 9e23c3dfb31f241a do not regrade against it. `_ReadOnlyStore` implemented only the two queries build_report used until a report began reading artifacts. Its own docstring invites the third: "if a page ever needs another query, add the one method it needs". Two deferrals recorded, neither ours to fix here: the vendored AutomationBench is plain 1.0.6 while achievable-50 declares 1.0.6+evalrepair.10 and the patched tree is not on this machine, so a run over that set would stop at the world-revision guard; and a Studio lock test spawns a subprocess with a 10 s timeout while importing wb_studio.runtime alone takes 9.4 s here, so the contender is killed before it can report the lock it correctly failed to take. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
The single-host-owner test spawns a second interpreter and gave it 10 seconds. Importing wb_studio.runtime alone takes about 9 s on a cold Windows interpreter, so the contender was killed before it could report the lock it had correctly failed to take, and a working lock read as a broken one. The budget is now 60 s and named, because how long an import takes is not what this test measures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
The same facts lived in three handoffs, five Portuguese duplicates, four near-identical tier sheets and a reconciliation note between two sessions. Consolidated, with every dropped fact folded into a surviving document. STATE-OF-THE-PROGRAM.md replaces the handoffs of 2, 3 and 8 September: where things stand, what is blocked and on whom, the decisions, the pre-round checklist. BOUNDARY-BENCH-AND-MONARCH.md carries the 9 September agreement in English -- where a mock exists the mock is the truth and the generator describes it; where only the real platform exists the vendor spec plus the observed body describe it. rounds/tier-rounds.md merges the four tier sheets, whose method was identical; the four task tables follow it byte-identically. Facts that existed in only one place and would have gone with the file: why the 5 and 6 September rounds are void (the grader was broken, not the competitors); that the Bedrock blocker is dead, which three documents still presented as live; that the 67% figure is a reconstruction whose source graph is gone; OneDrive deleting tracked files on 2 September; and the zod contract that silently drops undeclared keys, which the seed generator will meet the day it emits a new field. The tier sheets said "awaiting Carlos's go" beside hashes from before the 9-10 September re-derivation, so someone could have launched a US$ 20-40 round against hashes that no longer match. That sheet now opens by saying not to run it as written. Kept although they looked redundant: the create+run pilot sheet is the only record of how Monarch is set up per round, and the two achievable-50 sheets are separate pre-registrations for two tracks the methodology forbids pooling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
One finding took the whole session: the approval rules were broken, and four paid rounds had measured that rather than any competitor. Six defects, all in rules the bench derives itself, four of which made a rule name a path the world can never produce -- so no competitor could pass, however well it did the task. Recorded here rather than in a fourth handoff file: what was fixed, what it cost, and the six things still open, in the order they matter. The first is whether Monarch reads and does not write because the Airtable mock ignores filterByFormula -- worth settling before paying for a round that would measure the same trap four times. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
Three tasks: reconcile AGENTS.md with CLAUDE.md so nothing about method, tooling or governance changes because the harness did; keep fixing what is plainly broken (Carlos ruled on 10 Sep that these are corrections of things certainly wrong, not differences of approach, so they need no further sign-off); and redeploy Monarch on Railway from the branch where the remote main was merged into Carlos's fixes. Carries the one working rule that mattered most this week: a pattern over task files is a hypothesis, a reproduction against the simulator is evidence. Two sweeps were discarded for being heuristics, one of them reporting 1,168 false positives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GauKwDyVpHqdgh2Vqn9Nju
cgmattos
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WorkflowBench's approval-rule translation rejected valid simulator writes and missed collateral changes. This PR corrects those rules, exposes collateral counts, restores Monarch's front-door execution path, and adds execution and spending history to Langfuse.
AutomationBench's world, routes, task requests, seeds and assertions remain upstream originals. Only WorkflowBench's derived approval rules change. Previous rules, configurations, hashes and historical results remain available; affected attempts cannot be silently regraded under a new hash.
Changes:
41769e05aa5618a9→1912201bfea8a62atransition.wb telemetry status|flush|backfill|reconcileprovides inspection and recovery without model calls.Validation: the final code's Ubuntu CI passed 2,096 tests, with nine skips, plus both corpus checks. All 51 focused checks passed. Live zero-cost accounting, billing and summary observations were read from Langfuse; lost local acknowledgements were recovered by read-side lookup without reposting. The detached Windows run exposed five cases fixed by the final code and one local Studio socket failure (WinError 10053), also seen in isolation; the corresponding test passes in Ubuntu. The documentation-only follow-up also passed CI on the current head; GitHub reports the PR clean and mergeable. No model calls or infrastructure deployments were made for this validation.
Known limits remain explicit: Airtable ignores filterByFormula, the invoice assertion is weak, Jira content checks have limits, and achievable-50 stays blocked while evalrepair adoption is suspended. Langfuse history does not certify provider invoices or consolidate local/hosted ledgers. Hosted Studio needs deployment of this version to enable its new exporter.
Evidence: CI and telemetry, upstream limitations, pricing and diagnostic, historical accounting.