Skip to content

fix(studio): deep-dive fixes — tool-error bodies, public reasoning, handler 500s, event log costs - #5

Merged
LucasWaki merged 1 commit into
mainfrom
007-benchmark-foundations
Sep 10, 2026
Merged

fix(studio): deep-dive fixes — tool-error bodies, public reasoning, handler 500s, event log costs#5
LucasWaki merged 1 commit into
mainfrom
007-benchmark-foundations

Conversation

@LucasWaki

Copy link
Copy Markdown
Collaborator

Summary

  • Application error bodies ({"error": {...}} with status completed) now count as tool errors in attempt reports and stories; the story no longer claims every call succeeded.
  • Public-audience reports keep the provider's reasoning summaries out of the story timeline.
  • The HTTP handler answers unexpected defects with a 500 (exception name only, traceback to the log) instead of dropping the connection; the scheduler logs the exceptions it survives.
  • Studio.emit caches the event count instead of re-counting the file per event; the SSE stream tails the log by byte offset.
  • The CLI loop applies the Studio rule: no text or an abnormal stop reason is not a finished answer.
  • run_story handles repetitions; two O(n²) trace scans are one-pass; duplicate who in usage.py, shadowed field, late-binding lambda and unused imports removed.
  • Browser suite updated for the story-first attempt sheet and the new report section; the fixture server's process tree is killed on Windows.

Test plan

  • uv run python -m pytest tests -q — 2128 passed, 3 skipped
  • node tests/browser/suite.cjs — all checks pass, no stale listener left

🤖 Generated with Claude Code

…reasoning out, the handler answers defects, the event log and stream stop re-reading the file

- reports.tool_error reads {"error": ...} bodies the simulator returns with status completed; the story no longer claims every call succeeded
- narrative.without_reasoning keeps the provider's summaries out of public reports
- Handler.unexpected answers a 500 with the exception's name; the scheduler logs what it survives
- Studio.emit caches the event count; Studio.events_from tails the log by byte offset for the stream
- api_loop follows the Studio rule: no text or an abnormal stop is not a finished answer
- run_story handles repetitions; one-pass trace indexes; usage.py, config.py, agents.py and unused-import cleanups
- browser suite: story-first attempt sheet, the new report section, and the fixture server's process tree killed on Windows

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@LucasWaki
LucasWaki merged commit 9684117 into main Sep 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant