feat(agent): read this turn's edits back before the answer is sealed

[kovtcharov-amd]

Branch: feat/verify-edits-before-the-answer-is-sealed
Base: e0fd6717f609a5be7ca8676a6be2c869fd4fe4b2 (e0fd6717f)
Commits:

SHA	Subject
bdfc3402885aedfb8fc3488a3aaa85da3ae2c3ab	feat(agent): read this turn's edits back before the answer is sealed
836b57016695d59db4362ade466d60eb032d376f	fix(agent): read back only text-file edits, not every path a tool reports

Relationship to #4719 and #4745

#4719 ("keep the read-back check out of the user's answer") touches a
model-facing prompt string inside the completion check, and only fires when
there is an artifact gap — the model claimed a save the evidence ledger cannot
corroborate. It tells the model not to narrate that read-back in its reply.

This change is a different mechanism answering a different question: an
in-process, read-only check the loop runs over the files the turn actually
wrote, whenever it wrote any. It does not touch #4719's prompt string, does not
re-instruct the model to read anything, and does not reintroduce the narration
#4719 removed — its findings only enter the conversation when something is
wrong. The two compose; neither is redundant.

#4745 ("keep the answer a model sends alongside its last tool call") requires
the promoted answer to be in place before the seam's checks run. It is: the
promotion happens earlier in _process_query_impl than this check, so the
read-back still runs against exactly what the user will see.

The problem

A tool result saying "status": "success" is not the same as a file that is
still on disk, still has content, and still parses.

Today the only thing that closes that gap is the model choosing to check — and
whether it does is a property of the model, not of the loop. Given the same
tools and the same prompt, some models read their work back every time and some
never do. That means the behaviour disappears the moment the model changes,
which is not a property anyone can build on.

The change

The loop is better placed to do it. It already records which files this turn
wrote, so just before the answer is sealed it opens each of them and asks three
questions the tool result cannot answer:

• is it still there,
• does it still have content,
• and, for Python and JSON, does it still parse.

If anything comes back wrong, the findings go into the context and the turn gets
two steps to fix them.

The free steps are the load-bearing part

Those two steps are granted on top of max_steps, not taken out of it.

Verifying currently costs steps from the same budget as working, so a model
under budget pressure is right to skip it — and the pressure peaks at the end of
a turn, which is exactly where the edits are. Making the repair free removes the
incentive to look away. Charging for it would make this feature self-defeating.

Design notes

Where "which files did this turn write" comes from. The completion-evidence
ledger, which every write tool feeds. _turn_file_edits looks like the obvious
source and is not: it is appended only for a tool result carrying
operation == "edit_file", and nothing in file_io_tools sets that field, so
reading it would make the check silently do nothing for every shipped file tool.
There is a test driving the real tools through the real loop specifically to pin
that.

Deliberately narrow. Read-only, in-process, no shell, no subprocess, no
repository walk. It cannot be slow in proportion to the repository and cannot
touch anything the turn had not already touched, so it is safe to run every
time. A file type it cannot parse is checked for existence and content and
otherwise left alone, rather than having a syntax guessed at for it.

Bounded. Two rounds per turn, after which the turn is looping and the
findings go into the answer instead of driving another repair.

GAIA_AGENT_VERIFY_EDITS=0 turns it off.

What this is not

This does not tell you whether a change is correct — nothing that needs no
configuration could. It tells you the change is not obviously broken, which
is the failure the loop can see by itself. Judging correctness needs the
repository's own tests, which is a different and much larger piece of work.

Two existing tests opt out

Both register write tools that report success without touching the disk, so the
read-back correctly finds the file they named missing and spends a repair step
saying so. One pins the loop's exact model-call count; the other scripts a fixed
list of replies that the extra call exhausts. Neither is about writing files, so
the honest fix is to turn the check off for them and say why in the fixture, not
to weaken the check.

Fix: only text-file edits are read back

The first version of this branch took its file list straight from the
completion-evidence ledger. That ledger records more than edits: it also holds
the outputs of tools that generate files (generate_image,
text_to_speech, take_screenshot, ...) and paths an executor reported. So a
turn that generated an image had the image path read back as if the model had
edited it, found a "problem", and spent a repair round asking the model to fix
a file it never edited. tests/unit/agents/test_image_outcome_guard.py caught
it: two of its tests failed with an unexpected extra model call.

The second commit fixes the cause rather than the tests (which are unchanged):

• the ledger marks an entry edited when its last writer was one of the
  file-editing tools in WRITE_TOOLS, and only those are read back. Any later
  change by something else replaces the entry, so a file a script rewrote or
  deleted on purpose after its edit is not reported as broken;
• binary files are skipped, by extension or by a NUL byte in the content;
• reads go through the agent's read boundary without prompting, so a path the
  model could not read is not read here either;
• a file over 2 MiB is checked for presence only.

New regression tests cover a generated image (unit, and through the real loop
with the check on), a read-only tool, an executor output, a file deleted on
purpose, binary content and a path outside the read boundary; five of them fail
without the fix.

Risk

The findings enter the conversation and can add turns, so this changes model
behaviour and wants an eval run before merge.

Test plan

• pytest tests/unit/test_post_edit_verification.py tests/unit/agents/test_cut_off_reply_guard.py tests/unit/agents/test_parse_error_recovery.py tests/unit/agents/test_completion_evidence.py tests/unit/agents/test_read_file_line_ranges.py tests/unit/agents/test_image_outcome_guard.py tests/unit/eval/test_flagship_tasks.py::test_an_adversarial_task_rejects_the_shortcut
  on this branch alone — 251 passed, 2 skipped (test_post_edit_verification.py
  is 36 tests: a missing file, an emptied file, broken Python and broken JSON,
  the free repair steps not counting against max_steps, the two-round
  ceiling, an unparseable file type being left alone, the env knob, and the
  over-reach cases above)
• tests/unit/agents/test_image_outcome_guard.py — 8 passed, unmodified
• Full tests/unit — see below
• python util/lint.py --black --isort
• Eval (LLM-affecting): compare against the same run with
  GAIA_AGENT_VERIFY_EDITS=0

Full unit suite

tests/unit, Windows, --timeout=300, run on unmodified main (e0fd6717f)
and on a disposable build with all four open branches stacked (line endings in file tools, step-budget notices, post-edit
read-back, model-endpoint retry):

	Passed	Failed	Skipped
main e0fd6717f	19,373	5	410
stacked build	19,602	5	410

The same five tests fail on both, and none of them is touched by this branch:
three installer TUI-detection tests, test_file_access_scope.py::test_no_allowlist_still_defaults_to_cwd,
and test_lemonade_asr.py::TestRealServer::test_real_transcription_round_trip,
which needs a live server. No test fails on the stacked build that passes on
main.

Stacking with other open branches

Applies cleanly to main on its own. It conflicts textually with the step-budget notices branch (feat/tell-the-model-how-much-step-budget-is-left), whichever lands second: both add an env-knob function after _TOOL_USER_WAIT and per-turn state in __init__ and at the top of _process_query_impl in src/gaia/agents/base/agent.py. The resolution is to keep both sides — two separate functions and both sets of fields.

🤖 Generated with Claude Code

[github-actions]

Verdict: Request changes

Before the agent gives its final answer, it now reads back the files it edited during the turn. If a file is missing, empty, or no longer parses as Python or JSON, the agent gets two extra steps to fix it, and those steps don't count against its step budget. The design is sound and well tested. Three things need fixing before merge:

• A file that's still broken after the repair rounds is never mentioned. The PR says the problems go into the answer after two rounds. In the code they are collected and then nothing reads them, so the agent can still say "done" over a file that doesn't parse. That is the silent outcome this feature exists to prevent. Add them to the answer.
• Valid files get reported as broken. JSON files that allow comments (tsconfig.json, VS Code settings) and files saved with a UTF-8 byte-order mark (common on Windows) fail the strict parse. The model is then told to "fix" a file that is fine, which costs turns and may strip comments the user wanted.
• The agent eval hasn't run. This changes what the model sees at the end of every editing turn, so CLAUDE.md requires an eval before merge. The test plan leaves it unchecked.

Real-world evidence

N/A: this changes the agent loop internally and adds no CLI, API, or MCP surface, so the evidence bundle has nothing to capture. The verdict rests on the code review and the PR's unit-test results. How the change affects model behaviour is still unmeasured until the eval runs.

🔍 Technical details

🟡 Findings that outlive the round cap are dropped (src/gaia/agents/base/agent.py:7334)

_verify_turn_edits extends self._edit_verification_findings. A grep shows nothing outside its own reset and extend lines reads that list. The constant's comment says "Past that the turn is looping, and the gaps go into the answer instead", but once _edit_verification_rounds reaches _MAX_EDIT_VERIFICATIONS, the gate at ~9960 is skipped and finalize_answer runs as if nothing was found. Keep the last round's unresolved problems (not the running total, which also holds problems already fixed in round 1) and append them the way _turn_ungrounded feeds the scope note. Add a test that runs the loop to the ceiling and asserts the broken path appears in the final answer.

🟡 False "broken" reports on JSONC and BOM files (src/gaia/agents/base/agent.py:7317-7329)

• json.loads('{}') and ast.parse('x=1') both raise. Decoding with utf-8-sig fixes that:

              text = data.decode("utf-8-sig", errors="replace")
  

• .json files that allow comments or trailing commas (tsconfig.json, .vscode/*.json, devcontainer.json) never parsed as strict JSON. The message "is no longer valid JSON" is also wrong for them, because nothing checked the file before the edit. Two options: record whether the file parsed before the edit and report only a change from parsing to not parsing, or skip known JSONC names. Without one of these, the model may "fix" a file by removing comments the user wanted.

🟡 Eval required before merge

This touches the end-of-turn prompt flow (_EDIT_VERIFICATION_PROMPT and the extra steps), so it falls under CLAUDE.md's "Run agent evals when changing LLM-affecting code paths". Dispatch eval_flagship.yml and compare against tests/fixtures/eval_baselines/gaia-flagship/. Comparing with GAIA_AGENT_VERIFY_EDITS=0, as the PR proposes, is a useful extra check but doesn't replace the baseline comparison.

🟢 Nits

• Comments too long (agent.py:1131-1150, _files_written_this_turn docstring, the gate comment at ~9955): these multi-paragraph rationale blocks go against CLAUDE.md's "Code Comments — Short or Skip". The reasoning already lives in the PR body, so one line per invariant is enough.
• Private constant imported across modules (agent.py:44): _BINARY_SUFFIXES is private to completion.py. Make it public, or add a small is_binary_path() helper there.

Strengths

• Reading the file list from the evidence ledger instead of _turn_file_edits is correct, and a real-tool, real-loop test pins it so it can't silently become a no-op.
• The second commit fixes the cause of the false positives (generated images, executor outputs, deliberate deletions) with a new edited flag instead of loosening the tests. Each over-reach case has a regression test.
• Reads go through the read validator with prompt_user=False, are capped at 2 MiB, and never spawn a shell. That keeps an always-on check cheap and inside the read boundary.

[kovtcharov-amd]

Holding until v0.25.0-rc1 is tagged: this changes agent behavior, and the release eval now running (https://github.com/amd/gaia/actions/runs/37436249503) measures main at 9ce6598 without it. It merges right after the tag, and the next eval will measure it.