Skip to content

feat(agents): promote tool-call JSON repair to a ToolCallJsonRepair capability - #841

Open
mpfaffenberger wants to merge 4 commits into
mainfrom
feature/tool-call-json-repair-capability
Open

mpfaffenberger wants to merge 4 commits into
mainfrom
feature/tool-call-json-repair-capability

Conversation

@mpfaffenberger

@mpfaffenberger mpfaffenberger commented Aug 21, 2026 •

Copy link
Copy Markdown
Owner

ToolCallJsonRepair capability

Thirteenth in the capability series (#828–#836, #838–#840). Converts the tool-call JSON repair feature — until now an eager monkeypatch of ToolManager.validate_tool_call in pydantic_patches.py — into a first-class pydantic-ai capability on the before_tool_validate seam. This is the first conversion in the series that retires monkeypatch surgery rather than a constructor kwarg or a run-site closure.

What the feature does

LLMs sometimes emit slightly broken JSON in tool-call arguments (trailing commas, missing quotes, unclosed braces). Repairing the raw args before validation via json_repair avoids burning a retry round-trip.

How it moved

  • New code_puppy/agents/_json_repair.py — stateless ToolCallJsonRepair(AbstractCapability):
    • before_tool_validate repairs string args and mirrors the repair onto the live ToolCallPart (call.args = repaired) — the same in-place custody the patch performed, so message history records the JSON the tool actually ran with. Verified empirically: the part passed to the seam is the object recorded in run state.
    • build_tool_call_json_repair() returns [] when the optional json_repair dependency is absent (mirrors the patch's quiet skip and the build_tool_output_limits conditional-splice pattern).
    • Stateless plain dataclass → spec-constructible with inherited defaults.
  • Both construction sites (_builder.py, subagent_invocation.py) splice the capability. Position is inert — the tool-validate seam never interacts with the history hooks.
  • The patch is demoted to a guest-agent fallback — explicit-when-ours, fallback-for-guests, the same split feat: deliver Logfire tracing as an explicit Instrumentation capability #838 used for Logfire instrumentation. The patched validate_tool_call walks the run's ToolManager.root_capability tree with the public AbstractCapability.apply visitor and steps aside when ToolCallJsonRepair is present. Raw pydantic-ai agents built by plugins keep eager repair — the audit found one real dependent: wiggum's judge, which registers genuine read-only tools. btw's side-query agent has no tools (inert either way); shell_safety's agent builds through our builder (gets the capability).
    • The gate fails open: if the leaf-walk ever crashes, the patch repairs eagerly — old behavior, and idempotent alongside the capability (repairing repaired JSON is a no-op).

Parity audit

  • Seam placement: before_tool_validate is the only hook receiving raw pre-validation args; it fires exactly once per validate_tool_call (allow_partial=False only — no streaming-partial hazard).
  • Ordering vs the cp_ normalization patch: runtime order changes from repair args → normalize name → resolve to normalize name → resolve → repair args. Name normalization and arg repair are independent — inert reordering.
  • Bounded divergence (documented + pinned): calls to unknown/unavailable tools no longer get their recorded args repaired — _resolve_tool raises ModelRetry before the seam fires, where the patch repaired first and failed after. The call fails identically either way; history now keeps the model's true emitted bytes (arguably more honest). Pinned by test_unknown_tool_args_stay_raw_in_history.
  • Archaeology of the round: on pydantic-ai 2.31.0, tool-based structured output validates through validate_output_tool_call — a method the patch never wrapped. Output tools were never covered by this feature, so pydantic-ai's kind == 'output' hook carve-out changes nothing. (An output-tool repair via before_output_validate would be new behavior, deliberately out of scope — noted for a future PR if wanted.)

Tests

17 contract tests in tests/agents/test_tool_call_json_repair_capability.py:

  • direct-seam semantics: repair + custody, valid/dict/empty pass-throughs (object identity), swallowed repair failures, missing-dependency no-ops;
  • end-to-end FunctionModel runs: tool executes with repaired args, zero retries burned, history carries repaired bytes;
  • wire parity: guest-agent-under-patch vs capability-agent produce identical recorded args and output, plus retry-feedback parity on the unknown-tool path;
  • patch gating both ways: repair invoked exactly once with the capability present (patch stepped aside), guests still repaired; _run_owns_json_repair leaf-walk incl. CombinedCapability nesting;
  • wiring pins for both capabilities=[...] blocks + a seam-signature pin so a pydantic-ai hook-contract change fails loudly.

Full suite: 7615 passed, 0 failed (28 skipped, 1 xpassed). Reviewed by code-puppy clone: pass 1 APPROVE with 1 non-blocking + 1 nit (both applied in 103571e), pass 2 APPROVE with zero findings.

Merge-order note

Touches the same two capabilities=[...] blocks as the other twelve open capability PRs — whichever lands last eats a trivial rebase.

…apability

Convert the patch_tool_call_json_repair monkeypatch (eager
ToolManager.validate_tool_call surgery) into a first-class pydantic-ai
capability on the before_tool_validate seam.

- New code_puppy/agents/_json_repair.py: stateless ToolCallJsonRepair
  repairs malformed JSON string args via json_repair, mirrors the repair
  onto the live ToolCallPart (same history custody as the patch's
  in-place mutation), and returns the repaired args for validation.
  build_tool_call_json_repair() returns [] when json_repair is absent,
  mirroring the patch's quiet optional-dependency skip.
- Both construction sites (_builder.py + subagent_invocation.py) splice
  the capability; position inert (tool-validate seam never interacts
  with the history hooks).
- The monkeypatch is demoted to a guest-agent fallback: it steps aside
  when the run's root_capability tree contains ToolCallJsonRepair
  (public AbstractCapability.apply leaf-walk), and keeps repairing for
  raw pydantic-ai agents built by plugins (wiggum's tool-wielding
  judge). Explicit-when-ours, fallback-for-guests, same split as
  Logfire instrumentation.
- Bounded divergence, documented + pinned: unknown/unavailable tool
  calls no longer get their recorded args repaired (resolution raises
  before the seam fires); the call fails with the identical ModelRetry
  and history keeps the model's true emitted bytes.
- Archaeology: on pydantic-ai 2.31.0 output tools validate through
  validate_output_tool_call, which the patch never wrapped - output
  tools were never covered, so the seam's kind=='output' carve-out
  changes nothing.

16 contract tests: seam semantics, end-to-end FunctionModel runs with
history-custody pins, wire parity vs the eager patch, patch gating both
ways, unknown-tool divergence pin, wiring + seam-signature pins.
- Pin retry parity for the unknown-tool divergence: identical
  RetryPromptPart feedback and recovery under the eager patch and the
  capability; only the recorded args differ (raw vs repaired), which is
  the entire documented divergence. Note that unavailable tools share
  the same raise-before-seam code path.
- Strengthen the seam-signature pin to compare parameter names, kinds,
  and defaults (annotations excluded deliberately: the seam spells
  RawToolArgs, we spell the underlying union).
Resolve conflicts with main's HistoryCompaction, speculative code mode,
subagent recursion guard and shape-preserving tool-call JSON repair:
- keep ToolCallJsonRepair in both builders next to HistoryCompaction
- the fallback patch still steps aside when the capability owns repair,
  while the arguments-envelope sanitizer always runs
- ToolCallJsonRepair now reuses _repair_tool_call_json (strict parse
  first, reject shape-changing repairs) so both tiers share one policy
On main, patch_tool_call_callbacks writes a repaired dict view back to
call.args for unknown tools whenever apply_all_patches() has run (any
test importing cli_runner). The old history-bytes assertions therefore
depended on test ordering and failed in the full CI run. Assert the
capability's real contract instead: the before_tool_validate seam never
fires for an unresolvable tool.
@mpfaffenberger

Copy link
Copy Markdown
Owner Author

@douwebot please review

What I fixed on this pass:

  • Merged latest origin/main (it was 501 commits behind and conflicting). Conflicts were in _builder.py, subagent_invocation.py and pydantic_patches.py. ToolCallJsonRepair now sits next to main's new HistoryCompaction / speculative code mode / recursion guard. The fallback patch still steps aside when the capability owns repair, and main's {"arguments": ...} envelope sanitizer always runs.
  • Removed duplicated repair logic: ToolCallJsonRepair now reuses main's _repair_tool_call_json (strict parse first, rejects repairs that change the top-level shape), so the capability and the fallback patch follow one policy. Added a test for the shape guard.
  • Fixed CI: two unknown-tool tests asserted raw history bytes, but main's patch_tool_call_callbacks (applied whenever cli_runner is imported during the full suite) writes a repaired dict view back to call.args. So those tests failed depending on test order. They now check the capability's actual contract instead: the repair seam never fires for a tool that doesn't exist.
  • No review-bot findings were present. CI is green (quality, macOS tests, windows-encoding).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant