Skip to content

Defer undeclared tool-surface changes from check to verify#237

Merged
pengfei-threemoonslab merged 3 commits into
mainfrom
claude/check-scan-defer
Jun 23, 2026
Merged

Defer undeclared tool-surface changes from check to verify#237
pengfei-threemoonslab merged 3 commits into
mainfrom
claude/check-scan-defer

Conversation

@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor

Summary

  • shipgate check (and the MCP shipgate.check tool) returned allow on a capability-adding diff that touches a tool/capability surface the manifest does not declare — or when there is no manifest — even though the trigger already flags the diff in scope. Only verify compiles the capability delta, so an agent acting on that boundary allow could ship a change verify would block.
  • main already defers manifest-declared tool sources (capability_surfaces_changedcoverage_gap). This PR extends that coverage to the undeclared / no-manifest case (the common "agent just added a new, not-yet-registered tool file"):
    • Fires only when a content-driven run_shipgate trigger rule matched (e.g. TRIGGER-MCP-EXPORT-CHANGED), never the blanket force_run rule — so a docs-only edit in an opted-in repo stays allow (no noise).
    • Requires a changed file the boundary evaluator does not itself inspect (not is_boundary_path), so a clean allow over .codex/config.toml, AGENTS.md, or the Shipgate workflow stays authoritative.
    • Escalates allowwarn and routes to agents-shipgate detect (declare-then-verify), because verify only gates declared surfaces — routing straight to verify would falsely reassure on a surface it never scans.
  • Never downgrades a real boundary block / require_review. The MCP shipgate.check tool inherits the fix automatically via the shared build_codex_agent_result path.

Behavior

Diff Before After
Capability add, surface declared in tool_sources warn → verify unchanged
Capability add, no manifest allow warn → detect
Capability add, surface not declared allow warn → detect
.codex/config.toml repair (boundary-evaluated) allow unchanged
Docs edit in opted-in repo allow unchanged

Type

  • CLI or GitHub Action behavior

Verification

CI is authoritative for python -m ruff check ., python -m compileall -q src tests, and python -m pytest.

Additional local checks run:

  • pytest tests/test_codex_boundary_check.py tests/test_agent_protocol.py — green (incl. 5 new tests covering the undeclared/no-manifest gap, the never-downgrade-a-block invariant, and the no-noise-on-docs property).
  • ruff check on changed files — clean.
  • Full suite green except test_latency_budget::test_scenarios_scale_sublinearly, a load-sensitive perf test that fails identically on the unmodified base and passes in isolation (unrelated to this change).
  • Confirmed end-to-end that the MCP shipgate.check tool returns warndetect on an undeclared MCP tool-add.

Release-readiness notes

  • No user-code import added to default scan paths
  • No network access added to default scan paths
  • New or changed check IDs are documented in docs/checks.md — N/A: no new check IDs (adds a undeclared_capability_surface diagnostic + coverage trace only)
  • Report/schema changes are additive or documented in STABILITY.md — N/A: no schema change; uses existing AgentResultV1/CodexBoundaryResultV1 fields

🤖 Generated with Claude Code

pengfei-threemoonslab and others added 2 commits June 22, 2026 11:09
`shipgate check` (and the MCP `shipgate.check` tool) returned `allow` on a
capability-adding diff that touches a tool/capability surface the manifest does
not declare — or when there is no manifest — even though the trigger already
flags the diff in scope. That is a false clear: only `verify` compiles the
capability delta, so an agent acting on a boundary `allow` could ship a change
`verify` would block.

main already defers manifest-declared tool sources (`capability_surfaces_changed`
→ `coverage_gap`). This extends that coverage to the undeclared / no-manifest
case:

- Fire only when a content-driven `run_shipgate` trigger rule matched (e.g.
  TRIGGER-MCP-EXPORT-CHANGED), never the blanket `force_run` rule — so a
  docs-only edit in an opted-in repo stays `allow` (no noise).
- Require a changed file the boundary evaluator does not itself inspect
  (`not is_boundary_path`), so a clean `allow` over `.codex/config.toml`,
  `AGENTS.md`, or the Shipgate workflow stays authoritative.
- Escalate `allow` → `warn` and route to `agents-shipgate detect`
  (declare-then-verify), because `verify` only gates *declared* surfaces;
  routing straight to verify would falsely reassure on a surface it never scans.

Never downgrades a real boundary `block` / `require_review`. The MCP
`shipgate.check` tool inherits the fix automatically via the shared
`build_codex_agent_result` path. No schema or check-ID changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses two review findings on the boundary coverage-gap defer.

P1 — mixed declared + undeclared diffs only routed to `verify`. The undeclared
gap was gated on `not coverage_surfaces`, making declared and undeclared
mutually exclusive: a diff that changed a declared MCP export *and* an
undeclared `api/openapi.yaml` routed to `verify`, which never scans the
undeclared surface, so the new capability escaped the gate. Undeclared surfaces
are now computed independently, per-file, and take precedence — a mixed diff
routes to `detect` (declare-then-verify), which then covers both.

P2 — `_trigger_in_scope` treated every `run_shipgate` rule as a tool-surface
rule, so `shipgate.yaml` (TRIGGER-SHIPGATE-MANIFEST), `prompts/**`
(TRIGGER-PROMPTS-OR-POLICIES), and the CI gate were mislabeled
`undeclared_capability_surface` and routed to `detect`. Classification now keys
on the actual tool-source rules only (MCP export, OpenAPI, static tool
inventory, Codex plugin, n8n, @function_tool), excluding the manifest,
prompts/policies, CI workflow, and `.codex` boundary config (which the boundary
check itself evaluates).

Implementation: `build_codex_agent_result` classifies each changed file
per-file via the public trigger evaluator (glob rules match the path;
diff_contains rules match the file's added lines), excluding boundary paths and
already-declared files, and passes an explicit `undeclared_capability_surfaces`
list to `evaluate_codex_boundary_result`. The coarse `_trigger_in_scope` +
`is_boundary_path` heuristic is removed in favor of that explicit list.

Tests: added mixed declared+undeclared cases (direct + via `check`) and
manifest/prompts no-false-detect cases. Full suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

Addressed both findings in a721bba.

P1 (mixed declared + undeclared → only verify). Undeclared surfaces are now computed independently and per-file, not gated on not coverage_surfaces, and take precedence over the declared coverage gap. A diff that changes a declared MCP export and an undeclared api/openapi.yaml now routes to detect (declare-then-verify) and the diagnostic names the undeclared surface, instead of a verify that never scans it.

P2 (_trigger_in_scope matched non-tool-source rules). Removed the coarse run_shipgate-action heuristic. Each changed file is now classified against only the tool-source trigger rulesMCP-EXPORT, OPENAPI-SPEC, STATIC-TOOL-INVENTORY, CODEX-PLUGIN, N8N-WORKFLOW, FUNCTION-TOOL-DECORATOR — explicitly excluding TRIGGER-SHIPGATE-MANIFEST, TRIGGER-PROMPTS-OR-POLICIES, TRIGGER-SHIPGATE-CI-WORKFLOW, and TRIGGER-CODEX-BOUNDARY-CONFIG-CHANGED. A shipgate.yaml or prompts/system.md edit no longer reports undeclared_capability_surface/detect.

Classification reuses the public trigger evaluator per file (glob rules match the path; diff_contains rules match that file's added lines), so it stays in lock-step with the trigger catalog rather than duplicating glob logic. New tests cover the mixed-diff (P1) and manifest/prompts (P2) cases; full suite green.

`_undeclared_tool_surfaces_changed` treated any matched tool-source rule as a
tool surface, ignoring the evaluator's winning verdict. A docs or test file
that incidentally mentions `@tool` / `@function_tool` matches
`TRIGGER-FUNCTION-TOOL-DECORATOR` but ALSO `TRIGGER-DOCS-ONLY-NEGATIVE`, which
beats `run_shipgate` — so the catalog skips it. The classifier nonetheless
flagged `README.md` / `tests/test_*.py` as an `undeclared_capability_surface`
and routed to `detect`, contradicting the trigger catalog
(docs/triggers.json: docs/tests that incidentally mention `@tool` must not run).

Require the per-file trigger to actually run (`run_shipgate` true) in addition
to a tool-source rule appearing in `matched_rules`. A real `@function_tool`
source (run_shipgate true) still classifies; a docs/test mention (skip) no
longer does.

Tests: docs file and tests/ file that mention the decorator stay `allow`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

Fixed in 9528237.

The per-file classifier now requires the evaluator's winning verdict (run_shipgate true), not merely a tool-source rule in matched_rules. A docs/test file that incidentally mentions @tool matches TRIGGER-FUNCTION-TOOL-DECORATOR but also TRIGGER-DOCS-ONLY-NEGATIVE, which beats run_shipgate — so the catalog skips it and the file is no longer flagged undeclared_capability_surface/detect. A real @function_tool source (where the trigger actually runs) still classifies.

Verified: README.md and tests/test_x.py adding prose with @function_toolallow; agent.py adding @function_toolwarndetect. New tests cover both; full suite green.

@pengfei-threemoonslab pengfei-threemoonslab merged commit 56eb191 into main Jun 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant