Skip to content

fix(mcp): one dead MCP server no longer aborts the whole run - #909

Closed
breedx wants to merge 1 commit into
mpfaffenberger:mainfrom
breedx:fix/one-dead-mcp-server-aborts-the-run
Closed

breedx wants to merge 1 commit into
mpfaffenberger:mainfrom
breedx:fix/one-dead-mcp-server-aborts-the-run

Conversation

@breedx

@breedx breedx commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

One MCP server that fails to start kills the entire run — no model call, no answer.

_run_agent_task_body enters every agent_run_context into one shared AsyncExitStack. A dead server raises from __aenter__, the stack unwinds, and _do_run on the next line never executes.

Fix: enter each context in its own try. A failure is reported by name; the run continues without that server's tools.

 async with AsyncExitStack() as stack:
     for cm in run_ctxs:
-        await stack.enter_async_context(cm)
+        try:
+            await stack.enter_async_context(cm)
+        except Exception as exc:
+            _note_unusable_run_context(cm, exc, group_id)
     return await _do_run(prompt_payload)

Full suite: 7680 passed. ruff check + format --check clean.


Why it looks handled but isn't

The except* McpError handler downstream prints:

An MCP server failed during this run. Run /mcp logs <name> for details, or unbind it via /agents → B.

That reads like a warning attached to your answer. There is no answer — the coroutine falls off the end and returns None. The message makes a total failure look like a partial one, which is why this took a while to spot.

How it was found

A connector whose uvx dependency resolve failed. The process existed, so every status flag said RUNNING and pre-run health checks passed cleanly. It only failed at context-entry time — by which point the all-or-nothing stack had already made it fatal.

No amount of pre-run probing catches this; the failure happens after every check has run. That's why the fix has to be at the entry seam rather than in a health filter.

Design note

A connector is a capability, not a precondition. Losing one should cost its tools, not the answer — the agent still has its built-in file and shell tools, which is most of what it does.

Two details worth noting:

  • Servers that did enter are still owned by the stack, so a partial failure can't leak a subprocess.
  • A server failing early no longer discards the healthy ones after it. Previously the loop stopped at the first failure, so connectors later in the list were silently never entered either.
Tests

tests/agents/test_run_context_degrades.py:

  • the old shape raises, never reaches the run body, and stops entering at the first failure
  • the new shape reaches it, keeps every healthy connector including those after the failure, survives all servers failing, and still unwinds what it entered

The single suite failure (test_copilot_auth_model::test_ghe_host_is_passed_through) is pre-existing — verified to reproduce identically on an unmodified checkout of main.

`_run_agent_task_body` enters every `agent_run_context` into a shared
`AsyncExitStack`:

    async with AsyncExitStack() as stack:
        for cm in run_ctxs:
            await stack.enter_async_context(cm)
        return await _do_run(prompt_payload)

An MCP server whose subprocess never came up raises from `__aenter__`. Entered
as a single unit, that one failure unwinds the stack and `_do_run` — the model
call — is never reached. A user with six connectors and one broken one gets no
inference at all.

The `except* McpError` handler downstream hides how bad this is. It prints
"An MCP server failed during this run. Run /mcp logs <name> for details" and
returns, which reads as a warning attached to an answer. There is no answer:
the coroutine falls off the end and returns None.

Observed with a connector whose `uvx` dependency resolve failed: the process
existed, so every status flag said RUNNING and the pre-run health checks
passed. It only failed at context-entry time, by which point the all-or-
nothing stack had already made it fatal.

Each context now enters in its own try. A failure is reported by name and the
run continues without that server's tools. A connector is a capability, not a
precondition — losing one should cost its tools, not the answer. The agent
still has its built-in file and shell tools, which is most of what it does.

Servers that DID enter are still owned by the stack, so a partial failure
cannot leak a subprocess, and a server failing early no longer silently
discards the healthy ones after it in the list.

Tests state the bug directly: the old shape raises and never reaches the run
body, and stops entering at the first failure; the new one reaches it, keeps
every healthy connector including those after the failure, survives all
servers failing, and still unwinds what it entered.

Full suite: 7680 passed. The single failure
(`test_copilot_auth_model::test_ghe_host_is_passed_through`) is pre-existing
and reproduces identically on an unmodified checkout.
@breedx
breedx marked this pull request as draft September 3, 2026 15:26
@breedx

breedx commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Closing this outdated implementation. Guarding the outer run-context loop does not adequately cover the actual MCP toolset entry and recovery lifecycle. This needs a separately scoped connector-lifecycle fix with end-to-end failure-containment, recovery and cleanup tests, rather than retaining this patch as the proposed solution. The underlying MCP resilience issue remains open; identity/prompt work is separate.

@breedx breedx closed this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant