Skip to content

Agentic refactoring and full MCP support#1502

Open
gustavo-grieco wants to merge 65 commits into
masterfrom
dev-agents
Open

Agentic refactoring and full MCP support#1502
gustavo-grieco wants to merge 65 commits into
masterfrom
dev-agents

Conversation

@gustavo-grieco

@gustavo-grieco gustavo-grieco commented Dec 18, 2025

Copy link
Copy Markdown
Collaborator

This branch contains a rewrite of some core features of Echidna to transform workers into agents, which can receive commands and collaborate with each other. It also allows to easily add MCP commands to query and guide the fuzzing campaign.

image

Commands

This code is still a work in progress but it can already tested with the few command implemented:

  • status(): Show fuzzing campaign status
  • target(): Show the name and the ABI of the target contract
  • reload_corpus(): Reload the transactions from the corpus, but without replay them
  • dump_lcov(): Dump coverage in LCOV format"
  • inject_fuzz_transactions(string): Inject a sequence of transaction to fuzz with optional concrete arguments (e.g. f(1, ?, 3) ; g(?, -1))
  • inject_symb_transaction(string): Inject a single of transaction to explore using symbolic execution with optional concrete arguments (e.g. f(1, ?, 3))
  • clear_fuzz_priorities(): Clear the function prioritization list used in fuzzing
  • read_logs(): Read the last 100 log messages [disabled in the current code, perhaps not necessary]
  • show_coverage(string): Show coverage report for a particular contract
  • avoid_function(string): Exclude a transaction from the list of randomly generated transactions

All these commands are subject to change, and there will be an experimentation phase where we will challenge the agents to increase the coverage (or break properties) using different commands

There are some additional changes in the logs to convey useful information for the agent, in particular, when new coverage is found:

[2025-12-18 11:45:22.26] [Worker 2] New coverage: 15176 instr, 7 contracts, 19 seqs in corpus (mintAndApprove)

How to test this PR

  1. Open your echidna-compatible fuzzing project in Visual Studio Code, make sure you have Github Copilot installed.
  2. Compile this branch or download the echidna executable from our CI tests.
  3. Start/resume an Echidna campaign for the agent to guide. Very recommended to use text mode (--format text) for this test. Append the --server 3000 to your command line to start the MCP server in the port 3000.
  4. Create the .vscode/mcp.json file with the following content:
{
  "servers": {
    "Echidna fuzzing campaign": {
      "type": "http",
      "url": "http://localhost:3000/mcp"
    }
  }
}

Once you save the file, you will be able to click on the "Run" button next the server name to make sure the it is detected by Copilot. It should list the number of commands available (currently 7).
5. Everything should be ready to go, open the Chat window (Cmd + Shift + P and write /chat) and start prompting. My recommended option to quickly start:

An Echidna fuzzing campaign is currently running and has already achieved baseline coverage using the default strategy, meaning trivial execution paths are mostly explored.

Using the available MCP interface **only** (do not modify any code or functions):

1. **Identify the campaign context**
   - Determine the fuzzing target and current campaign status using the appropriate MCP commands.

2. **Analyze coverage**
   - Inspect coverage for the relevant contracts using `show_coverage`.
   - Identify execution paths or code regions with **low coverage** that are *theoretically reachable* from the target logic.

3. **Design targeted fuzzing sequences**
   - Once you understand the contract behavior, use `inject_fuzz_transactions` to prioritize one or more transaction sequences.
   - Separate multiple calls with `;` (e.g., `f(1,?,?) ; g(?,2,5)`).
   - Combine **concrete values and random parameters** strategically.
   - **Do not use `?` for all parameters**, as those cases are already well-covered by the existing campaign.
   - **Avoid making all parameters concrete when possible**. As a *recommendation* (not a strict requirement), leave at least one parameter as `?` per call to allow the fuzzer to continue exploring the input space rather than repeatedly replaying a single fixed execution. Fully concrete calls may still be used when they intentionally target a very specific path or invariant.
   - **Injection semantics:** the prioritized sequence is **not executed in isolation**. Echidna starts from an existing transaction sequence in the current corpus and **inserts the injected sequence at a random position within it**, not necessarily at the beginning. Design injected calls assuming relevant state may already exist.

4. **Evaluate results**
   - After injecting transactions, run `sleep 20` to allow fuzzing to progress.
   - Then call `status` to check whether additional coverage was discovered or any invariant (e.g., `assert`) failed.

5. **Reset priorities**
   - Clear prioritized calls using `clear_fuzz_priorities` to return the fuzzer to its default random sampling strategy.

Focus on crafting transaction sequences that are most likely to exercise uncovered logic or trigger invariant violations, while still allowing the fuzzer enough freedom to explore variations.

@CLAassistant

CLAassistant commented Dec 18, 2025

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

datradito added a commit to datradito/echidna-mcp that referenced this pull request Mar 18, 2026
The clean PR branch is based on upstream/dev-agents (PR crytic#1502) which uses
the '--server' flag name. Documentation was written for the development
branch which used '--mcp-port', causing inconsistency.

Changes:
- AGENT_TESTING_GUIDE.md: Update all command examples
- test-mcp-client.py: Fix error message
- examples/README.md: Update all 3 command examples
- examples/simple_agent.py: Fix error message
- examples/langgraph_agent.py: Fix error message
- tests/mcp/conftest.py: Fix pytest fixture command
- .gitignore: Add Python cache patterns

This ensures documentation matches the actual upstream implementation.
datradito added a commit to datradito/echidna-mcp that referenced this pull request Mar 18, 2026
- Update examples/README.md to list correct 7 tools
- Fix mcp_client_wrapper.py to use upstream tool names:
  * inject_fuzz_transactions (not inject_transaction)
  * clear_fuzz_priorities (not clear_priorities)
  * status, target, show_coverage, reload_corpus, dump_lcov
- Mark old test files as skipped (use old tool names):
  * test_corpus.py - get_corpus_size, inspect_corpus, find_transaction
  * test_injection.py - inject_transaction (old signature)
  * test_prioritization.py - prioritize_function
  * test_read_logs.py - read_logs (commented out in upstream)
- Update docstring: 7 active tools (not 9)

These tests are preserved for reference but skipped until updated to match
upstream API from PR crytic#1502.
datradito added a commit to datradito/echidna-mcp that referenced this pull request Mar 18, 2026
The clean PR branch is based on upstream/dev-agents (PR crytic#1502) which uses
the '--server' flag name. Documentation was written for the development
branch which used '--mcp-port', causing inconsistency.

Changes:
- AGENT_TESTING_GUIDE.md: Update all command examples
- test-mcp-client.py: Fix error message
- examples/README.md: Update all 3 command examples
- examples/simple_agent.py: Fix error message
- examples/langgraph_agent.py: Fix error message
- tests/mcp/conftest.py: Fix pytest fixture command
- .gitignore: Add Python cache patterns

This ensures documentation matches the actual upstream implementation.
datradito added a commit to datradito/echidna-mcp that referenced this pull request Mar 18, 2026
- Update examples/README.md to list correct 7 tools
- Fix mcp_client_wrapper.py to use upstream tool names:
  * inject_fuzz_transactions (not inject_transaction)
  * clear_fuzz_priorities (not clear_priorities)
  * status, target, show_coverage, reload_corpus, dump_lcov
- Mark old test files as skipped (use old tool names):
  * test_corpus.py - get_corpus_size, inspect_corpus, find_transaction
  * test_injection.py - inject_transaction (old signature)
  * test_prioritization.py - prioritize_function
  * test_read_logs.py - read_logs (commented out in upstream)
- Update docstring: 7 active tools (not 9)

These tests are preserved for reference but skipped until updated to match
upstream API from PR crytic#1502.
gustavo-grieco and others added 18 commits March 18, 2026 13:22
Vendors the out-of-process MCP test harness, example LangGraph/Claude agent,
test contracts, and CI workflow authored by Dani Tradito (@datradito) in
#1509, as the base for a robust external MCP testing strategy
on this branch (protocol-conformance + Claude/Codex client simulation are
layered on top in follow-up commits).

Source: #1509

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Layers an out-of-process protocol-conformance and client-compatibility suite on
top of the integration scaffolding vendored from #1509:

- conftest.py: one session-scoped Echidna+MCP server shared by all modules
  (honors ECHIDNA_MCP_URL to target an already-running server)
- _mcp_client.py: dependency-light wire-level JSON-RPC/HTTP helpers
- test_mcp_conformance.py: asserts the transport guarantees strict clients need
  (notification -> 202 empty body, GET SSE -> 405, handshake, error codes)
- test_mcp_codex.py: replays Codex's strict rmcp handshake
- test_mcp_claude.py: drives the server with the official `mcp` SDK (Claude)
- test_mcp.py: refactored to share the conftest fixture (originally @datradito)
- mcp-tests.yml: CI runs the full suite (tools + conformance + Claude + Codex)

Verified end-to-end: 12/12 pass against the current (fixed) server; against the
pre-fix haskell-mcp-server (9fd60af) the conformance + Codex tests fail on the
`200 {}` notification (the bug that broke Codex) while the lenient Claude SDK
passes -- which is exactly why the strict conformance/Codex layer is needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread .github/workflows/mcp-tests.yml Fixed
gustavo-grieco and others added 5 commits June 16, 2026 22:00
Addresses the CodeQL 'workflow does not contain permissions' alert on
#1502: the job only checks out the repo and runs tests, so
restrict GITHUB_TOKEN to contents: read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MCP job was building echidna from source (cold 'nix build .#echidna', no
cache), taking 30-60 min and never reaching the test steps. Mirror the repo's
other Nix jobs: install via DeterminateSystems and pull prebuilt closures from
the trailofbits Cachix, so the build takes minutes. Add a concurrency group so
a new push cancels the previous in-flight run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The job had the echidna binary and Python but not crytic-compile/solc, so
echidna exited immediately ('crytic-compile not installed') and every MCP test
errored in the fixture. Install crytic-compile + solc-select and select solc
0.8.25 for the test contract.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The nix build pinned the pre-fix haskell-mcp-server (9fd60af), so the
nix-built echidna (CI, docker, releases) shipped the codex-incompatible MCP
transport even though stack.yaml was updated. Point the flake at the fixed
commit too. sha256 placeholder is corrected in the follow-up once CI reports it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants