feat(bench): phase-2 suites, execution-result grader, LLM judge, and cross-arm report (#943)

[cjimti]

Summary

Part of #930. Implements phase 2 (#943): the full agent-effectiveness benchmark instrument, extending the phase-1 pilot (#942) from a 10-task A0-vs-A2 probe to all four arms, three suites, deterministic plus execution-result graders, a versioned LLM judge, and a cross-arm report generator.

The design principle is unchanged: the benchmark ablates the PLATFORM, not the model. Every run holds the model, prompt scaffold, seed data, and task set constant, and varies only the platform configuration. The benchmark module stays a separate Go module, excluded from make verify by design, and its checks run via make bench-test.

Arms

All four arms are platform config profiles under bench/config/, not code forks. Each ablates one layer.

Arm	Profile	What the agent gets
a0	baseline	Raw toolkit tools only (trino_*, s3_*); no semantic provider, no search, cross-enrichment off, search-first gate off.
a1	enrichment	A0 plus semantic cross-enrichment: trino_*/s3_* results carry DataHub context automatically, but the persona withholds search and datahub_*, so enrichment runs without the discovery layer. This isolates enrichment's effect.
a2	knowledge	A1 plus the search tool, the search-first gate, and seeded knowledge pages. The shipped semantic-first configuration.
a3	lifecycle	A2 plus the memory/insight and apply_knowledge surface. On the single-episode S1-S4 suites it tracks A2 (nothing to recall yet); the lifecycle effect is measured by the S5 protocols in #944.

All four profiles parse and validate against the real platform.LoadConfig and Config.Validate. The DataHub arms (a1, a2, a3) require a DataHub quickstart seeded via make bench-seed-datahub.

Task set

87 tasks, generated deterministically from one fixed-seed dataset model; every ground truth is computed from the seeded rows, never hand-typed. Every task applies to all four arms.

• S1 discovery (17): "which table answers X", entity-graded. Several are knowledge-dependent (the deprecation of legacy_orders, the gross-only and stale nature of the pre-aggregated index).
• S2 analytical accuracy (45): exact numeric questions at BIRD-style tiers (single-table, join, temporal, cross-tab, top-N), plus four SQL-producing tasks graded by execution-result comparison. S2 states monetary units explicitly (cents), so it measures query formulation rather than the units trap.
• S3 knowledge traps (25) across six seeded trap classes, each mirroring a fixture the generator plants and each answerable plausibly-but-wrongly without the knowledge layer:

Trap class	The disambiguating fact, and where it lives
units_cents	Monetary columns are integer cents; stated in the column and dataset descriptions (enrichment-visible).
net_revenue	Revenue = amount - discount over completed orders only; in the dataset description and the revenue-policy page. The gross leader and the net leader differ by construction.
fiscal_calendar	The fiscal year runs Feb 1 through Jan 31; ONLY in the fiscal-calendar page, so it separates the knowledge arm from enrichment.
freshness_cutoff	The daily index stops at 2025-11-30, so post-cutoff questions must use raw orders; in the index description and page.
tier_boundary	A "key account" is any plus- or enterprise-tier customer; only in the tier-definitions page.
deprecated_table	legacy_orders is a partial, deprecated extract; the deprecation lives in metadata and the warehouse page.

The generator asserts, for every trap, that the naive reading differs from the knowledge-informed reading, so a trap that stops trapping fails the build.

Execution-result grader (BIRD-style)

SQL-producing tasks put a query on the FINAL ANSWER line. The grader extracts it, executes the candidate and the task's reference query, and compares result sets as multisets of rows by cell value (column aliasing and row order do not matter). The grader runs its queries through a dedicated admin-credentialed session, separate from every attempt handle, so its own tool calls never perturb an attempt's audit accounting; the reference result is cached per task. A transport failure on a candidate query is classified as a harness failure, not scored as a wrong answer, while a query that does not run is the agent's graded miss.

LLM judge

The deterministic graders score numbers and entities; the judge scores only the judgment-call rubric items they cannot, namely whether an S3 answer carried the required caveat. The judge model is pinned and the rubric versioned (bench/judge/rubric.yaml), so a judge or wording change bumps the version and historical scores stay attributable. The judge's agreement with human labels is computed over a committed 30-item calibration set (bench/judge/calibration.yaml) via make bench-calibrate.

Cross-arm report

benchrun -compare loads one results JSON per arm and renders the arm-by-suite comparison: accuracy with 95% percentile bootstrap CIs, pass^k, median and p90 tool calls, the S3 trap-class breakdown, and the headline per-arm accuracy deltas versus the baseline. The bootstrap uses a fixed resampling seed, so identical inputs produce identical intervals. make bench-compare writes the markdown page; the report also flags a run whose arms drifted in task-set hash or model.

Identity pool

The per-attempt identity pool grows from 32 to 264 keys, sized to cover the full 87-task set at k=3 (261 attempts). The keys use compact flow-style YAML; a regeneration snippet is documented in bench/README.md.

Commands

make bench-up BENCH_ARM=a0|a1|a2|a3   # compose stack + seeded warehouse + platform
make bench-smoke                       # scripted, no-API-key end-to-end validation
make bench-run BENCH_ARM=a2 K=3        # real run (needs ANTHROPIC_API_KEY)
make bench-compare                     # cross-arm tables + bootstrap CIs -> markdown
make bench-calibrate                   # judge-vs-human agreement rate

Verification

• make verify green; the root runtime is unchanged (the bench module is excluded from ./...).
• make bench-test green: build, vet, go test ./..., and golangci-lint (0 issues, same complexity ceilings as the root).
• New-code coverage: gen 98%, grade 97%, judge 91%, report 95%. The pipeline's execution-result path is exercised by an in-process integration test that assembles a real streamable-HTTP MCP server and drives the grader end-to-end; its remaining uncovered lines are connection-plumbing error branches, consistent with the existing mcpc layer.
• Two adversarial code-review passes; all findings addressed with targeted tests.

What remains before #943 fully closes

• A live end-to-end run against the compose stack plus a DataHub quickstart: the scripted smoke and the consistency of each new task's reference SQL against its computed ground truth are validated by unit tests and mirror the pilot's verified patterns, but have not yet been exercised against a live Trino.
• The published judge agreement rate and the first full-run numbers (a model-API run; see the budget and adapter notes in bench: claude-cli adapter — run benchmark episodes via claude -p (subscription, no API key) #949 and bench(phase 4): published benchmarks page, results history, regression baseline mode, smoke workflow #945).

One documented grader limitation: an unfenced multi-line SQL answer with no code block is truncated to its first line. The task prompt and the run's format instruction steer models to a single line or a fenced block (both fully supported), so this only affects a non-compliant answer shape.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.04%. Comparing base (3794bf2) to head (a558ad7).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #952   +/-   ##
=======================================
  Coverage   89.04%   89.04%           
=======================================
  Files         424      424           
  Lines       48581    48581           
=======================================
  Hits        43260    43260           
  Misses       3547     3547           
  Partials     1774     1774           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.