feat(bench): cold-start knowledge-growth suite (#963)

[cjimti]

Summary

Adds a cold-start knowledge-growth suite to the agent-effectiveness benchmark (bench/). Where S1-S3 and S5 ablate the platform with a pre-seeded knowledge base, this suite starts from an empty enrichment layer and measures the platform getting smarter as knowledge accumulates: a learning curve whose independent variable is the amount of promoted, shared knowledge, holding the model, prompt, task set, and dataset constant.

It is the highest-value item of the #958 epic (part 1 of 4; siblings #964, #965, #966 remain open).

What it measures

The suite runs on the a3 arm against an empty baseline: an undocumented DataHub (seed/datahub/bench_mces_empty.json, entities present but no descriptions, column docs, tags, or glossary) and no knowledge pages (bench-up with BENCH_SEED_PAGES=0). Over an ordered curriculum of six lessons, one per S3 trap class, the harness:

1. Teaches each fact (a teacher identity states it and captures it via memory_capture), then promotes it to its sink through apply_knowledge: a DataHub entity description (units, freshness, deprecation) or a portal knowledge page (net-revenue policy, fiscal calendar, tier definitions). Each lesson teaches the same S3 trap fact the A2 seed pre-loads, so the trap suite reaches its A2 accuracy ceiling once all six are promoted. Capture and promotion are verified through the admin insights and changesets APIs, never inferred from transcripts.
2. Evaluates at every checkpoint (the empty baseline and after each lesson) by re-running the fixed S3 trap suite with a fresh, never-taught evaluator identity. Its only knowledge source is what the platform surfaces (cross-enrichment for the DataHub-sink facts, search for the page-sink facts), so accuracy climbs only because promotion pushed the fact into shared knowledge. This isolates the delivery of promoted knowledge, the coupling between the lifecycle and the enrichment layer, not an evaluator's own memory.

The report (make bench-cold-start-report) is a learning curve: per checkpoint (indexed 0 for the empty baseline through 6 after the last promotion) it prints the eval set's accuracy and enrichment coverage against the count of lessons promoted so far, followed by a per-trap-class baseline-to-final breakdown showing which lesson unlocked which class. Enrichment coverage is the fraction of tool calls whose response carried cross-enrichment, read from the audit trail.

Changes

Area	Change
internal/curriculum	Curriculum schema, loader, and set hash (new, 90.0% covered)
internal/coldstart	Curriculum runner, fresh-evaluator eval loop, learning-curve metrics, results model (new, 81.8% covered)
internal/promote	Shared reviewer-promotion path (approve + apply_knowledge + verify), extracted so S5 lifecycle and cold-start share one implementation (new, 82.4% covered)
internal/lifecycle	Rewired to the shared promote package; behavior preserved (the pre-extraction approve note string is retained)
internal/gen	Empty-DataHub emitter (bench_mces_empty.json), curriculum generator, and scripted-smoke generator; committed artifacts are drift-checked
internal/auditapi	Metrics.EnrichmentCoverage() (EnrichedCalls / AuditedCalls), no new audit fields
benchrun	-cold-start run mode and -cold-start -summarize
Makefile	bench-cold-start / -smoke / -report, bench-seed-datahub-empty, and BENCH_SEED_PAGES=0 on bench-up for the empty baseline
docs/	bench/README.md, docs/reference/benchmarks.md, docs/llms.txt, docs/llms-full.txt

Committed generated artifacts: bench/curriculum/cs-traps.yaml, bench/curriculum/scripted-cold-start-smoke.json, bench/seed/datahub/bench_mces_empty.json. Regenerate with make bench-gen; a drift test fails if they diverge from the fixed-seed generator.

Design notes

The curriculum reuses the exact facts and knowledge-page bodies the A2 seed encodes, so the promoted description text is byte-identical to A2's for the entities it touches and the trap suite's ceiling is the A2 level. apply_knowledge restores the fact-bearing description and page channels but not A2's auxiliary aspects (globalTags, the structured deprecation aspect, column-level docs); the S3 traps read the fact text, which is restored.

Lesson order is the curve's x-axis, run foundational-first (units before net-revenue, then the calendar / freshness / tier / deprecation facts), so a multi-fact trap flips to correct only once every fact it needs has landed.

Identity assignment keeps teachers and evaluators disjoint: a distinct teacher per lesson (so capture verification is cleanly scoped to that lesson's insight) and fresh evaluators per checkpoint. A run refuses to start when the lessons plus per-checkpoint evaluators exceed the identity pool.

Validation

• In-process integration test (TestColdStartCurveClimbs) wires the real Run over a fake platform (a real MCP server plus the admin insight/changeset/audit REST) and asserts the curve climbs [0, 0.5, 1] as lessons are promoted, proving the teach -> capture -> promote -> eval loop is correctly assembled.
• Live end-to-end run of the scripted smoke against a running a3 platform: 0 harness failures, 6/6 lessons captured and promoted (6 real changesets across both sink types), 25 eval tasks x 7 checkpoints graded.
• Bench module gates green: make bench-test (build, vet, race tests) and golangci-lint run ./... (0 issues), make doc-check OK.

A real model-driven learning-curve run is gated by model budget; the whole loop is validated with no API key by make bench-cold-start-smoke. The suite supports the subscription claude-cli path as well (make bench-cold-start LLM=claude-cli MODEL=sonnet K=1).

How to run

make bench-up BENCH_ARM=a3 BENCH_SEED_PAGES=0   # boot a3 with an empty enrichment layer
make bench-seed-datahub-empty                    # entities present, undocumented
make bench-cold-start-smoke                      # scripted no-API-key loop validation
make bench-cold-start K=1                         # real learning-curve run (needs a model)
make bench-cold-start-report                      # print the learning curve

Notes for reviewers

bench/ is a separate Go module, deliberately excluded from root make verify (same rationale as test/load); its gates are make bench-test and its own golangci-lint. This PR touches no root-module Go code.

Closes #963.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.04%. Comparing base (e50de5f) to head (d9af6a3).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #967   +/-   ##
=======================================
  Coverage   89.04%   89.04%           
=======================================
  Files         424      424           
  Lines       48581    48581           
=======================================
  Hits        43257    43257           
  Misses       3549     3549           
  Partials     1775     1775           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.