bench(phase 3): S5 memory-insight-knowledge lifecycle protocols with API-verified state (#944)

[cjimti]

Closes #944.

Summary

This adds the S5 memory-insight-knowledge lifecycle suite to the agent-effectiveness benchmark (bench/, issue #930, phase 3). Where S1-S4 ask single-session questions and ablate the platform arm, S5 measures the platform's own lifecycle: teach a fact in one session, recall it in another, promote it into shared knowledge, answer it from a different identity, correct it, and refuse to invent facts that were never taught. Every lifecycle state transition is verified through the platform's admin insights and changesets APIs rather than inferred from a transcript. It runs on the a3 arm, and this PR also publishes the first live results.

What a protocol does

Each protocol is a sequence of fresh MCP sessions. Every protocol runs teach, recall, and abstain; each additionally runs either promote+transfer or supersede (they are mutually exclusive, see below):

1. Teach — an identity states a fact conversationally and saves it; the harness verifies via the insights API that a pending insight was captured and entity-linked.
2. Recall — the same identity, a fresh session, answers a question that needs the fact; graded deterministically, and the run records whether search surfaced the memory unprompted.
3. Promote — the harness plays reviewer: approve, then apply_knowledge to one of two sinks (an entity description or a knowledge page), verified through the changesets API.
4. Transfer — a different identity, a fresh session, answers the same question, which it can only do because promotion pushed the fact into shared knowledge.
5. Update — the teacher corrects the fact; recall must flip to the new value and the taught insight must show superseded, not left live as a duplicate.
6. Abstain — a question about a fact never taught must be answered "I do not know", not fabricated.

Design decisions

• Novel facts. Each protocol teaches a definition deliberately absent from the seeded knowledge pages and catalog metadata, so recall and transfer are clean: the only way to answer is the taught memory or the promoted knowledge, never a pre-seeded fixture. Recall and update answers are computed from the seeded dataset, never hand-typed.
• Promote and update are mutually exclusive per protocol (enforced by the protocol validator). The platform deliberately never supersedes an already-applied insight, so a promoted (applied) fact cannot then be cleanly superseded. The set is ten promote+transfer protocols and five supersede protocols, each on its own fact; both mechanics are measured.
• API-verified state. Capture, promotion, and supersede are read back from the admin insights and changesets endpoints. Duplicate detection is scoped to the specific taught insight, so a persistent knowledge store that reuses pool identities across runs does not cross-contaminate the metrics.
• Identity pool. Each attempt consumes a distinct teacher and learner identity so the search-first gate's per-user discovery scope never leaks; the lifecycle requires a pool (teacher and learner must differ) and refuses to start when protocols x k x 2 exceeds it.
• Embedding provider. The a3 arm enables Ollama nomic-embed-text: recall-first supersede detects a restatement by vector similarity, so the supersede mechanic is unmeasurable without it.
• Self-accounting, crash-safe runs. Each episode records its token usage (rolled up into the run metrics), and both the lifecycle and S1-S3 runners flush the results file after every protocol/attempt, so a run reports its own cost basis and never loses completed work to an interruption.

Components

• bench/internal/protocol — the multi-episode protocol schema, loader, validation, and canonical hash.
• bench/internal/lifecycleapi — the admin insights and changesets read-back plus the reviewer-side approve and apply drivers.
• bench/internal/lifecycle — the protocol runner (a fresh session and rotating identity per episode), the stage graders, the API-verified state checks, and the metrics model.
• bench/internal/gen/emit_protocols.go — generates the 15 protocols and the no-API-key scripted lifecycle smoke, with answers computed from the seeded dataset.
• benchrun -lifecycle mode and the bench-lifecycle, bench-lifecycle-smoke, and bench-lifecycle-report Makefile targets.

Published pilot results

First live run on a3 (claude-sonnet-5, k=1), 13 of the 15 protocols in one pass. Full manifest, findings, and caveats are in docs/reference/benchmarks.md; raw results are under bench/results/v1.102.0-s5-partial/.

Metric	Result	What it measures
Capture rate	85% (11/13)	the agent recorded the taught fact and entity-linked it
Personal recall	62% (8/13)	fresh-session recall of the taught fact
Unprompted surface	100% (11/11)	search surfaced the memory without prompting
Transfer rate	67% (4/6)	a different identity answered correctly after promotion
Update correctness	100% (5/5)	a correction flipped recall to the new value
Duplicate rate (lower better)	40% (2/5)	supersede left the old insight live
Abstention	100% (13/13)	never fabricated a fact that was never taught
Full-lifecycle pass	38% (5/13)	every applicable stage of the protocol succeeded

Abstention and update-correctness both held at 100% with a real model: the knowledge layer suppressed rather than amplified fabrication, and every correction the memory layer detected flipped recall to the new value. Personal recall (62%) and the similarity-gated duplicate rate (40%) are the honest weak spots. Teach-once-answer-forever held on 4 of 6 promotion chains across both sinks.

Running

make bench-up BENCH_ARM=a3        # boot the lifecycle arm (needs a DataHub quickstart + ollama)
make bench-lifecycle-smoke        # scripted, no-API-key end-to-end validation
make bench-lifecycle K=1          # real run (needs ANTHROPIC_API_KEY)
make bench-lifecycle-report       # human summary of the last run

Testing

make bench-test (build, vet, race-enabled unit tests) and golangci-lint are green. The lifecycle runner is exercised end to end against a fake platform assembled from a real MCP server plus the admin REST surface, so the harness is proven through genuine protocol wiring and genuine API-verified transitions. The generated protocols and the lifecycle smoke are covered by the committed-artifact determinism test. Bench is deliberately not part of make verify, mirroring the load harness.

Scope

This is the S5 harness plus a first directional pilot, not the full published whitepaper. The pilot is a single arm at k=1, so it carries no pass^k reliability and no arm comparison; treat the numbers as directional. A statistically robust run (k=3 with confidence intervals) and the phase-4 regression gate (#945) follow.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.04%. Comparing base (160225d) to head (69715e4).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #953   +/-   ##
=======================================
  Coverage   89.04%   89.04%           
=======================================
  Files         424      424           
  Lines       48581    48581           
=======================================
  Hits        43257    43257           
  Misses       3549     3549           
  Partials     1775     1775           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.