Skip to content

docs(bench): frame the benchmark report as one knowledge system, two halves#960

Merged
cjimti merged 1 commit into
mainfrom
bench-959-coupling-reframe
Jul 15, 2026
Merged

docs(bench): frame the benchmark report as one knowledge system, two halves#960
cjimti merged 1 commit into
mainfrom
bench-959-coupling-reframe

Conversation

@cjimti

@cjimti cjimti commented Jul 15, 2026

Copy link
Copy Markdown
Member

Summary

Reframes the published benchmark report (docs/reference/benchmarks.md) so the S1–S3 semantic ablation and the S5 memory/knowledge lifecycle read as one coupled knowledge system, not two independent features with two independent verdicts. Documentation-only: no benchmark numbers, costs, or data change — only the framing and narrative.

Why

The report presented the semantic ablation and the memory lifecycle as adjacent result sets, which invites the wrong conclusion — "semantic enrichment is a big win, the knowledge/memory system is sporadic" — as if they were separable products. They are not. The lifecycle (memory → insight → apply_knowledge) promotes captured knowledge into a sink — a DataHub entity/column description or a knowledge page — and the surfacing half (cross-enrichment and search) delivers that knowledge to the agent. The lifecycle populates the sinks that enrichment and search read from: there is no enrichment value without knowledge in the sinks, and no cross-session value without surfacing. Enrichment is a delivery channel for curated knowledge, not a standalone feature.

Changes (docs/reference/benchmarks.md)

  • Top-of-report framing. A new "one knowledge system, two halves" section (capture/curation half vs surfacing half) with a Mermaid diagram of agent teaches → memory → insight → apply_knowledge → sink → surface (enrichment / search).
  • S1–S3 reframed as validating the surfacing half: with knowledge already in the sinks, does the platform reliably deliver it? It does (+56 points on the knowledge traps). The narrative and the trap-class breakdown now name the two delivery channels and which facts each carries — units_cents / net_revenue live in DataHub descriptions and are recovered by the cross-enrichment channel alone (arm a1); fiscal_calendar / tier_boundary live only in knowledge pages and need the search channel (arm a2).
  • The "why a2 ties a3" section and the closing caveat now use the two-halves framing: a2 ties a3 because those single-session tasks exercise only the surfacing half, and capture-and-propagation is what produces the sink contents S1–S3 shows are effective once present.
  • S5's ~45% cross-identity transfer ceiling is located upstream. Because S1–S3 already validates surfacing, a transfer failure is very unlikely to be a delivery failure; it points at the capture-and-propagation half — the apply_knowledge promotion landing in an aspect enrichment reads (the mcp-datahub GetEntity fully populates dataset/dashboard entities but reads other entity types more sparsely), or the second identity surfacing the fact but not using it. Added as a Limitations note and a diagnosis in the scorecard narrative.
  • Section 1/2 headings retitled to name the halves ("Surfacing half (semantic layer) …", "Capture-and-propagation half (memory / knowledge lifecycle) …").
  • docs/llms.txt and docs/llms-full.txt updated to match (CLAUDE.md rule 11).

Notes

No code changes; make bench-test is unaffected. The Mermaid diagram follows the repo's diagram convention (CLAUDE.md rule 9). This addresses the framing gap without touching any measured result — the numbers, ranges, variance findings, and token-spend figures are exactly as published.

Closes #959

…halves (#959)

Reframe docs/reference/benchmarks.md so the S1-S3 semantic ablation and the S5
lifecycle read as one coupled knowledge system rather than two independent
features. Add a top-level capture -> sink -> surface framing with a Mermaid
diagram: the lifecycle (memory -> insight -> apply_knowledge) populates the sinks
(DataHub descriptions, knowledge pages) that the surfacing half (cross-enrichment
and search) delivers from.

- S1-S3 is framed as validating the surfacing half, naming the two delivery
  channels and which facts each carries (descriptions via enrichment, pages via
  search).
- The a2-ties-a3 explanation and the closing caveat use the two-halves framing.
- S5's ~45% cross-identity transfer ceiling is located upstream at capture and
  propagation (surfacing is already validated by S1-S3), with the apply_knowledge
  write path and entity-type coverage as the likely cause.
- Section 1/2 headings and docs/llms.txt, docs/llms-full.txt updated to match.

Refs #959.
@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.03%. Comparing base (85b7ca7) to head (56038f5).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #960   +/-   ##
=======================================
  Coverage   89.02%   89.03%           
=======================================
  Files         424      424           
  Lines       48581    48581           
=======================================
+ Hits        43251    43252    +1     
  Misses       3552     3552           
+ Partials     1778     1777    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cjimti cjimti merged commit 037ead9 into main Jul 15, 2026
10 checks passed
@cjimti cjimti deleted the bench-959-coupling-reframe branch July 15, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(bench): reframe the benchmark report to articulate the memory/knowledge <-> enrichment coupling

1 participant