docs(bench): frame the benchmark report as one knowledge system, two halves#960
Merged
Conversation
…halves (#959) Reframe docs/reference/benchmarks.md so the S1-S3 semantic ablation and the S5 lifecycle read as one coupled knowledge system rather than two independent features. Add a top-level capture -> sink -> surface framing with a Mermaid diagram: the lifecycle (memory -> insight -> apply_knowledge) populates the sinks (DataHub descriptions, knowledge pages) that the surfacing half (cross-enrichment and search) delivers from. - S1-S3 is framed as validating the surfacing half, naming the two delivery channels and which facts each carries (descriptions via enrichment, pages via search). - The a2-ties-a3 explanation and the closing caveat use the two-halves framing. - S5's ~45% cross-identity transfer ceiling is located upstream at capture and propagation (surfacing is already validated by S1-S3), with the apply_knowledge write path and entity-type coverage as the likely cause. - Section 1/2 headings and docs/llms.txt, docs/llms-full.txt updated to match. Refs #959.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #960 +/- ##
=======================================
Coverage 89.02% 89.03%
=======================================
Files 424 424
Lines 48581 48581
=======================================
+ Hits 43251 43252 +1
Misses 3552 3552
+ Partials 1778 1777 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reframes the published benchmark report (
docs/reference/benchmarks.md) so the S1–S3 semantic ablation and the S5 memory/knowledge lifecycle read as one coupled knowledge system, not two independent features with two independent verdicts. Documentation-only: no benchmark numbers, costs, or data change — only the framing and narrative.Why
The report presented the semantic ablation and the memory lifecycle as adjacent result sets, which invites the wrong conclusion — "semantic enrichment is a big win, the knowledge/memory system is sporadic" — as if they were separable products. They are not. The lifecycle (
memory→ insight →apply_knowledge) promotes captured knowledge into a sink — a DataHub entity/column description or a knowledge page — and the surfacing half (cross-enrichment and search) delivers that knowledge to the agent. The lifecycle populates the sinks that enrichment and search read from: there is no enrichment value without knowledge in the sinks, and no cross-session value without surfacing. Enrichment is a delivery channel for curated knowledge, not a standalone feature.Changes (
docs/reference/benchmarks.md)agent teaches → memory → insight → apply_knowledge → sink → surface (enrichment / search).units_cents/net_revenuelive in DataHub descriptions and are recovered by the cross-enrichment channel alone (arm a1);fiscal_calendar/tier_boundarylive only in knowledge pages and need the search channel (arm a2).apply_knowledgepromotion landing in an aspect enrichment reads (the mcp-datahubGetEntityfully populates dataset/dashboard entities but reads other entity types more sparsely), or the second identity surfacing the fact but not using it. Added as a Limitations note and a diagnosis in the scorecard narrative.docs/llms.txtanddocs/llms-full.txtupdated to match (CLAUDE.md rule 11).Notes
No code changes;
make bench-testis unaffected. The Mermaid diagram follows the repo's diagram convention (CLAUDE.md rule 9). This addresses the framing gap without touching any measured result — the numbers, ranges, variance findings, and token-spend figures are exactly as published.Closes #959