fix(bench): modern DataHub seeding, enrichment-volume reporting, and the first published pilot results#951
Merged
Merged
Conversation
…the first published pilot results
Standing the a2 arm up on a current DataHub (v1.6.0 quickstart on
OpenSearch) surfaced two harness defects, both exercised live after the fix:
- The MCE emitter wrote the legacy datahub put --file bulk format, which
current datahub CLIs no longer accept. Aspects are now emitted in the MCP
file format (GenericAspect {"json": ...}), and bench-seed-datahub drives
datahub ingest with a generated file-source recipe (BENCH_DATAHUB_GMS
overrides the sink). Seed artifacts regenerated.
- Suite reports summed enrichment_tokens_dedup, which is zero when
enrichment runs in full mode, so the a2 arm's enrichment volume read as
zero. Metrics and reports now carry enrichment_tokens_full (the delivered
volume) alongside dedup, and the suite table prints the full volume.
docs/reference/benchmarks.md is the new living results page (mkdocs nav +
llms pair updated): phase 1 pilot manifest, arm-by-suite and trap-class
tables, honest caveats, and a result-history table that future phases
append to. Raw pilot run JSON (claude-sonnet-5, k=3, 60 episodes, zero
harness failures) is committed under bench/results/v1.102.0-pilot/ as the
first history entry.
Pilot headline recorded on #930: knowledge-trap accuracy 60% (a0) vs 100%
(a2) with the policy-revenue tasks going 0/6 vs 6/6, median knowledge-suite
tool calls 16 vs 8, wall clock 67s vs 25s.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #951 +/- ##
==========================================
+ Coverage 89.03% 89.04% +0.01%
==========================================
Files 424 424
Lines 48581 48581
==========================================
+ Hits 43253 43260 +7
+ Misses 3551 3547 -4
+ Partials 1777 1774 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #930. Fixes two bench-harness defects surfaced by standing the
a2arm up on a current DataHub, and starts the living results page that the benchmark phases (#943-#945) will grow, seeded with the phase 1 pilot numbers.DataHub seeding works on current CLIs
The MCE emitter wrote the legacy
datahub put --filebulk format, which no longer exists in currentdatahubCLIs (putis now a subcommand group requiring--urn). The emitter now writes the MCP file format thedatahub ingestfile source consumes (each aspect wrapped as a GenericAspect{"json": {...}}), the seed artifacts are regenerated, andmake bench-seed-datahubgenerates a file-source recipe and drivesdatahub ingestagainstBENCH_DATAHUB_GMS(defaulthttp://localhost:8080). Verified live: 12 events ingested into a DataHub v1.6.0 quickstart running on OpenSearch.Enrichment volume is no longer invisible in reports
Suite reports summed
enrichment_tokens_dedup, which is zero when enrichment runs in full mode, so thea2arm's enrichment volume displayed as zero even while audit rows showedenrichment_applied: truewith per-call token counts.auditapi.Metricsand the suite summaries now carryenrichment_tokens_full(the delivered volume) alongside the dedup counter, and the human summary prints the full volume. Verified live: the a2 pilot run reports ~7.3k enrichment tokens across the knowledge suite.Published results: a living page plus committed history
docs/reference/benchmarks.mdis the new reference page (mkdocs nav and thedocs/llms.txt/docs/llms-full.txtpair updated). It states the reading rule (arm-vs-arm on a pinned model; the model is disclosed, never the subject), and publishes the phase 1 pilot: manifest, arm-by-suite tables, per-trap-class breakdown, honest caveats, and a result-history table each future phase appends to. The raw run JSON (claude-sonnet-5, k=3, 60 episodes, zero harness failures) is committed underbench/results/v1.102.0-pilot/as the first history entry, per the #930 results-history convention.Pilot headline, also recorded on #930: knowledge-trap accuracy 60% (a0, raw tools) vs 100% (a2, platform), with the two tasks whose disambiguating fact lives only in the knowledge layer going 0/6 vs 6/6 while the baseline exhausted its 30-call budget on every attempt; knowledge-suite median tool calls 16 vs 8; median wall clock 67s vs 25s. Caveats are published alongside: both arms score 100% on trivial discovery (where a2 pays a small search-first friction cost), and two trap tasks fell to raw model inference; the page marks all pilot numbers as directional pending phase 2's larger suites and judge calibration.
Verification
make verifygreen on this branch (root gates, including doc-sync checks over the new docs).