Skip to content

fix(bench): modern DataHub seeding, enrichment-volume reporting, and the first published pilot results#951

Merged
cjimti merged 1 commit into
mainfrom
fix/942-datahub-ingest-format
Jul 13, 2026
Merged

fix(bench): modern DataHub seeding, enrichment-volume reporting, and the first published pilot results#951
cjimti merged 1 commit into
mainfrom
fix/942-datahub-ingest-format

Conversation

@cjimti

@cjimti cjimti commented Jul 13, 2026

Copy link
Copy Markdown
Member

Part of #930. Fixes two bench-harness defects surfaced by standing the a2 arm up on a current DataHub, and starts the living results page that the benchmark phases (#943-#945) will grow, seeded with the phase 1 pilot numbers.

DataHub seeding works on current CLIs

The MCE emitter wrote the legacy datahub put --file bulk format, which no longer exists in current datahub CLIs (put is now a subcommand group requiring --urn). The emitter now writes the MCP file format the datahub ingest file source consumes (each aspect wrapped as a GenericAspect {"json": {...}}), the seed artifacts are regenerated, and make bench-seed-datahub generates a file-source recipe and drives datahub ingest against BENCH_DATAHUB_GMS (default http://localhost:8080). Verified live: 12 events ingested into a DataHub v1.6.0 quickstart running on OpenSearch.

Enrichment volume is no longer invisible in reports

Suite reports summed enrichment_tokens_dedup, which is zero when enrichment runs in full mode, so the a2 arm's enrichment volume displayed as zero even while audit rows showed enrichment_applied: true with per-call token counts. auditapi.Metrics and the suite summaries now carry enrichment_tokens_full (the delivered volume) alongside the dedup counter, and the human summary prints the full volume. Verified live: the a2 pilot run reports ~7.3k enrichment tokens across the knowledge suite.

Published results: a living page plus committed history

docs/reference/benchmarks.md is the new reference page (mkdocs nav and the docs/llms.txt / docs/llms-full.txt pair updated). It states the reading rule (arm-vs-arm on a pinned model; the model is disclosed, never the subject), and publishes the phase 1 pilot: manifest, arm-by-suite tables, per-trap-class breakdown, honest caveats, and a result-history table each future phase appends to. The raw run JSON (claude-sonnet-5, k=3, 60 episodes, zero harness failures) is committed under bench/results/v1.102.0-pilot/ as the first history entry, per the #930 results-history convention.

Pilot headline, also recorded on #930: knowledge-trap accuracy 60% (a0, raw tools) vs 100% (a2, platform), with the two tasks whose disambiguating fact lives only in the knowledge layer going 0/6 vs 6/6 while the baseline exhausted its 30-call budget on every attempt; knowledge-suite median tool calls 16 vs 8; median wall clock 67s vs 25s. Caveats are published alongside: both arms score 100% on trivial discovery (where a2 pays a small search-first friction cost), and two trap tasks fell to raw model inference; the page marks all pilot numbers as directional pending phase 2's larger suites and judge calibration.

Verification

  • make verify green on this branch (root gates, including doc-sync checks over the new docs).
  • Bench module gates green: build, vet, tests, golangci-lint.
  • Both fixes exercised against a live stack: seeding into DataHub v1.6.0/OpenSearch, then a full paid a2 run (30 episodes) whose report shows the enrichment volume; the a0/a2 result pair committed here is that run.

…the first published pilot results

Standing the a2 arm up on a current DataHub (v1.6.0 quickstart on
OpenSearch) surfaced two harness defects, both exercised live after the fix:

- The MCE emitter wrote the legacy datahub put --file bulk format, which
  current datahub CLIs no longer accept. Aspects are now emitted in the MCP
  file format (GenericAspect {"json": ...}), and bench-seed-datahub drives
  datahub ingest with a generated file-source recipe (BENCH_DATAHUB_GMS
  overrides the sink). Seed artifacts regenerated.
- Suite reports summed enrichment_tokens_dedup, which is zero when
  enrichment runs in full mode, so the a2 arm's enrichment volume read as
  zero. Metrics and reports now carry enrichment_tokens_full (the delivered
  volume) alongside dedup, and the suite table prints the full volume.

docs/reference/benchmarks.md is the new living results page (mkdocs nav +
llms pair updated): phase 1 pilot manifest, arm-by-suite and trap-class
tables, honest caveats, and a result-history table that future phases
append to. Raw pilot run JSON (claude-sonnet-5, k=3, 60 episodes, zero
harness failures) is committed under bench/results/v1.102.0-pilot/ as the
first history entry.

Pilot headline recorded on #930: knowledge-trap accuracy 60% (a0) vs 100%
(a2) with the policy-revenue tasks going 0/6 vs 6/6, median knowledge-suite
tool calls 16 vs 8, wall clock 67s vs 25s.
@codecov

codecov Bot commented Jul 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.04%. Comparing base (1984dd2) to head (ac20012).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #951      +/-   ##
==========================================
+ Coverage   89.03%   89.04%   +0.01%     
==========================================
  Files         424      424              
  Lines       48581    48581              
==========================================
+ Hits        43253    43260       +7     
+ Misses       3551     3547       -4     
+ Partials     1777     1774       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cjimti cjimti merged commit 3794bf2 into main Jul 13, 2026
11 checks passed
@cjimti cjimti deleted the fix/942-datahub-ingest-format branch July 13, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant