Skip to content

Commit a93bf1e

Browse files
feat(xtest): read benchmark arms and provenance from the installation manifest
When $BENCH_INSTALLATION_MANIFEST names a manifest (`otdf-local env` exports it after `otdf-sdk-mgr install benchmark`), pytest --bench measures exactly the arms preparation installed, in request order, without anyone restating them. Explicit --bench-refs or --bench-baseline/ --bench-candidate still win. - perf/installation.py reads the manifest contract (schema v1alpha1) without importing otdf-sdk-mgr: the arms, the pins, the status and metadata.sdk. It checks the schema version and refuses a manifest it cannot fully read. An unknown commit stays None rather than "None". - Set but unreadable is a usage error at configure time, not a quiet fallback to whatever is in dist/. - A neutral preparation skips that SDK's cells with its reason; the run reports NEUTRAL instead of NOTHING MEASURED and does not fail. A failed or partial preparation also skips, and the run fails as having measured nothing. - The JSON artifact carries metadata.installation (each arm's alias, commit and dist tag) and metadata.preparation. - test_sdk_performance requests the payload, ciphertext, attribute and scratch fixtures only once a cell is known to be measurable, so a neutral preparation needs no running platform. Part 3c of the DSPX-4372-t1 plan; completes stage 2's item 7. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
1 parent 656c11e commit a93bf1e

11 files changed

Lines changed: 617 additions & 23 deletions

File tree

‎.github/workflows/check.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ jobs:
5252
uv run --frozen --no-build pytest --no-header -q
5353
test_bench_stats.py test_bench_measure.py test_bench_runner.py
5454
test_bench_arms.py test_bench_config.py test_bench_report.py
55-
test_sdk_commands.py test_tdfs_units.py test_encryption_units.py
55+
test_bench_installation.py test_sdk_commands.py test_tdfs_units.py test_encryption_units.py
5656
test_sizes_units.py test_zip64_units.py test_assertions_units.py
5757
working-directory: xtest
5858
- name: Lint and test otdf-local

‎xtest/conftest.py‎

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -257,9 +257,9 @@ def _add_benchmark_options(parser: pytest.Parser):
257257
"--bench-refs",
258258
help="2 to 4 builds of one sdk to compare, comma- or space-separated, "
259259
"e.g. 'go@v0.29.0,go@main,go@my-branch'. The first is the reference "
260-
"every other build is compared against. Defaults to the newest "
261-
"installed release against main. Supersedes "
262-
"--bench-baseline/--bench-candidate",
260+
"every other build is compared against. Defaults to the arms recorded "
261+
"in $BENCH_INSTALLATION_MANIFEST when set, else the newest installed "
262+
"release against main. Supersedes --bench-baseline/--bench-candidate",
263263
)
264264
group.addoption(
265265
"--bench-baseline",
@@ -456,8 +456,8 @@ def _parametrize_bench_cells(metafunc: pytest.Metafunc):
456456
def _bench_sdk_names(config: pytest.Config) -> list[str]:
457457
"""SDKs the benchmark fans out over, in --sdks order."""
458458
# --sdks may be version-qualified (go@main); benchmark arms come from
459-
# --bench-refs or --bench-baseline/--bench-candidate instead, so only the
460-
# name matters.
459+
# --bench-refs, --bench-baseline/--bench-candidate, or the installation
460+
# manifest instead, so only the name matters.
461461
specs = config.getoption("--sdks") or " ".join(typing.get_args(tdfs.sdk_type))
462462
return list(dict.fromkeys(s.split("@", 1)[0] for s in str(specs).split()))
463463

@@ -485,6 +485,7 @@ def pytest_configure(config: pytest.Config):
485485
arm_count,
486486
config_from_options,
487487
disk_shortfall,
488+
installation_for,
488489
payloads_from_options,
489490
)
490491
from perf.cells import payloads_to_generate
@@ -495,6 +496,8 @@ def pytest_configure(config: pytest.Config):
495496
validate_sdk_name,
496497
)
497498

499+
# An unreadable manifest is a usage error now, not a fallback later.
500+
installation_for(config)
498501
bench_config = config_from_options(config)
499502
payloads = payloads_from_options(config)
500503
refs = config.getoption("--bench-refs", default=None)
@@ -626,7 +629,7 @@ def pytest_sessionfinish(session: pytest.Session, exitstatus: int):
626629
reporter = config.pluginmanager.get_plugin("terminalreporter")
627630
if reporter is not None:
628631
reporter.write_sep("=", "benchmark results")
629-
reporter.write_line(gate.summary)
632+
reporter.write_line(report.headline(recorder, gate))
630633
reporter.write_line(f"raw samples and statistics: {json_path}")
631634

632635
if config.getoption("--bench-no-gate", default=False):
@@ -636,7 +639,14 @@ def pytest_sessionfinish(session: pytest.Session, exitstatus: int):
636639
# it with a green tick and an empty table is the one outcome nobody
637640
# inspects, so a benchmark that has quietly stopped measuring can survive
638641
# indefinitely. Every reason a cell skips is already in the report.
639-
if gate.should_fail or gate.nothing_measured:
642+
#
643+
# The one exception is a neutral preparation: every requested alias
644+
# resolved to one commit, so there was deliberately nothing to compare.
645+
# That is an answer, reported as such, and it reads the same locally and
646+
# in CI.
647+
if gate.should_fail:
648+
session.exitstatus = pytest.ExitCode.TESTS_FAILED
649+
elif gate.nothing_measured and not bench.preparation_is_neutral(config):
640650
session.exitstatus = pytest.ExitCode.TESTS_FAILED
641651

642652

‎xtest/fixtures/bench.py‎

Lines changed: 89 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
The experiment matrix, the payload files, the arm selection, and the shared
44
time budget all live here. The measurement loop itself is in ``perf/runner.py``
55
and the statistics in ``perf/stats.py``; this module is the glue that turns
6-
pytest's world (options, fixtures, SDK discovery) into the runner's world
7-
(2 to 4 arms and a config).
6+
pytest's world (options, fixtures, SDK discovery, the installation manifest)
7+
into the runner's world (2 to 4 arms and a config).
88
"""
99

1010
from __future__ import annotations
@@ -22,11 +22,14 @@
2222

2323
import abac
2424
import tdfs
25-
from perf import report
25+
from perf import installation, report
2626
from perf.cells import BenchCell, Payload, parse_payloads, payloads_to_generate
2727
from perf.config import budget_seconds, requested_arms
2828
from perf.runner import Arm, BenchConfig, Budget, Invocation
2929

30+
#: The run's installation manifest, read once in ``pytest_configure``.
31+
INSTALLATION_KEY: pytest.StashKey[installation.Installation | None] = pytest.StashKey()
32+
3033

3134
class ArmSelectionError(Exception):
3235
"""The builds a comparison needs are not all installed."""
@@ -117,8 +120,30 @@ def resolve(spec: str, role: str) -> tdfs.SDK:
117120
# --- Session-scoped configuration -------------------------------------------
118121

119122

123+
def installation_for(config: pytest.Config) -> installation.Installation | None:
124+
"""The manifest this run was prepared with, or None if it was not.
125+
126+
Read once and kept on the config, so every fixture and the end-of-run
127+
report see the same record even if the file changes mid-session.
128+
"""
129+
if INSTALLATION_KEY not in config.stash:
130+
try:
131+
config.stash[INSTALLATION_KEY] = installation.from_env()
132+
except installation.ManifestError as e:
133+
raise pytest.UsageError(str(e)) from e
134+
return config.stash[INSTALLATION_KEY]
135+
136+
120137
def arm_specs_from_options(config: pytest.Config) -> tuple[str, ...] | None:
121-
"""The run's ordered arm specs, reference first, or None for the defaults."""
138+
"""The run's ordered arm specs, reference first, or None for the defaults.
139+
140+
Explicit options win. Without them, a prepared run measures exactly what
141+
preparation installed, in the order it was requested -- so a local run and
142+
a CI run that prepared the same request measure the same arms without
143+
anyone restating them. A manifest that cannot be measured (failed,
144+
partial, or neutral) yields no specs here; :func:`preparation_problem`
145+
says why, per SDK.
146+
"""
122147
refs = cast(str | None, config.getoption("--bench-refs"))
123148
baseline = cast(str | None, config.getoption("--bench-baseline"))
124149
candidate = cast(str | None, config.getoption("--bench-candidate"))
@@ -132,9 +157,44 @@ def arm_specs_from_options(config: pytest.Config) -> tuple[str, ...] | None:
132157
raise pytest.UsageError(f"invalid benchmark options: {e}") from e
133158
if arms:
134159
return tuple(str(a) for a in arms)
160+
prepared = installation_for(config)
161+
if prepared is not None and prepared.usable:
162+
return prepared.arm_specs()
135163
return None
136164

137165

166+
def preparation_problem(config: pytest.Config, sdk: str) -> str | None:
167+
"""Why this SDK's prepared arms cannot be measured, or None.
168+
169+
Only consulted when no arms were named explicitly: an explicit request is
170+
a statement that the manifest is not what this run should measure.
171+
"""
172+
if any(
173+
config.getoption(o) is not None
174+
for o in ("--bench-refs", "--bench-baseline", "--bench-candidate")
175+
):
176+
return None
177+
prepared = installation_for(config)
178+
if prepared is None or prepared.sdk != sdk or prepared.usable:
179+
return None
180+
if prepared.neutral:
181+
return f"neutral preparation: {prepared.status_message}"
182+
return (
183+
f"preparation {prepared.status or 'incomplete'}: "
184+
f"{prepared.status_message or 'fewer than two arms were installed'}"
185+
)
186+
187+
188+
def preparation_is_neutral(config: pytest.Config) -> bool:
189+
"""True if this run's only prepared arms collapsed onto one commit."""
190+
prepared = installation_for(config)
191+
return (
192+
prepared is not None
193+
and prepared.neutral
194+
and arm_specs_from_options(config) is None
195+
)
196+
197+
138198
def arm_count(config: pytest.Config) -> int:
139199
"""How many arms this run will measure per cell."""
140200
specs = arm_specs_from_options(config)
@@ -451,15 +511,20 @@ def __init__(
451511
*,
452512
baseline_spec: str | None = None,
453513
candidate_spec: str | None = None,
514+
problem: dict[str, str | None] | None = None,
454515
) -> None:
455516
self._specs = tuple(specs) if specs else None
456517
self._baseline_spec = baseline_spec
457518
self._candidate_spec = candidate_spec
519+
self._problem = problem or {}
458520
self._cache: dict[str, BenchArms] = {}
459521

460522
def __call__(self, sdk: str) -> BenchArms:
461523
cached = self._cache.get(sdk)
462524
if cached is None:
525+
problem = self._problem.get(sdk)
526+
if problem:
527+
raise ArmSelectionError(problem)
463528
cached = self._cache[sdk] = BenchArms(
464529
select_arms(
465530
sdk,
@@ -476,10 +541,12 @@ def bench_arms(request: pytest.FixtureRequest) -> ArmResolver:
476541
"""Resolver for the arms of any SDK in the run, reference first."""
477542
config = request.config
478543
specs = arm_specs_from_options(config)
544+
cells = _selected_cells(config)
479545
return ArmResolver(
480546
specs,
481547
baseline_spec=None if specs else config.getoption("--bench-baseline"),
482548
candidate_spec=None if specs else config.getoption("--bench-candidate"),
549+
problem={c.sdk: preparation_problem(config, c.sdk) for c in cells},
483550
)
484551

485552

@@ -688,6 +755,24 @@ def runner_metadata(config: pytest.Config) -> dict[str, object]:
688755
"github_run_id": os.environ.get("GITHUB_RUN_ID", ""),
689756
"platform_version": _platform_version(),
690757
"seed": config.getoption("--bench-seed"),
758+
**_installation_metadata(config),
759+
}
760+
761+
762+
def _installation_metadata(config: pytest.Config) -> dict[str, object]:
763+
"""Provenance from the installation manifest, when the run was prepared.
764+
765+
Carried into the artifact so a result names the commits it measured
766+
rather than only their dist tags, which a moving branch reuses.
767+
``preparation`` is the explicit status a reader checks first: ``neutral``
768+
means every requested alias named one commit and nothing was compared.
769+
"""
770+
prepared = installation_for(config)
771+
if prepared is None:
772+
return {}
773+
return {
774+
"installation": prepared.to_dict(),
775+
"preparation": "neutral" if preparation_is_neutral(config) else prepared.status,
691776
}
692777

693778

‎xtest/perf/README.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,10 @@ gate reads, and `p_value_faster` / `p_adjusted_faster` are the matching
7373
Benjamini–Hochberg values within that key's correction family and are `null`
7474
for controls and censored keys, which enter no family.
7575

76+
When the run was prepared with `otdf-sdk-mgr install benchmark`,
77+
`metadata.installation` records each arm's requested alias, commit and dist
78+
tag, and `metadata.preparation` the preparation status.
79+
7680
### The table
7781

7882
```markdown
@@ -265,8 +269,9 @@ things; the noise floor will tell you whether you succeeded.
265269
| `stats.py` | Pure functions: log-ratios, bootstrap CI, Wilcoxon, BH, the decision rule |
266270
| `report.py` | Session recorder, JSON artifact (`SCHEMA_VERSION`), step-summary markdown |
267271
| `config.py` | Option parsing and validation shared by pytest and CI: `BenchConfig`, `--bench-refs`, the arm ceiling, the budget default. No pytest |
272+
| `installation.py` | Reads the installation manifest `otdf-sdk-mgr install benchmark` writes. No pytest |
268273
| `docs/` | The figures in this README, and the Monte Carlo behind them. Documentation only; imported by nothing in the harness |
269-
| `../fixtures/bench.py` | The pytest glue: arm selection (options or defaults), payloads, ciphertexts, budget |
274+
| `../fixtures/bench.py` | The pytest glue: arm selection (options, manifest, defaults), payloads, ciphertexts, budget |
270275
| `../test_benchmarks.py` | One test per cell. **Records; never asserts** |
271276
| `../conftest.py` | `--bench*` options, cell parametrization, the session-finish gate |
272277

0 commit comments

Comments
 (0)