Repository navigation
Commit a93bf1e
committed
feat(xtest): read benchmark arms and provenance from the installation manifest
When $BENCH_INSTALLATION_MANIFEST names a manifest (`otdf-local env`
exports it after `otdf-sdk-mgr install benchmark`), pytest --bench
measures exactly the arms preparation installed, in request order, without
anyone restating them. Explicit --bench-refs or --bench-baseline/
--bench-candidate still win.
- perf/installation.py reads the manifest contract (schema v1alpha1)
without importing otdf-sdk-mgr: the arms, the pins, the status and
metadata.sdk. It checks the schema version and refuses a manifest it
cannot fully read. An unknown commit stays None rather than "None".
- Set but unreadable is a usage error at configure time, not a quiet
fallback to whatever is in dist/.
- A neutral preparation skips that SDK's cells with its reason; the run
reports NEUTRAL instead of NOTHING MEASURED and does not fail. A failed
or partial preparation also skips, and the run fails as having measured
nothing.
- The JSON artifact carries metadata.installation (each arm's alias,
commit and dist tag) and metadata.preparation.
- test_sdk_performance requests the payload, ciphertext, attribute and
scratch fixtures only once a cell is known to be measurable, so a
neutral preparation needs no running platform.
Part 3c of the DSPX-4372-t1 plan; completes stage 2's item 7.
Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>1 parent 656c11e commit a93bf1e
11 files changed
Lines changed: 617 additions & 23 deletions
File tree
- .github/workflows
- xtest
- fixtures
- perf
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | | - | |
| 55 | + | |
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
257 | 257 | | |
258 | 258 | | |
259 | 259 | | |
260 | | - | |
261 | | - | |
262 | | - | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
263 | 263 | | |
264 | 264 | | |
265 | 265 | | |
| |||
456 | 456 | | |
457 | 457 | | |
458 | 458 | | |
459 | | - | |
460 | | - | |
| 459 | + | |
| 460 | + | |
461 | 461 | | |
462 | 462 | | |
463 | 463 | | |
| |||
485 | 485 | | |
486 | 486 | | |
487 | 487 | | |
| 488 | + | |
488 | 489 | | |
489 | 490 | | |
490 | 491 | | |
| |||
495 | 496 | | |
496 | 497 | | |
497 | 498 | | |
| 499 | + | |
| 500 | + | |
498 | 501 | | |
499 | 502 | | |
500 | 503 | | |
| |||
626 | 629 | | |
627 | 630 | | |
628 | 631 | | |
629 | | - | |
| 632 | + | |
630 | 633 | | |
631 | 634 | | |
632 | 635 | | |
| |||
636 | 639 | | |
637 | 640 | | |
638 | 641 | | |
639 | | - | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
640 | 650 | | |
641 | 651 | | |
642 | 652 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | | - | |
| 6 | + | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
| 25 | + | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
30 | 33 | | |
31 | 34 | | |
32 | 35 | | |
| |||
117 | 120 | | |
118 | 121 | | |
119 | 122 | | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
120 | 137 | | |
121 | | - | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
122 | 147 | | |
123 | 148 | | |
124 | 149 | | |
| |||
132 | 157 | | |
133 | 158 | | |
134 | 159 | | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
135 | 163 | | |
136 | 164 | | |
137 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
138 | 198 | | |
139 | 199 | | |
140 | 200 | | |
| |||
451 | 511 | | |
452 | 512 | | |
453 | 513 | | |
| 514 | + | |
454 | 515 | | |
455 | 516 | | |
456 | 517 | | |
457 | 518 | | |
| 519 | + | |
458 | 520 | | |
459 | 521 | | |
460 | 522 | | |
461 | 523 | | |
462 | 524 | | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
463 | 528 | | |
464 | 529 | | |
465 | 530 | | |
| |||
476 | 541 | | |
477 | 542 | | |
478 | 543 | | |
| 544 | + | |
479 | 545 | | |
480 | 546 | | |
481 | 547 | | |
482 | 548 | | |
| 549 | + | |
483 | 550 | | |
484 | 551 | | |
485 | 552 | | |
| |||
688 | 755 | | |
689 | 756 | | |
690 | 757 | | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
691 | 776 | | |
692 | 777 | | |
693 | 778 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
76 | 80 | | |
77 | 81 | | |
78 | 82 | | |
| |||
265 | 269 | | |
266 | 270 | | |
267 | 271 | | |
| 272 | + | |
268 | 273 | | |
269 | | - | |
| 274 | + | |
270 | 275 | | |
271 | 276 | | |
272 | 277 | | |
| |||
0 commit comments