Bump the spec pin to v0.112.0 and absorb 0114-0118 - #270
Merged
Conversation
Advances the submodule, the three tracked version strings, and the smoke-test literal, and regenerates the bundled AGENTS.md. Records 0114 / 0116 / 0117 / 0118 implemented and 0115 partial: its adapter-capability half ships, its provider-faithful Langfuse fake and langfuse_client construction directive do not. v0.112.0 narrows what a failed observation may carry under the default posture, which contradicted five shipped fixture cases; the spec reconciles them in the same release. Fixtures 098 / 137 / 138 are un-deferred and now run against the post-0118 shape, and 159 is wired, which needed a plain input comparator on a Langfuse observation. 157 and 158 stay deferred on the missing fake, and 150 / 151 stay deferred on the separate mock-raises wiring. The leak assertions 157 and 158 carry are declared in the expectations model so those fixtures parse, since the expected block forbids extras. Declaring a field to make a fixture parse is how an assertion becomes dead, so a guard rejects any activated case that reaches one, and the guard is tested against every key and against the fixtures themselves rather than against its own list.
There was a problem hiding this comment.
Pull request overview
Advances the pinned OpenArmature spec version to v0.112.0 and updates the conformance harness to match the post-0118 Langfuse isolation and payload-gating semantics, including wiring fixture 159 and activating previously deferred reconciled fixtures.
Changes:
- Bump the spec pin/version across runtime, pyproject, conformance manifest, and smoke tests (0.107.0 → 0.112.0).
- Extend the Langfuse conformance runner and expectations model to (a) support a plain
inputassertion and (b) reject activated fixtures that reach declared-but-unimplemented leak assertions. - Update observability fixture activation/deferrals and add a dedicated test module validating the unimplemented-assertion guard behavior.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_smoke.py | Updates the asserted __spec_version__ and keeps pin sync checks aligned. |
| tests/conformance/test_unimplemented_assertion_guard.py | Adds tests to ensure declared-but-unimplemented leak assertions fail loudly when activated. |
| tests/conformance/test_observability.py | Activates reconciled fixtures (098/137/138), wires 159, and narrows deferral reasons for remaining deferred fixtures. |
| tests/conformance/test_observability_langfuse.py | Adds the unimplemented-assertion guard, adds input support in observation comparator, and updates fixture activation set. |
| tests/conformance/harness/expectations.py | Declares leak-assertion keys so deferred fixtures parse while remaining guarded when activated. |
| src/openarmature/AGENTS.md | Regenerated bundled agent guide reflecting the new spec pin. |
| src/openarmature/init.py | Updates __spec_version__ constant to 0.112.0. |
| pyproject.toml | Updates [tool.openarmature].spec_version to 0.112.0. |
| conformance.toml | Advances [manifest].spec_pin and records proposal 0114–0118 statuses/notes. |
| CHANGELOG.md | Updates v0.17.0 notes to reflect the spec pin advance and fixture activation state. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The guard ran only in the multi-case loop, so the single-case path and the hand-built 134 branch reached their assertions unchecked. Three of the twenty activated Langfuse fixtures take the single-case path, so this was live rather than latent. It now lives in _run_case, which both the loop and the single-case path funnel through, leaving 134 as the only dispatch path that has to remember, because it bypasses _run_case entirely. The tests parametrize over all three dispatch paths. Covering one path is how the gap shipped: the tests exercised the path that was already wired, so they confirmed the guard worked where it worked.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Suppressed comments (3)
conformance.toml:35
- The version bump makes the capability-table comment below stale: the manifest is now pinned to v0.112.0, and fixture 158 at this pin does carry
requires_capability. Please update the comment so it no longer says the table is declared ahead of the fixtures that consume it.
spec_pin = "v0.112.0"
conformance.toml:1113
- This manifest note says four leak assertions are declared, but
ObservabilityExpectedand_UNIMPLEMENTED_OBSERVABILITY_ASSERTIONSdeclare five. The count should match the published conformance record.
note = "The adapter-capability half is implemented: conformance.toml declares langfuse_bound_provider_detection and the harness gates a case on requires_capability, treating a mismatched arm as a recognized skip and an undeclared capability name as an error. The §6.4 provider-faithful Langfuse fake (records observation content AND emits it as OTel spans through its bound TracerProvider, so a leak to a shared provider is catchable) and the langfuse_client construction directive are NOT yet built, so fixtures 157 / 158 stay deferred. The four leak assertions are declared in the expectations model so those fixtures parse at this pin, and the Langfuse runner fails loudly if an activated fixture reaches for one, so a declared-but-unimplemented assertion cannot read as coverage."
tests/conformance/test_observability.py:267
- Activating fixtures 098, 137, and 138 leaves the
_DEFERRED_FIXTURESpreamble at lines 337-351 factually stale: it still says the pin is v0.107.0, describes these three fixtures as deferred, and says the pin bump remains a prerequisite. Remove that obsolete block now that these entries have been removed.
"098-langfuse-tool-observation",
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advances the pinned spec from v0.107.0 to v0.112.0, absorbing proposals 0114 through 0118: the Langfuse client-ownership and provider-isolation arc, opened by a registered consumer's leak report and settled across five spec releases. The source behavior shipped ahead of the pin in #266, #267 and #268; this brings the conformance side up to it.
Five places hold the version, not the three
AGENTS.mddocuments:pyproject.toml,src/openarmature/__init__.py,conformance.toml, the submodule, and a literal intests/test_smoke.py. That last one is deliberate rather than drift, since it is the guard the other three are checked against. The bundledAGENTS.mdis regenerated.conformance.tomlrecords 0114 / 0116 / 0117 / 0118 asimplemented. 0115 ispartial, and deliberately so: its adapter-capability half shipped in #269, but the provider-faithful Langfuse fake and thelangfuse_clientconstruction directive are not built. Marking itimplementedwould overstate what the published record can back.Fixture reconciliation
Spec v0.112.0 is flagged breaking for existing fixtures under the pre-1.0 allowance, because narrowing what a failed observation may carry under the default posture contradicted five shipped cases. The spec reconciles them in the same release, and this pin adopts that:
metadata_absentdirective from Fail on Langfuse directives the harness does not implement #267 and the error-message gating from Gate the harvested error message on the payload flag #268. Breaking the gate turns all three red, so the assertions are live rather than merely present.inputequality, which 159 needs to pininput: nullon a suppressed observation.metadata_absent: [error_message]assertion at this pin and passed immediately, which is the adopt-ahead pattern doing its job.Declaring assertions without implementing them
Fixtures 157 and 158 carry leak assertions no comparator implements, and the
expectedblock forbids extras, so those keys have to be declared for the fixtures to parse at all.That is precisely how an assertion becomes dead: the key validates, nothing reads it, and the fixture reads like coverage. So a guard rejects any activated case that reaches one, and the guard itself is tested three ways, because a guard nothing exercises is the same defect one level up. Both fixtures are deferred, so nothing in the corpus reaches it today.
The coverage check for that guard is anchored to the fixtures rather than to its own list. A first attempt parametrized over the set under test, which meant dropping a key from the set silently shrank the run instead of failing it: a test that derives its expectations from the thing under test cannot notice that thing shrinking. It now reads the leak assertions out of 157 and 158 and asserts each is guarded, so a sixth assertion arriving at a future pin fails until it is either implemented or guarded.
Testing
2064 passed, 504 skipped. Ruff, pyright, and
check_conformance_manifest.py(118 proposals, 118 entries) all clean.Every guard added here was mutation tested: broken deliberately, confirmed red, restored. That covers the error-message gate against the three un-deferred fixtures, the
inputcomparator against 159, and the unimplemented-assertion guard against all five of its keys.Follow-up
About ten bullets in the 0.17.0 changelog section still read "beyond the current v0.88.0 pin ... the fixtures ride the pin bump". Those are now wrong twice over: the pin has moved, and several of those fixtures run. This adds the missing pin-advance bullet for the cycle and reconciles the isolation entry, but the rest is a per-proposal verification sweep across proposals this PR does not touch, and pattern-editing it would swap one set of false statements for another. Tracked as a release blocker, since a stale changelog is what gets baked into release notes at tag time.