Skip to content

Bump the spec pin to v0.112.0 and absorb 0114-0118 - #270

Merged
chris-colinsky merged 2 commits into
mainfrom
chore/spec-pin-v0112
Aug 13, 2026
Merged

Bump the spec pin to v0.112.0 and absorb 0114-0118#270
chris-colinsky merged 2 commits into
mainfrom
chore/spec-pin-v0112

Conversation

@chris-colinsky

Copy link
Copy Markdown
Member

Advances the pinned spec from v0.107.0 to v0.112.0, absorbing proposals 0114 through 0118: the Langfuse client-ownership and provider-isolation arc, opened by a registered consumer's leak report and settled across five spec releases. The source behavior shipped ahead of the pin in #266, #267 and #268; this brings the conformance side up to it.

Five places hold the version, not the three AGENTS.md documents: pyproject.toml, src/openarmature/__init__.py, conformance.toml, the submodule, and a literal in tests/test_smoke.py. That last one is deliberate rather than drift, since it is the guard the other three are checked against. The bundled AGENTS.md is regenerated.

conformance.toml records 0114 / 0116 / 0117 / 0118 as implemented. 0115 is partial, and deliberately so: its adapter-capability half shipped in #269, but the provider-faithful Langfuse fake and the langfuse_client construction directive are not built. Marking it implemented would overstate what the published record can back.

Fixture reconciliation

Spec v0.112.0 is flagged breaking for existing fixtures under the pre-1.0 allowance, because narrowing what a failed observation may carry under the default posture contradicted five shipped cases. The spec reconciles them in the same release, and this pin adopts that:

  • 098 / 137 / 138 are un-deferred and now run against the post-0118 shape. They rest on the metadata_absent directive from Fail on Langfuse directives the harness does not implement #267 and the error-message gating from Gate the harvested error message on the payload flag #268. Breaking the gate turns all three red, so the assertions are live rather than merely present.
  • 159 is wired, which is the one piece of genuinely new coverage here. It needs no isolation machinery, only the existing per-observer payload flag, so it runs today instead of waiting on the fake. Activating it surfaced a gap the known-key guard caught loudly: the Langfuse comparator had two shaped input directives but no plain input equality, which 159 needs to pin input: null on a suppressed observation.
  • 157 / 158 stay deferred on the missing fake and construction directive. 150 / 151 stay deferred on the separate proposal 0107 mock-raises wiring; their 0118 half resolved here, so their deferral reasons are narrowed to what actually remains.
  • 123 needed no work at all. It gained a metadata_absent: [error_message] assertion at this pin and passed immediately, which is the adopt-ahead pattern doing its job.

Declaring assertions without implementing them

Fixtures 157 and 158 carry leak assertions no comparator implements, and the expected block forbids extras, so those keys have to be declared for the fixtures to parse at all.

That is precisely how an assertion becomes dead: the key validates, nothing reads it, and the fixture reads like coverage. So a guard rejects any activated case that reaches one, and the guard itself is tested three ways, because a guard nothing exercises is the same defect one level up. Both fixtures are deferred, so nothing in the corpus reaches it today.

The coverage check for that guard is anchored to the fixtures rather than to its own list. A first attempt parametrized over the set under test, which meant dropping a key from the set silently shrank the run instead of failing it: a test that derives its expectations from the thing under test cannot notice that thing shrinking. It now reads the leak assertions out of 157 and 158 and asserts each is guarded, so a sixth assertion arriving at a future pin fails until it is either implemented or guarded.

Testing

2064 passed, 504 skipped. Ruff, pyright, and check_conformance_manifest.py (118 proposals, 118 entries) all clean.

Every guard added here was mutation tested: broken deliberately, confirmed red, restored. That covers the error-message gate against the three un-deferred fixtures, the input comparator against 159, and the unimplemented-assertion guard against all five of its keys.

Follow-up

About ten bullets in the 0.17.0 changelog section still read "beyond the current v0.88.0 pin ... the fixtures ride the pin bump". Those are now wrong twice over: the pin has moved, and several of those fixtures run. This adds the missing pin-advance bullet for the cycle and reconciles the isolation entry, but the rest is a per-proposal verification sweep across proposals this PR does not touch, and pattern-editing it would swap one set of false statements for another. Tracked as a release blocker, since a stale changelog is what gets baked into release notes at tag time.

Advances the submodule, the three tracked version strings, and the
smoke-test literal, and regenerates the bundled AGENTS.md. Records
0114 / 0116 / 0117 / 0118 implemented and 0115 partial: its
adapter-capability half ships, its provider-faithful Langfuse fake and
langfuse_client construction directive do not.

v0.112.0 narrows what a failed observation may carry under the default
posture, which contradicted five shipped fixture cases; the spec
reconciles them in the same release. Fixtures 098 / 137 / 138 are
un-deferred and now run against the post-0118 shape, and 159 is wired,
which needed a plain input comparator on a Langfuse observation. 157
and 158 stay deferred on the missing fake, and 150 / 151 stay deferred
on the separate mock-raises wiring.

The leak assertions 157 and 158 carry are declared in the expectations
model so those fixtures parse, since the expected block forbids extras.
Declaring a field to make a fixture parse is how an assertion becomes
dead, so a guard rejects any activated case that reaches one, and the
guard is tested against every key and against the fixtures themselves
rather than against its own list.
Copilot AI lite review requested due to automatic review settings August 13, 2026 06:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Advances the pinned OpenArmature spec version to v0.112.0 and updates the conformance harness to match the post-0118 Langfuse isolation and payload-gating semantics, including wiring fixture 159 and activating previously deferred reconciled fixtures.

Changes:

  • Bump the spec pin/version across runtime, pyproject, conformance manifest, and smoke tests (0.107.0 → 0.112.0).
  • Extend the Langfuse conformance runner and expectations model to (a) support a plain input assertion and (b) reject activated fixtures that reach declared-but-unimplemented leak assertions.
  • Update observability fixture activation/deferrals and add a dedicated test module validating the unimplemented-assertion guard behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/test_smoke.py Updates the asserted __spec_version__ and keeps pin sync checks aligned.
tests/conformance/test_unimplemented_assertion_guard.py Adds tests to ensure declared-but-unimplemented leak assertions fail loudly when activated.
tests/conformance/test_observability.py Activates reconciled fixtures (098/137/138), wires 159, and narrows deferral reasons for remaining deferred fixtures.
tests/conformance/test_observability_langfuse.py Adds the unimplemented-assertion guard, adds input support in observation comparator, and updates fixture activation set.
tests/conformance/harness/expectations.py Declares leak-assertion keys so deferred fixtures parse while remaining guarded when activated.
src/openarmature/AGENTS.md Regenerated bundled agent guide reflecting the new spec pin.
src/openarmature/init.py Updates __spec_version__ constant to 0.112.0.
pyproject.toml Updates [tool.openarmature].spec_version to 0.112.0.
conformance.toml Advances [manifest].spec_pin and records proposal 0114–0118 statuses/notes.
CHANGELOG.md Updates v0.17.0 notes to reflect the spec pin advance and fixture activation state.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/conformance/test_observability_langfuse.py Outdated
The guard ran only in the multi-case loop, so the single-case path and
the hand-built 134 branch reached their assertions unchecked. Three of
the twenty activated Langfuse fixtures take the single-case path, so
this was live rather than latent.

It now lives in _run_case, which both the loop and the single-case path
funnel through, leaving 134 as the only dispatch path that has to
remember, because it bypasses _run_case entirely.

The tests parametrize over all three dispatch paths. Covering one path
is how the gap shipped: the tests exercised the path that was already
wired, so they confirmed the guard worked where it worked.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (3)

conformance.toml:35

  • The version bump makes the capability-table comment below stale: the manifest is now pinned to v0.112.0, and fixture 158 at this pin does carry requires_capability. Please update the comment so it no longer says the table is declared ahead of the fixtures that consume it.
spec_pin = "v0.112.0"

conformance.toml:1113

  • This manifest note says four leak assertions are declared, but ObservabilityExpected and _UNIMPLEMENTED_OBSERVABILITY_ASSERTIONS declare five. The count should match the published conformance record.
note = "The adapter-capability half is implemented: conformance.toml declares langfuse_bound_provider_detection and the harness gates a case on requires_capability, treating a mismatched arm as a recognized skip and an undeclared capability name as an error. The §6.4 provider-faithful Langfuse fake (records observation content AND emits it as OTel spans through its bound TracerProvider, so a leak to a shared provider is catchable) and the langfuse_client construction directive are NOT yet built, so fixtures 157 / 158 stay deferred. The four leak assertions are declared in the expectations model so those fixtures parse at this pin, and the Langfuse runner fails loudly if an activated fixture reaches for one, so a declared-but-unimplemented assertion cannot read as coverage."

tests/conformance/test_observability.py:267

  • Activating fixtures 098, 137, and 138 leaves the _DEFERRED_FIXTURES preamble at lines 337-351 factually stale: it still says the pin is v0.107.0, describes these three fixtures as deferred, and says the pin bump remains a prerequisite. Remove that obsolete block now that these entries have been removed.
        "098-langfuse-tool-observation",

@chris-colinsky
chris-colinsky merged commit 18659c1 into main Aug 13, 2026
7 checks passed
@chris-colinsky
chris-colinsky deleted the chore/spec-pin-v0112 branch August 13, 2026 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants