Add the conformance requires_capability audience gate

[chris-colinsky]

Adds the requires_capability audience gate from conformance-adapter §5.5 (proposal 0116), the first of the harness prerequisites for observability fixtures 157/158/159.

Some contracts have arms only one kind of adapter can take. 0116's raise arm needs the adapter to be able to establish which TracerProvider a Langfuse client is bound to, which is not portably guaranteed, so a fixture case declares its audience with requires_capability and each adapter declares what it can do. A case selecting the other arm is a recognized skip rather than a failure.

conformance.toml gains an [adapter_capabilities] table declaring langfuse_bound_provider_detection = true. The declaration lives there rather than in the harness so the published conformance record and the test gate are the same statement, read once. The section it cites arrives with the pin bump, so the table is declared ahead of the fixtures that read it rather than orphaned.

Not passing green while asserting nothing

Both ways of excluding a case, a capability gate and the existing per-case deferral, are deliberately quiet. That is what makes an empty run invisible: a fixture whose cases were all excluded reports success having checked nothing, and the output is identical to one that asserted everything. So the exclusion paths carry explicit guards.

• The vacuity check counts executions rather than inferring them from the exclusion map. Deriving the count from total - len(excluded) would miss cases that share a name, since those collapse into one map entry and make an empty run look partial.
• The check spans both exclusion channels. A fixture with every case deferred is as empty as one entirely gated out, and belongs in the fixture-level deferral table where it skips visibly.
• Each recognized skip is warned, so a passing run still shows which cases did not run.
• A case naming a capability the manifest does not declare is an error, not a skip. §5.5 permits treating an undeclared capability as absent, but silence reads the same whether the capability is genuinely missing or the manifest has not caught up with a new name, and in the second case every case gating on it switches itself off while still reporting green.
• Values are required to be real booleans rather than coerced, since bool("false") is True and a quoted value would select the capable arm while the published record renders false.

The gate is wired into all three of the Langfuse runner's case paths, including the hand-built 134 runner that returns before the generic loop. The OTel runner implements no gate, so it rejects one loudly rather than ignoring it and asserting an arm meant for a different adapter class. A repo-wide sweep covers the runners that have neither, since requires_capability is a general directive and CaseSpec allows extra keys, so a gate landing in an llm-provider or checkpointing fixture would parse cleanly and then be dropped on the floor.

Testing

No fixture at the current v0.107.0 pin carries requires_capability, so the wiring cannot be demonstrated by running the corpus. Rather than leave it unexercised until the pin bump, the tests inject a gate into a real activated fixture and assert on the cases that actually ran. The repo-wide sweep is likewise searching an empty haystack today, so its detector is pinned separately against a synthetic tree.

Every guard here was mutation tested: each was broken deliberately and confirmed to go red. One fix did not survive that on the first pass. The duplicate-name defense was tested only against the helper, so the runner's own arithmetic went unchecked until a test drove a fixture whose cases share a name.

The manifest test asserts the declaration against the Langfuse SDK actually installed rather than against its own literal, which would only prove the file parses. An upgrade that renames the internal the adapter probes would otherwise demote every client to undetectable with nothing going red.

2042 passed, 504 skipped.

Follow-ups

The provider-faithful Langfuse fake was pulled from this PR and will land with the directives that consume it, so each of its fidelity claims is exercised by an assertion as it is written. An adversarial review of the combined change found nine defects in it, none of which the test suite could have caught, because nothing imported it.

A fail-loud allowlist for case-level and expected keys in the Langfuse runner is deferred to the PR that adds the 157/158 directives. _OBSERVATION_DIRECTIVES guards observation-level keys precisely because an expected block is a chain of if key in expected checks, so an unimplemented key is silently skipped. There is no such guard one level up, and it needs to exist before those fixtures are activated.

[Copilot]

Pull request overview

Adds the conformance-adapter §5.5 requires_capability audience gate to the Python conformance harness so fixture cases can select the correct contract arm per adapter capability, while preventing “green but asserted nothing” vacuous passes via explicit non-vacuity guards and recognized-skip reporting. This sets up upcoming observability fixtures (157/158/159) that require adapter-specific capabilities (notably Langfuse bound TracerProvider detection).

Changes:

• Introduces a single-sourced adapter capability manifest ([adapter_capabilities] in conformance.toml) and a shared gating helper (tests/conformance/harness/capabilities.py).
• Wires the gate into the Langfuse observability runner (including the hand-built 134 path) and adds a fail-loud rejection for gates in the OTel runner.
• Adds unit + conformance wiring tests that inject requires_capability into existing activated fixtures to exercise the behavior before the spec pin bump.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/test_conformance_capability_gate.py	Unit tests for capability declaration enforcement, gate semantics, and non-vacuity guard behavior.
tests/conformance/harness/capabilities.py	New shared capability manifest reader, gate evaluator, recognized-skip reporter, and non-vacuity assertion helper.
tests/conformance/test_observability_langfuse.py	Applies the gate to Langfuse fixture execution paths and enforces non-vacuity after in-loop exclusions.
tests/conformance/test_observability.py	Rejects requires_capability in the OTel runner until that runner implements per-case gating.
tests/conformance/test_capability_gate_wiring.py	End-to-end wiring tests that monkeypatch fixture loads to inject gates and assert runners consult them correctly.
conformance.toml	Declares adapter capabilities (starting with langfuse_bound_provider_detection = true) for the harness to consult.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Suppressed comments (5)

tests/conformance/test_capability_gate_wiring.py:260

• This truthiness check recreates the malformed-falsy hole that capability_skip_reason() guards against. For example, requires_capability: [], false, or "" is silently omitted from the sweep, so a runner outside observability can ignore the directive without this guard failing. Treat non-null, non-mapping values as detected invalid gates while preserving an empty mapping as ungated.

        if any(isinstance(c, dict) and cast("dict[str, Any]", c).get("requires_capability") for c in cases):

tests/conformance/test_observability.py:654

• This runner also silently accepts falsy malformed gates. A case with requires_capability: [] reaches the OTel assertions even though the Langfuse path rejects the same shape, potentially exercising the wrong contract arm. Validate the value as a mapping before deciding whether it is empty.

    gated = [cast("str", c.get("name") or "<unnamed>") for c in cases if c.get("requires_capability")]

tests/conformance/test_observability_langfuse.py:555

• The wiring suite exercises the generic multi-case loop and the special 134 loop, but never injects a gate into this single-case branch. Since the pinned fixtures carry no gates, deleting this call would not fail any test. Add a single-case runner test for both matching and mismatched requirements.

        gate = capability_skip_reason(spec.get("requires_capability"))

tests/conformance/harness/capabilities.py:57

• The non-boolean manifest guard has no negative-path coverage: the current manifest contains a real boolean, and the malformed-value test only exercises fixture requirements. A quoted "false" declaration could stop being rejected without any test failing. Add a test using a temporary manifest and clear adapter_capabilities' cache around it.

        if not isinstance(value, bool):

tests/conformance/test_observability.py:682

• The new wiring call is not covered end to end: the added OTel test invokes _reject_unsupported_capability_gate() directly. Removing this line would therefore leave all tests green at the current ungated spec pin. Add a runner-level test that injects a gate into a supported OTel fixture and verifies the entry point rejects it.

    _reject_unsupported_capability_gate(fixture_id, spec)