Activate fixture 157 against a real Langfuse client

[chris-colinsky]

Activates observability fixture 157, which gates the two provider-isolation MUSTs from proposal 0114: that openarmature isolates the Langfuse client it builds itself, and that it never mutates one the caller supplies.

Two modes, two doubles

langfuse_client has two modes and they need different things.

mode: supplied hands openarmature an object satisfying its client protocol. That is the new fake: it records observations as the bundled in-memory client does, and also emits each one through its bound TracerProvider, the way a Langfuse v4 client does, so a leak onto a shared provider becomes observable. Its spans carry the SDK's real langfuse.* attributes rather than a harness-invented summary, because a double that stamps its own verdict and an assertion that reads that verdict together test the double's opinion instead of the property.

mode: credentials is not faked, and this is the substantive design decision in the PR. There openarmature constructs the client itself and wraps it in LangfuseSDKAdapter, which drives the SDK's private surface: start_observation, _otel_tracer, _create_remote_parent_span, _get_otel_trace_id, and the real langfuse._client.span classes for back-dated observations. Faking that was tried first. Each internal satisfied uncovered another, and provider observations would not have landed in a recorded side regardless, because the adapter builds real SDK span objects for them.

So that mode uses a real client with only its egress removed. The SDK attaches its own exporting span processor to whatever provider it is given; replacing that one class leaves the client, the adapter path, the provider binding and the per-credential singleton genuinely real, and the fixture's own exporter still sees every span. It is more faithful than the fake would have been, and it makes preexisting_same_key_client real for the follow-up rather than simulated.

Non-vacuity

A real client has no recorded side, so the fixture's expected observation tree is read off the provider openarmature isolated. That tree is what the fixture itself labels its non-vacuity proof, and using it means "no observation leaked" cannot be satisfied by a run that emitted nothing. An earlier draft substituted a weaker check of its own, which the node span alone satisfied.

Both MUSTs are mutation tested. Breaking isolation fails the mode-b case; rebinding a supplied client fails the mode-a case; renaming the Generation or suppressing its output despite the payload flag fails the tree assertion.

A guard on the private SDK surface

The shipped adapter reaches for private symbols inside the declared langfuse>=4.6,<5 range. The only guard was the live-account integration test, which CI deselects, so on a normal run nothing checked them. tests/unit/test_langfuse_sdk_internals.py enumerates them and fails on a rename, which matters because losing one does not raise to the caller: the graph observer isolates observer errors, so an observation simply stops being emitted and a leak assertion reads clean.

uv.lock moves to langfuse 4.7.1, matching the version the compatibility page records as verified.

Directive allowlists

Case-level and expected-level keys now have allowlists, so a key nobody reads fails loudly instead of being ignored while the case still reads like coverage. Two properties are deliberate:

• Per-runner, not file-global. Three runners live in this file with different coverage, and a shared set let one runner's implementation vouch for a key another silently dropped.
• Keys nothing reads are recorded by name with a reason, not left sitting in the implemented set, because a reader takes that set as a contract. Deleting an entry without implementing the key makes the guard fire.

Building them surfaced a live silent skip that predates this PR: langfuse_observer_config was read nowhere, so fixture 059's two cases both ran at the default and its state-payload-enabled case was not exercising the configuration it names. The spelling is now folded onto langfuse_observer at load. Note that this makes the directive take effect; whether 059's assertions distinguish its two cases is a separate fixture-design question and is left alone.

Testing

2094 passed, 504 skipped. Ruff, pyright and check_conformance_manifest.py clean. Conformance and unit suites re-run for order contamination, which is the real risk here: the runner mutates and restores three process-global things per case, and each is snapshotted inside the block that restores it.

conformance.toml is reconciled: three notes still said the §6.4 double was unbuilt and 157 deferred.

Follow-ups

Fixture 158 stays deferred on the payload-bearing classification, expected_construction_error, the level key on expected.log_records, and the preexisting_same_key_client / accept_shared_provider sub-directives. Those sub-directives and the log_records key are recorded as named deferrals rather than quietly allowlisted, so activating 158 has to address them.

The langfuse>=4.6,<5 floor is worth revisiting separately: the code cites v4.7-specific behaviour in several places, and the range currently resolves to 4.14.x.

[Copilot]

Pull request overview

Activates observability conformance fixture 157 by extending the Langfuse conformance harness to exercise provider-isolation behavior against (a) a provider-faithful supplied-client fake and (b) a real Langfuse SDK client with its network egress neutralized, plus adds CI-visible guards for the private Langfuse SDK internals the shipped adapter relies on.

Changes:

• Adds a provider-faithful Langfuse client fake (exports real langfuse.* span attributes via a bound TracerProvider) and a “real SDK, no egress” harness context for credential-mode cases.
• Updates the Langfuse conformance runner to run fixture 157 with stricter directive/expectation allowlists and leak-assertion accounting.
• Bumps langfuse in uv.lock to 4.7.1 and updates conformance proposal notes to reflect fixture 157 now running.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
uv.lock	Bumps Langfuse SDK to 4.7.1.
tests/unit/test_langfuse_sdk_internals.py	Adds unit tests that guard the private Langfuse SDK symbols the shipped adapter depends on.
tests/unit/test_langfuse_provider_fake.py	Drives the new provider-faithful fake against a real TracerProvider + in-memory exporter.
tests/conformance/test_unimplemented_assertion_guard.py	Strengthens the guard to ensure leak-assertion keys are either implemented or explicitly guarded.
tests/conformance/test_observability.py	Moves fixture 157 from deferred to supported for observability conformance runs.
tests/conformance/test_observability_langfuse.py	Adds fixture-157 dedicated runner, directive allowlists, and observer-config normalization.
tests/conformance/harness/langfuse_real_client.py	Introduces a context manager that runs a real Langfuse client with its exporting processor swapped out and cache/env sanitized.
tests/conformance/harness/langfuse_provider_fake.py	Introduces the provider-faithful supplied-client fake and span selection helper.
conformance.toml	Updates proposal notes to reflect fixture 157 activation and remaining 158 deferrals.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.