Gate the harvested error message on the payload flag

[chris-colinsky]

Summary

A failed observation's error_message is harvested exception text, so disable_provider_payload now governs it for every failure category across all four provider observations (Generation, Embedding, Tool, Retriever). It previously covered only structured_output_invalid, on the reasoning that other categories describe the call rather than the response. That does not hold: a provider 4xx routinely quotes the request or the flagged prompt, and the category does not tell you what the string contains.

error_type moves out of the gate. It is a classification token rather than harvested content, and it is the only discriminator a tool failure has, since a tool failure carries no error category. The category still rides as the status message where the event supplies one, and a tool failure's status message stays null rather than falling back to the message it just withheld.

The separate per-emission isolation check is retired. Once the flag covers the field the existing arms decide every configuration, and the replacement predicate still consults isolation, so the protection is inherited rather than dropped.

Coverage

Only the tool path had an end-to-end test of the withheld message. Reverting any of the other three handlers to the old predicate passed the entire suite, so three of four sites had nothing guarding the property this change exists to establish. This adds default-posture tests for the LLM, embedding and rerank paths, plus a canary case with payloads disabled on an isolated client, which is the one configuration that separates the two predicates. Each of the three reverts now fails.

Deferred fixtures

Three fixtures (098, 137, 138) assert the pre-0118 shape at the current pin and are deferred until it advances. Their reasons name both the pin bump and the metadata_absent directive, since re-listing them while that directive is unimplemented restores a vacuous pass rather than coverage. The tool runner also now reads the per-observer langfuse_observer block that the embedding and rerank runners already honour, which fixture 098 needs at the bump.

Testing

2015 passed, 504 skipped. ruff, pyright and the docs build are clean. Verified against a scratch checkout of the newer spec with the absence assertion live: the reconciled fixtures pass on the behaviour rather than on the assertion being skipped.

Raised, not decided

Whether error_message should be subject to the payload_byte_cap truncation every other payload-classified field uses. It is written verbatim today; fixtures exist to assert the message literally, which truncation would contradict, so this goes to the batched spec review rather than changing rendering unilaterally. Noted in a comment at the predicate.

[Copilot]

Pull request overview

This pull request tightens Langfuse observability privacy by ensuring a failed observation’s harvested error_message is treated as provider payload and therefore consistently governed by disable_provider_payload across Generation, Embedding, Tool, and Retriever observations. It also clarifies the remaining failure signal surface by ungating error_type and expanding test coverage to prevent regressions across the four handlers and the isolation edge cases.

Changes:

• Gate harvested error_message emission on disable_provider_payload for all provider observation failure paths, while keeping error_type ungated as a classification token.
• Update isolation-policy logging and retire the separate per-emission isolation predicate in favor of the existing payload and isolation arms.
• Add unit and canary tests to cover default-posture behavior and the isolated-client plus payloads-off configuration, and defer conformance fixtures that assert the pre-0118 shape at the current spec pin.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/openarmature/observability/langfuse/observer.py	Implements unified gating for harvested error_message and updates isolation-policy messaging and failure mappings.
tests/unit/test_langfuse_provider_isolation.py	Adds default-posture and isolation-status coverage for harvested error message emission across provider observation types.
tests/unit/test_langfuse_payload_leak_canary.py	Adds a canary ensuring payloads-off suppresses all harvested channels even when isolation is established.
tests/conformance/test_observability.py	Defers fixtures that assert pre-0118 shapes at the current pin and adds per-observer Langfuse config override for tool cases.
docs/concepts/observability.md	Documents that disable_provider_payload governs failed observation error_message and what remains visible for triage.
docs/agent/non-obvious-shapes.md	Updates the non-obvious shapes guidance to reflect the new gating semantics.
src/openarmature/AGENTS.md	Updates bundled agent docs to reflect the new gating semantics.
CHANGELOG.md	Updates the Langfuse observer entry to include proposal 0118 behavior and revised error-field semantics.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/openarmature/observability/langfuse/observer.py:504

• This log is emitted for an active observer whose observations still go to the shared provider; only harvested payload fields are disabled. Saying "nothing is being exported" can mislead operators into thinking no telemetry reaches that backend. Narrow this to "no harvested payload" while retaining the fail-closed explanation.

                "no payload channel is enabled, so nothing is being exported to it; "

tests/unit/test_langfuse_provider_isolation.py:529

• This default-posture test covers the new message gate but leaves error_type unset, so it does not protect the other half of the change: LLM error_type must remain present while payloads are disabled. The tool test checks this, but reverting only the LLM handler to gate both fields would still pass. Supply a non-null type and assert it remains in metadata.

    assert "error_message" not in obs.metadata

tests/unit/test_langfuse_provider_isolation.py:564

• This test verifies withholding error_message, but it does not verify that embedding error_type moved outside the gate. Since the reconciled conformance fixture is deferred, gating both fields again in only this handler currently has no regression test. Add a non-null type and assert it is retained under the default-off posture.

    assert "error_message" not in obs.metadata

tests/unit/test_langfuse_provider_isolation.py:602

• The rerank test covers the new message gate but not the newly ungated error_type. With fixture 138 deferred, a future regression that puts both fields back behind the payload predicate in this handler would pass the suite. Set error_type and assert it remains emitted while error_message is absent.

    assert "error_message" not in obs.metadata