diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f8809a..6bfa571 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The - **Langfuse parallel-branches mapping parity** (proposal 0088, observability §8.4.8 / §8.3 / §8.4.2 / §3.4, spec v0.83.0). Brings the Langfuse observer's parallel-branches rendering to parity with the OTel side. The observer already synthesized the three-level Observation tree (the parallel-branches node Span, a per-branch dispatch Span named by the `branch_name`, and the branch's inner observations) and already emitted the dispatch-span `parallel_branches_parent_node_name` and `branch_name`; the two node-span attributes `parallel_branches_branch_count` and `parallel_branches_error_policy` are now flattened onto the node Span's `observation.metadata` (mirroring the `fan_out_*` attributes), the one §8.4.2 row the observer had never mapped. The three `parallel_branches_*` keys join the reserved caller-metadata set (26 to 29), so a caller passing one as invocation metadata is rejected at the `invoke()` boundary rather than shadowing the OA-emitted field. The OTel side was already complete. Conformance fixture 136 (the dedicated three-level-tree pin) is un-deferred; fixture 030's incidental coverage stands. - **Adaptive call-level retry: per-attempt request override** (proposal 0095, llm-provider §7.1, spec v0.91.0). The LLM-completion call-level retry loop gains an opt-in per-attempt request override. A new `LlmRetryConfig` (the llm-provider-scoped superset of the generic `RetryConfig`, exported from `openarmature.llm`) carries a `per_attempt_override`: a schedule of `RuntimeConfig` partials applied to retries. Attempt 0 uses the caller's base `config` unchanged; retry `i` merges `per_attempt_override[i]` onto the base (the override's non-None fields replace; a None or unspecified field inherits the base, per the §6 null-skip semantics), and the last entry carries forward when the schedule is shorter than the retry count. The canonical use is an escalating temperature schedule that breaks the "temperature 0 replays the same output" determinism trap on a retried structured-output call. `complete()` never mutates the caller's `config` (each attempt config is a fresh copy), and a plain `RetryConfig` preserves the existing byte-identical replay. The per-attempt OTel span carries a new `openarmature.llm.retry_reason` attribute (`transient`) on retries, absent on the base attempt. This is the first half of proposal 0095; the structured-output reask half follows. Spec v0.91.0 is beyond the current v0.88.0 pin, so the behavior ships ahead of the pin (unit-tested); the conformance fixtures 061-066 ride the v0.17.0 pin bump. - **Adaptive call-level retry: structured-output reask** (proposal 0095, llm-provider §7.1, spec v0.91.0). The second half of 0095. `LlmRetryConfig` gains an opt-in `reask` builder (`Callable[[StructuredOutputInvalid], str]`). When present, a `structured_output_invalid` failure becomes retryable for that call (a call-level convenience, not a classifier change; without a builder it stays non-transient and raises on the first occurrence). On each such failure the loop appends two messages to a working transcript, the model's raw output as an `assistant` message and the builder's returned correction as a `user` message, so the retry is informed rather than a byte-identical replay. OA authors no prompt of its own (the caller owns every word beyond the model's output); the builder receives the raised `StructuredOutputInvalid` (its `raw_content` and `failure_description`). The transcript accumulates reask pairs across reask retries and consumes the `max_attempts` budget; a transient retry interleaved in a reask loop re-sends the accumulated transcript unchanged. `complete()` never mutates the caller's `messages` (each reask replaces the transcript with a fresh list rather than appending in place). The retry span's `openarmature.llm.retry_reason` is `reask` on a reask retry, `transient` otherwise. A reask always appends the model output as a fresh `assistant` message (never continues a trailing one): §3 requires the last message before a call to be `user`/`tool`, so the transcript never ends in `assistant`. Ships ahead of the pin (unit-tested); fixtures 062-066 ride the pin bump. -- **Langfuse observer: credentials-in construction with tracer-provider isolation** (proposals 0114 + 0116 + 0117 + 0118, observability §6 / §8.9 / §8.4, spec v0.108.0 / v0.110.0 / v0.111.0 / v0.112.0). The Langfuse observer gains a second construction mode alongside today's caller-supplied client: `LangfuseObserver.from_credentials(public_key=..., secret_key=..., host=...)` (over the lower-level `LangfuseSDKAdapter.from_credentials(...)`) builds an OA-owned `Langfuse` client on a dedicated `TracerProvider` by default, so its observations no longer bind the global provider and leak onto the application's OTel backend. A Langfuse v4 client constructed with no `tracer_provider=` attaches its span processor to the globally-registered provider, so in any service that registers a global provider (the standard app-tracing setup) attaching the Langfuse observer silently exported every observation, prompts and completions included, to the app backend. Because the Langfuse SDK caches one client per `public_key`, a dedicated provider takes effect only when OA is the first constructor for that credential; OA reuses one isolated provider per credential and reads the actual binding back after construction. The invariant covers every payload OA harvests from the runtime -- the provider payload (`disable_provider_payload`), the Trace-level state input/output (`disable_state_payload` and the `trace_input_from_state` / `trace_output_from_state` hooks), and a failed Tool / Embedding / Retriever / LLM observation's `error_message` -- but not the dimensions the caller deliberately attaches (`correlation_id` / `session_id` / `userId` / trace name / caller metadata), which stay verbatim as cross-backend join keys. When any construction-determinable channel is live and OA establishes the client is bound to a provider it did not isolate, construction fails loud with a categorized `LangfuseProviderIsolationUnavailable` before any observation is emitted, rather than leaking payloads to a shared backend; where OA cannot establish the binding at all (a future SDK), it suppresses every channel and logs a warning. A failed observation's `error_message` is harvested exception text, so `disable_provider_payload` governs it for every failure category on all four provider observations: with payloads off it is not rendered, and the error category still rides as the status message where the event carries one. A Tool failure has no category, so its status message is null rather than falling back to the exception string. `error_type` is a classification token rather than harvested content and is never gated, which matters most for a Tool failure where it is the only remaining discriminator; it is optional, so it is emitted only where the failure event supplies one. A single `accept_shared_provider=True` opt-out turns the whole thing into a warn-and-proceed onto the shared provider. With no channel live (the default privacy posture), an un-isolatable client neither raises nor warns. The existing caller-supplied path (mode a) is unchanged and never mutated: a caller who builds their own client stays responsible for isolating its `tracer_provider`, and OA documents the remedy rather than reaching into the supplied client. The `secret_key` is accepted as a `pydantic.SecretStr`, masked in OA's own reprs and logs with the plaintext read only at the SDK call (`public_key` and `host` stay plain strings), and a blank credential is rejected at the boundary rather than falling through to the SDK's ambient `LANGFUSE_*` environment fallback. A `sample_rate` passed for the client is applied to the isolated provider, since the SDK only honors it on a provider it builds itself. `accept_shared_provider` binds the provider the application already registered rather than letting the SDK construct and globally register one of its own, which would capture OTel's single-assignment global slot. The new `LangfuseProviderIsolationUnavailable` derives from an `ObservabilityError` base, a fourth hierarchy alongside the graph-engine, llm-provider, and checkpoint ones. Spec v0.108.0 through v0.112.0 are beyond the current v0.107.0 pin, so this ships ahead of the pin (unit-tested); the conformance fixtures (157 / 158 / 159, proposals 0115 / 0116 / 0117 / 0118) ride the pin bump, as do fixtures 098 / 137 / 138, which are deferred meanwhile because they still assert the pre-0118 shape. +- **Langfuse observer: credentials-in construction with tracer-provider isolation** (proposals 0114 + 0116 + 0117 + 0118, observability §6 / §8.9 / §8.4, spec v0.108.0 / v0.110.0 / v0.111.0 / v0.112.0). The Langfuse observer gains a second construction mode alongside today's caller-supplied client: `LangfuseObserver.from_credentials(public_key=..., secret_key=..., host=...)` (over the lower-level `LangfuseSDKAdapter.from_credentials(...)`) builds an OA-owned `Langfuse` client on a dedicated `TracerProvider` by default, so its observations no longer bind the global provider and leak onto the application's OTel backend. A Langfuse v4 client constructed with no `tracer_provider=` attaches its span processor to the globally-registered provider, so in any service that registers a global provider (the standard app-tracing setup) attaching the Langfuse observer silently exported every observation, prompts and completions included, to the app backend. Because the Langfuse SDK caches one client per `public_key`, a dedicated provider takes effect only when OA is the first constructor for that credential; OA reuses one isolated provider per credential and reads the actual binding back after construction. The invariant covers every payload OA harvests from the runtime -- the provider payload (`disable_provider_payload`), the Trace-level state input/output (`disable_state_payload` and the `trace_input_from_state` / `trace_output_from_state` hooks), and a failed Tool / Embedding / Retriever / LLM observation's `error_message` -- but not the dimensions the caller deliberately attaches (`correlation_id` / `session_id` / `userId` / trace name / caller metadata), which stay verbatim as cross-backend join keys. When any construction-determinable channel is live and OA establishes the client is bound to a provider it did not isolate, construction fails loud with a categorized `LangfuseProviderIsolationUnavailable` before any observation is emitted, rather than leaking payloads to a shared backend; where OA cannot establish the binding at all (a future SDK), it suppresses every channel and logs a warning. A failed observation's `error_message` is harvested exception text, so `disable_provider_payload` governs it for every failure category on all four provider observations: with payloads off it is not rendered, and the error category still rides as the status message where the event carries one. A Tool failure has no category, so its status message is null rather than falling back to the exception string. `error_type` is a classification token rather than harvested content and is never gated, which matters most for a Tool failure where it is the only remaining discriminator; it is optional, so it is emitted only where the failure event supplies one. A single `accept_shared_provider=True` opt-out turns the whole thing into a warn-and-proceed onto the shared provider. With no channel live (the default privacy posture), an un-isolatable client neither raises nor warns. The existing caller-supplied path (mode a) is unchanged and never mutated: a caller who builds their own client stays responsible for isolating its `tracer_provider`, and OA documents the remedy rather than reaching into the supplied client. The `secret_key` is accepted as a `pydantic.SecretStr`, masked in OA's own reprs and logs with the plaintext read only at the SDK call (`public_key` and `host` stay plain strings), and a blank credential is rejected at the boundary rather than falling through to the SDK's ambient `LANGFUSE_*` environment fallback. A `sample_rate` passed for the client is applied to the isolated provider, since the SDK only honors it on a provider it builds itself. `accept_shared_provider` binds the provider the application already registered rather than letting the SDK construct and globally register one of its own, which would capture OTel's single-assignment global slot. The new `LangfuseProviderIsolationUnavailable` derives from an `ObservabilityError` base, a fourth hierarchy alongside the graph-engine, llm-provider, and checkpoint ones. The behavior shipped ahead of the pin (unit-tested) and the pin has since advanced to v0.112.0. Fixture 159 now runs, and fixtures 098 / 137 / 138 are un-deferred and reconciled to the post-0118 shape. Fixtures 157 / 158 stay deferred: they need the conformance-adapter provider-faithful Langfuse fake and the `langfuse_client` construction directive, which are not yet built, and the source behavior is covered meanwhile by the unit suite. ### Changed @@ -34,6 +34,8 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The - **The OpenAI embedding mapping decodes a base64 `data[].embedding`** (proposal 0106, retrieval-provider §8.3, spec v0.101.0). A caller may set `encoding_format: "base64"` through the extras bag to request the compact wire encoding, so the `/v1/embeddings` response returns each embedding as a base64 string rather than a JSON number array. The mapping now decodes `data[].embedding` by wire shape: a JSON number array is the float vector verbatim (unchanged); a base64 string is decoded as consecutive little-endian IEEE-754 float32 values (the same vector, more compact on the wire); any other shape (null, a number, a bool, an object, an empty value, or an array with a non-number) is a malformed response (`provider_invalid_response`). A base64 value that is not valid base64, or whose decoded byte length is not a positive multiple of 4, is likewise malformed and fails loud rather than returning a truncated or padded vector. The verbatim base64 string is preserved on `raw` (only `vectors` carries the decoded floats), and the decode composes with batch chunking. `encoding_format` stays an ordinary unmanaged extras key, since the consumer keys on the response shape rather than the request parameter, and the wire default remains `"float"`, so existing callers see no change. Previously a base64 request failed `provider_invalid_response` because the consumer could not read a base64 string as a vector, so this makes a previously-broken advertised knob work. Spec v0.101.0 is beyond the current v0.88.0 pin, so this ships ahead of the pin (unit-tested); the base64 round-trip and malformed-boundary fixtures ride the pin bump. - **Token-budget failure-path parity on Langfuse, and unrecognized `token_budget` keys are tolerated** (proposal 0109, observability §8.4.3 + prompt-management §3 / §5, spec v0.104.0). Two additive edges from the 0083 token-budget work. (1) The Langfuse Generation now carries a flat `metadata.token_budget_exceeded` boolean whenever a budget is declared and at least one bound is evaluable: `true` if any evaluated bound was crossed, `false` if all held, absent when no bound is evaluable (a not-reported counter contributes nothing, per 0101). It is emitted regardless of the observation level, so on a `structured_output_invalid` failure that also exceeds budget the flag survives the `ERROR`-precedence rule (the `ERROR` level and category `statusMessage` still win as the primary signal), giving the Langfuse failure path parity with the OTel `openarmature.llm.token_budget.exceeded` span attribute. (2) The filesystem prompt backend now tolerates and filters an unrecognized key in a `token_budget` config: a stray or future key alongside a valid bound is ignored and the recognized bound applies, rather than rejecting the whole budget to fallback, converging with the Langfuse backend that already field-filters. A malformed value for a recognized bound still fails loud (fallback-eligible), which 0109 deliberately leaves to each backend's trust model. Spec v0.104.0 is beyond the current v0.88.0 pin, so this ships ahead of the pin (unit-tested); the conformance fixtures ride the pin bump. +- **Pinned spec advances v0.88.0 to v0.112.0** across the v0.17.0 cycle, in two steps. The first reaches v0.107.0 (proposals 0094-0113); the second reaches v0.112.0, absorbing the Langfuse client-ownership and provider-isolation arc (proposals 0114-0118), which was opened by a registered consumer's leak report and settled across five spec releases. Per-proposal implementation status lives in the `conformance.toml` manifest: 0114 / 0116 / 0117 / 0118 are `implemented`, and 0115 is `partial` (its adapter-capability half ships, its provider-faithful Langfuse fake and `langfuse_client` construction directive do not). Spec v0.112.0 is flagged breaking for existing fixtures under the pre-1.0 allowance, because narrowing what a failed observation may carry under the default posture contradicted five shipped cases; those are reconciled at this pin. Observability fixtures 098 / 137 / 138 are un-deferred and now run against the post-0118 shape, and fixture 159 is wired. Fixtures 157 / 158 stay deferred on the missing §6.4 fake, with the source behavior covered meanwhile by the unit suite. Fixtures 150 / 151 stay deferred on the separate proposal 0107 mock-raises harness wiring. + ### Fixed - **The Langfuse failure-isolation marker no longer carries the caught exception's message** (proposal 0118, observability §8.4). **Behavioral for the Langfuse mapping.** The `openarmature.failure_isolated` marker span wrote the caught exception's message into `observation.metadata.error_message`, but that span is a graph-mechanism marker no §8.4.x mapping table covers, so writing harvested exception content onto it was non-conforming over-emission: an exception message that can echo application data (PII, tool arguments, an upstream API error body) reached the Langfuse backend under every privacy setting, since no knob gated it. The marker now carries only `error_category` (plus the caller-supplied `failure_isolation_event_name` and the node name), matching the node span's treatment, so an isolated node failure and an ordinary node failure render the same. The full exception is unaffected on the OTel side, where the failure-isolation span still carries it as the `openarmature.failure_isolation.message` attribute. A sweep of every bundled Langfuse handler against the same rule found no other unmapped harvested-content emission. diff --git a/conformance.toml b/conformance.toml index 54d61ef..9f9a6de 100644 --- a/conformance.toml +++ b/conformance.toml @@ -32,7 +32,7 @@ [manifest] implementation = "openarmature-python" -spec_pin = "v0.107.0" +spec_pin = "v0.112.0" # Adapter-level conformance capabilities (conformance-adapter §5.5, proposal # 0116). A fixture case may carry `requires_capability`, which selects the arm of @@ -1093,3 +1093,42 @@ note = "0111 scoped the collect-channel warning to inputs-based seeding; it now status = "implemented" since = "0.17.0" note = "0113 pins the malformed case of the §6 Managed-field collision MERGE arm: a caller's extras value that is not the field's expected list shape, or a list containing any element not of the expected element type/shape, is treated as ABSENT (the mapping sends only the value(s) it would send with no such extra present -- all-or-nothing, no partial salvage, no raise / diagnostic); malformation is judged structurally (no semantic vocabulary check). The shared apply_managed_extras merge arm gates on well-formedness (a string, or a list whose every element is a string): a malformed extra is skipped and the managed value stands alone. Fixture 081 (llm stop) exercises the general rule; the retrieval-provider §8.4 embedding_types arm (a malformed extras embedding_types -> wire [\"float\"], fixture 053) already shipped via 0099's bespoke Cohere merge." + +# ---- v0.17.0 spec-pin bump v0.107.0 -> v0.112.0 (proposals 0114-0118). +# The Langfuse client-ownership and provider-isolation arc, opened by a +# registered consumer's leak report and settled across five spec releases. + +# Spec v0.108.0 (proposal 0114). Langfuse client ownership and +# tracer-provider isolation (observability §6 / §8.9). +[proposals."0114"] +status = "implemented" +since = "0.17.0" +note = "Pins the two-mode Langfuse client ownership model. Mode (b): LangfuseObserver.from_credentials(public_key=..., secret_key=..., host=...) (over LangfuseSDKAdapter.from_credentials) builds an OA-owned client on a dedicated TracerProvider, so its observations no longer bind the globally-registered provider and export onto the application's OTel backend. Mode (a), a caller-supplied client, is never mutated: the caller stays responsible for isolating its tracer_provider and OA documents the remedy. secret_key is a pydantic SecretStr, masked in OA's reprs and logs with the plaintext read only at the SDK call; a blank credential is rejected at the boundary rather than falling through to the SDK's ambient LANGFUSE_* environment fallback. Fixture 157 is deferred: it needs the §6.4 provider-faithful fake and the langfuse_client construction directive, and is unit-tested meanwhile in tests/unit/test_langfuse_provider_isolation.py." + +# Spec v0.109.0 (proposal 0115). Conformance primitives for Langfuse +# provider isolation (conformance-adapter §5.5 / §6.4). +[proposals."0115"] +status = "partial" +since = "0.17.0" +note = "The adapter-capability half is implemented: conformance.toml declares langfuse_bound_provider_detection and the harness gates a case on requires_capability, treating a mismatched arm as a recognized skip and an undeclared capability name as an error. The §6.4 provider-faithful Langfuse fake (records observation content AND emits it as OTel spans through its bound TracerProvider, so a leak to a shared provider is catchable) and the langfuse_client construction directive are NOT yet built, so fixtures 157 / 158 stay deferred. The four leak assertions are declared in the expectations model so those fixtures parse at this pin, and the Langfuse runner fails loudly if an activated fixture reaches for one, so a declared-but-unimplemented assertion cannot read as coverage." + +# Spec v0.110.0 (proposal 0116). Fail closed when Langfuse payloads +# would reach a shared provider (observability §6). +[proposals."0116"] +status = "implemented" +since = "0.17.0" +note = "The Langfuse v4 SDK caches one client per public_key, so mode (b)'s dedicated provider takes effect only when OA is the first constructor for that credential; otherwise the SDK returns the cached client on its original provider and discards OA's. OA reuses one isolated provider per credential and reads the actual binding back after construction. Where a construction-determinable payload channel is live and OA establishes the client is bound to a provider it did not isolate, construction raises a categorized LangfuseProviderIsolationUnavailable before any observation is emitted; where OA cannot establish the binding at all it suppresses every channel and logs one WARNING; accept_shared_provider=True is the single opt-out, binding the provider the application already registered rather than letting the SDK construct and globally register one. With no channel live (the default posture) an un-isolatable client neither raises nor warns. Fixture 158 is deferred on the same §6.4 machinery as 157." + +# Spec v0.111.0 (proposal 0117). Payload-leak invariant broadened to all +# harvested-payload channels (observability §6 / §8.4). +[proposals."0117"] +status = "implemented" +since = "0.17.0" +note = "0116 guarded only the provider-payload channel. A sweep of OA's Langfuse observation surface found two more channels carrying payload OA harvests from the runtime: the Trace-level state input / output (disable_state_payload plus the trace_input_from_state / trace_output_from_state hooks, which emit regardless of the knob) and a failed observation's error_message. All three are covered. The scoping clause exempting caller-ATTACHED dimensions (correlation_id, session_id, the promoted userId, trace name, arbitrary caller metadata) is honored: those are opaque cross-backend join keys the caller owns and they ride verbatim. Fixture 158's six added cases ride with it." + +# Spec v0.112.0 (proposal 0118). The harvested error message comes under +# the payload flag (observability §5.5.4 / §8.4 / §8.7). +[proposals."0118"] +status = "implemented" +since = "0.17.0" +note = "disable_provider_payload now gates a failed provider call's harvested exception TEXT on all four provider observations, closing a hole in the flag's promise: a structured_output_invalid message commonly quotes the model output that failed validation, so a caller could set the flag, watch generation.output disappear as promised, and still receive that text in metadata.error_message. error_type is NOT gated: it is a classification token, so suppressing it buys no privacy while removing the only failure discriminator on a Tool observation, which has no error category at all; for the same reason it no longer counts toward the payload-bearing classification. The §8.4 exhaustive-mapping rule is honored too: the failure-isolation marker span carried the caught exception's message although no §8.4.x table maps it, which is non-conforming over-emission, and it now carries only error_category. Fixtures 098 / 137 / 138 are un-deferred and reconciled at this pin, and 159 is wired." diff --git a/openarmature-spec b/openarmature-spec index 0e08044..a770f82 160000 --- a/openarmature-spec +++ b/openarmature-spec @@ -1 +1 @@ -Subproject commit 0e080449ef68dc0b63ba2046b94e67dfacde8c1b +Subproject commit a770f824dd0ebff149e4650cb0423b44faa94b39 diff --git a/pyproject.toml b/pyproject.toml index 27b0535..af69af6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -63,7 +63,7 @@ Specification = "https://github.com/LunarCommand/openarmature-spec" openarmature = "openarmature.cli:main" [tool.openarmature] -spec_version = "0.107.0" +spec_version = "0.112.0" [dependency-groups] dev = [ diff --git a/src/openarmature/AGENTS.md b/src/openarmature/AGENTS.md index 08b242d..655d09e 100644 --- a/src/openarmature/AGENTS.md +++ b/src/openarmature/AGENTS.md @@ -1,6 +1,6 @@ # OpenArmature — Agent documentation -*This is the agent guide bundled with the openarmature Python package, version 0.16.0 (spec v0.107.0). For the full docs site see [openarmature.ai](https://openarmature.ai). For the canonical spec text see [openarmature.org/capabilities](https://openarmature.org/capabilities/). For project-specific conventions for the code you're editing, see the host project's `AGENTS.md` or `CLAUDE.md`.* +*This is the agent guide bundled with the openarmature Python package, version 0.16.0 (spec v0.112.0). For the full docs site see [openarmature.ai](https://openarmature.ai). For the canonical spec text see [openarmature.org/capabilities](https://openarmature.org/capabilities/). For project-specific conventions for the code you're editing, see the host project's `AGENTS.md` or `CLAUDE.md`.* ## TL;DR @@ -10,7 +10,7 @@ OpenArmature is a workflow framework for LLM pipelines and tool-calling agents: ## Capability contracts -_Sourced from openarmature-spec v0.107.0. Each entry below reproduces §1 (Purpose) and §2 (Concepts) of the capability's `spec.md` verbatim — including additions from accepted proposals that this Python implementation may not yet ship. For per-proposal implementation status (implemented / partial / textual-only / not-yet), see the `conformance.toml` manifest at the repo root. For the full spec text (execution model, error semantics, determinism, observer hooks, etc.) see the linked docs site._ +_Sourced from openarmature-spec v0.112.0. Each entry below reproduces §1 (Purpose) and §2 (Concepts) of the capability's `spec.md` verbatim — including additions from accepted proposals that this Python implementation may not yet ship. For per-proposal implementation status (implemented / partial / textual-only / not-yet), see the `conformance.toml` manifest at the repo root. For the full spec text (execution model, error semantics, determinism, observer hooks, etc.) see the linked docs site._ ### Capability: `graph-engine` diff --git a/src/openarmature/__init__.py b/src/openarmature/__init__.py index d0d1a3c..651c7c3 100644 --- a/src/openarmature/__init__.py +++ b/src/openarmature/__init__.py @@ -25,7 +25,7 @@ """ __version__ = "0.16.0" -__spec_version__ = "0.107.0" +__spec_version__ = "0.112.0" # Proposal 0052 (spec observability §5.1 / §8.4.1): canonical # package-registry name for this implementation. Surfaces on every # OTel invocation span as ``openarmature.implementation.name`` and on diff --git a/tests/conformance/harness/expectations.py b/tests/conformance/harness/expectations.py index 0f44a0e..f42b343 100644 --- a/tests/conformance/harness/expectations.py +++ b/tests/conformance/harness/expectations.py @@ -198,6 +198,17 @@ class ObservabilityExpected(_ForbidExtras): no_openarmature_spans_on_global: bool | None = None no_edge_spans: bool | None = None no_llm_provider_span: bool | None = None + # Langfuse provider-isolation leak assertions (proposals 0115 / 0116, + # fixtures 157 / 158). Declared so those fixtures PARSE at this pin; the + # assertions themselves are not implemented yet, and both fixtures are + # deferred accordingly. `_UNIMPLEMENTED_OBSERVABILITY_ASSERTIONS` in the + # Langfuse runner fails loudly if an ACTIVATED fixture reaches for one, so a + # declared-but-unimplemented key cannot pass as coverage. + no_langfuse_observations_on_global: bool | None = None + no_langfuse_observations_on_private: bool | None = None + langfuse_observations_on_global: bool | None = None + no_payload_bearing_langfuse_observations_on_global: bool | None = None + payload_bearing_langfuse_observations_on_global: bool | None = None # Invariants block (fixture 011 determinism). invariants: dict[str, Any] | None = None determinism_check: dict[str, Any] | None = None @@ -259,6 +270,12 @@ class ObservabilityExpected(_ForbidExtras): "no_openarmature_spans_on_global", "no_edge_spans", "no_llm_provider_span", + # proposals 0115 / 0116 (fixtures 157 / 158) Langfuse leak assertions. + "no_langfuse_observations_on_global", + "no_langfuse_observations_on_private", + "langfuse_observations_on_global", + "no_payload_bearing_langfuse_observations_on_global", + "payload_bearing_langfuse_observations_on_global", "determinism_check", "invocation_count", # proposal 0064 (fixture 084) Langfuse Trace-level expectations diff --git a/tests/conformance/test_observability.py b/tests/conformance/test_observability.py index c8c2ded..0e8ff03 100644 --- a/tests/conformance/test_observability.py +++ b/tests/conformance/test_observability.py @@ -259,6 +259,12 @@ def _reset_otel_global_tracer_provider(restore_to: object) -> None: "095-tool-call-id-links-to-llm-request", "096-tool-call-payload-gating", "097-otel-tool-span-attributes", + # 098 (proposal 0118, spec v0.112.0): reconciled at this pin. Its + # failure case moves to disable_provider_payload=false (asserting the + # message literally is its purpose) and it gains a default-posture + # failure case carrying the Tool anti-smuggling clause, which 158 + # cannot cover for a detection-capable adapter (that arm raises first). + "098-langfuse-tool-observation", # v0.16.0 — proposal 0059 embedding observability (0059b). A # calls_embed node awaits OpenAIEmbeddingProvider.embed() inside the # node body; the typed EmbeddingEvent / EmbeddingFailedEvent drive the @@ -278,6 +284,10 @@ def _reset_otel_global_tracer_provider(restore_to: object) -> None: "081-embedding-event-active-prompt-populated", "082-otel-embedding-span-attributes", "083-langfuse-embedding-observation", + # 137 / 138 (proposal 0118): the payload-suppressed cases now assert the + # harvested error message ABSENT with error_type retained, via the + # metadata_absent directive that landed in #267. + "137-langfuse-embedding-failure-observation", "139-otel-embedding-no-usage-input-tokens-omitted", "140-langfuse-embedding-no-usage-usagedetails-omitted", # proposal 0067 §11 embedding metrics: token.usage (input only) + @@ -304,6 +314,7 @@ def _reset_otel_global_tracer_provider(restore_to: object) -> None: "106-rerank-event-active-prompt-populated", "107-otel-rerank-span-attributes", "108-langfuse-rerank-observation", + "138-langfuse-rerank-failure-observation", "141-otel-rerank-no-usage-attributes-omitted", "142-langfuse-rerank-no-usage-usagedetails-omitted", "109-rerank-metrics-token-and-duration", @@ -338,18 +349,6 @@ def _reset_otel_global_tracer_provider(restore_to: object) -> None: # pin bump is now the only remaining prerequisite; the source behavior here is # covered meanwhile by the default-posture unit tests in # tests/unit/test_langfuse_provider_isolation.py. - "098-langfuse-tool-observation": ( - "Proposal 0118 error-message gating: fixture asserts the pre-0118 shape; " - "reconciled at the v0.112.0 pin bump" - ), - "137-langfuse-embedding-failure-observation": ( - "Proposal 0118 error-message gating: fixture asserts the pre-0118 shape; " - "reconciled at the v0.112.0 pin bump" - ), - "138-langfuse-rerank-failure-observation": ( - "Proposal 0118 error-message gating: fixture asserts the pre-0118 shape; " - "reconciled at the v0.112.0 pin bump" - ), # Proposal 0045 IS implemented (v0.11.0), but the nested-case Langfuse # fixture stays deferred: it needs runtime-state item-list lookup for # nested fan-outs plus an augment_metadata_from_outer_item directive @@ -438,17 +437,35 @@ def _reset_otel_global_tracer_provider(restore_to: object) -> None: ), # Proposal 0107 (spec v0.102.0) mock_embedding / mock_rerank raises # sub-directive -> literal error-field assertion. + # The 0118 half of these two resolved at this pin (both cases moved to + # disable_provider_payload=false, since asserting the message literally is + # their purpose). What remains is the 0107 mock-raises harness wiring. "150-langfuse-embedding-failure-literal-error-fields": ( - "Proposal 0107 mock-raises literal error fields; harness wiring rides the v0.17.0 " - "fixture-wiring PR. Also asserts the pre-0118 shape at this pin: spec moves both " - "cases to disable_provider_payload=false at v0.112.0, since asserting the message " - "literally is their purpose" + "Proposal 0107 mock_embedding raises sub-directive not yet wired; rides the " + "remaining v0.17.0 fixture-wiring PR (observability 144-156)" ), "151-langfuse-rerank-failure-literal-error-fields": ( - "Proposal 0107 mock-raises literal error fields; harness wiring rides the v0.17.0 " - "fixture-wiring PR. Also asserts the pre-0118 shape at this pin: spec moves both " - "cases to disable_provider_payload=false at v0.112.0, since asserting the message " - "literally is their purpose" + "Proposal 0107 mock_rerank raises sub-directive not yet wired; rides the " + "remaining v0.17.0 fixture-wiring PR (observability 144-156)" + ), + # 157 / 158 (proposals 0115 / 0116 / 0117 / 0118, spec v0.109.0-v0.112.0). + # The src side shipped ahead of this pin and is unit-tested; what these need + # is harness machinery: the provider-faithful Langfuse fake of + # conformance-adapter 6.4 (records observation content AND emits it through + # its bound TracerProvider, so a leak to a shared provider is catchable), the + # langfuse_client construction directive, expected_construction_error, and + # the four payload-scoped leak assertions. 158 additionally gates arms on the + # requires_capability audience gate. + "157-langfuse-provider-isolation": ( + "needs the conformance-adapter 6.4 provider-faithful Langfuse fake and the " + "langfuse_client construction directive; src side is unit-tested meanwhile in " + "tests/unit/test_langfuse_provider_isolation.py" + ), + "158-langfuse-payload-leak-fail-closed": ( + "needs the provider-faithful fake, the langfuse_client singleton sub-directives, " + "expected_construction_error, and the payload-scoped leak assertions; src side is " + "unit-tested meanwhile in tests/unit/test_langfuse_provider_isolation.py and " + "tests/unit/test_langfuse_payload_leak_canary.py" ), # Spec v0.103.1 conformance coverage (0084 orphan-fallback arms + the # embedding failure-metrics counterpart). @@ -498,6 +515,10 @@ def _reset_otel_global_tracer_provider(restore_to: object) -> None: # sibling harness; the generic topology path cannot model the # `calls_llm_from_wrapper` orphan-call primitive. "134-langfuse-nested-fan-out-parent-resolution", + # 159 (proposal 0118): the failed Generation's harvested error message + # under `disable_provider_payload`. Its langfuse_trace shape lives in the + # sibling harness, like 123 and 130. + "159-langfuse-llm-failure-error-message", } ) diff --git a/tests/conformance/test_observability_langfuse.py b/tests/conformance/test_observability_langfuse.py index 34947d5..6600074 100644 --- a/tests/conformance/test_observability_langfuse.py +++ b/tests/conformance/test_observability_langfuse.py @@ -146,6 +146,14 @@ # `calls_llm_from_wrapper` orphan primitive are not modeled by the generic # ``_run_case`` topology path (precedent: the 039 hand-built runner). "134-langfuse-nested-fan-out-parent-resolution", + # 159 (proposal 0118, spec v0.112.0): `disable_provider_payload` governs a + # failed Generation's harvested exception text. Both cases run on a normal + # provider and need no isolation machinery, only the existing per-observer + # payload flag, so this rides the pin bump rather than waiting on the + # provider-faithful fake that 157 / 158 need. Each case keeps the other + # non-vacuous: flag on asserts the message absent with the category + # retained, flag off asserts it present. + "159-langfuse-llm-failure-error-message", } ) @@ -502,6 +510,11 @@ async def test_langfuse_fixture(fixture_path: Path) -> None: report_recognized_skip(fixture_stem, case_name, gate) continue ran_134 += 1 + # 134 bypasses _run_case, where the guard otherwise lives, so it is + # the one dispatch path that has to call it itself. + _reject_unimplemented_assertions( + fixture_stem, case_name, cast("Mapping[str, Any]", case.get("expected") or {}) + ) try: await _run_langfuse_134(case) except AssertionError as e: @@ -1049,6 +1062,15 @@ def _resolve_detached_wrapper_names(case: Mapping[str, Any]) -> frozenset[str]: async def _run_case(case: Mapping[str, Any], *, fixture_stem: str | None = None) -> None: + # Chokepoint for the guard: both the multi-case loop and the single-case path + # funnel through here, so wiring it at the call sites instead would leave a + # future dispatch path free to skip it silently. `_run_langfuse_134` bypasses + # this function entirely and so carries its own call. + _reject_unimplemented_assertions( + fixture_stem or "", + cast("str", case.get("name") or ""), + cast("Mapping[str, Any]", case.get("expected") or {}), + ) # 039 additionally enforces proposal 0045's MUST-NOT scoping (an augmented # key absent from an observation's expected metadata must be absent in the # actual); other fixtures keep the established subset semantics. @@ -2342,9 +2364,42 @@ def _assert_observation_tree( # how the 0118 ``metadata_absent`` directive would have landed. Anything outside # this set fails loudly instead, so a spec-side directive we have not built yet # surfaces as a gap rather than a false pass. +# Assertion keys the expectations model accepts so fixtures 157 / 158 parse at the +# v0.112.0 pin, but which no comparator implements yet (they need the +# provider-faithful fake and the `langfuse_client` construction directive). +# +# Declaring a field to make a fixture parse is how an assertion quietly becomes +# dead: the key validates, the comparator never looks at it, and the fixture reads +# like coverage. Both fixtures are deferred, so nothing should reach these today; +# this fails loudly if an activated one ever does, rather than waiting for someone +# to notice the assertion was never running. +_UNIMPLEMENTED_OBSERVABILITY_ASSERTIONS = frozenset( + { + "no_langfuse_observations_on_global", + "no_langfuse_observations_on_private", + "langfuse_observations_on_global", + "no_payload_bearing_langfuse_observations_on_global", + "payload_bearing_langfuse_observations_on_global", + } +) + + +def _reject_unimplemented_assertions(fixture_stem: str, case_name: str, expected: Mapping[str, Any]) -> None: + """Fail if a case reaches for a leak assertion no comparator implements.""" + unimplemented = sorted(set(expected) & _UNIMPLEMENTED_OBSERVABILITY_ASSERTIONS) + if unimplemented: + raise AssertionError( + f"{fixture_stem}::{case_name} declares assertions this harness accepts for parsing " + f"but does not implement: {unimplemented}. They are declared in " + f"harness/expectations.py so fixtures 157 / 158 parse at this pin; implement them " + f"with the provider-faithful fake before activating a fixture that uses them." + ) + + _OBSERVATION_DIRECTIVES = frozenset( { "children", + "input", "input_is_raw_string_with_marker", "input_parses_as_messages", "level", @@ -2434,6 +2489,15 @@ def _assert_observation( assert "[truncated," in actual.input, ( f"observation {actual.name!r} input missing truncation marker: {actual.input!r}" ) + if "input" in expected: + # Plain equality on the rendered input, the counterpart of `output`. The + # two shaped forms above (`input_parses_as_messages`, + # `input_is_raw_string_with_marker`) assert structure; this asserts the + # value, which is what a fixture needs to pin `input: null` on a payload- + # suppressed observation. + assert actual.input == expected["input"], ( + f"observation {actual.name!r} input: expected {expected['input']!r}, got {actual.input!r}" + ) if "output" in expected: assert actual.output == expected["output"], ( f"observation {actual.name!r} output: expected {expected['output']!r}, got {actual.output!r}" diff --git a/tests/conformance/test_unimplemented_assertion_guard.py b/tests/conformance/test_unimplemented_assertion_guard.py new file mode 100644 index 0000000..934b850 --- /dev/null +++ b/tests/conformance/test_unimplemented_assertion_guard.py @@ -0,0 +1,114 @@ +"""The guard on assertion keys declared for parsing but not implemented.""" + +# Fixtures 157 / 158 arrive with the v0.112.0 pin carrying leak assertions no +# comparator implements yet, and the `expected` model forbids extras, so those +# keys must be declared for the fixtures to parse at all. Declaring a field to +# make a fixture parse is how an assertion quietly becomes dead: the key +# validates, no comparator reads it, and the fixture reads like coverage. +# +# `_reject_unimplemented_assertions` exists to make that loud. Both fixtures are +# deferred, so NOTHING in the corpus reaches the guard today -- which means +# without these tests the guard would itself be unverified, the exact shape it +# was written to prevent. + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest + +from tests.conformance.harness.expectations import ObservabilityExpected + +from . import test_observability_langfuse as langfuse_runner + + +def test_set_covers_every_leak_assertion_the_deferred_fixtures_use() -> None: + # Derived from the CORPUS, not from the set under test. The parametrized test + # below takes its cases from the set itself, so dropping a key there would + # silently shrink the run rather than fail; this reads the fixtures and + # notices a leak assertion that nothing guards -- including one a future pin + # adds to 158. + used: set[str] = set() + for stem in ("157-langfuse-provider-isolation", "158-langfuse-payload-leak-fail-closed"): + spec = langfuse_runner._load(langfuse_runner.CONFORMANCE_DIR / f"{stem}.yaml") + for case in spec.get("cases") or [spec]: + used |= set(case.get("expected") or {}) + + leak_keys = {k for k in used if "langfuse_observations" in k or "payload_bearing" in k} + assert leak_keys, "no leak assertions found in 157 / 158; this check is reading nothing" + unguarded = sorted(leak_keys - langfuse_runner._UNIMPLEMENTED_OBSERVABILITY_ASSERTIONS) + assert not unguarded, ( + f"fixtures 157 / 158 assert {unguarded}, which the expectations model accepts for parsing " + f"but no comparator implements and nothing guards. Add them to " + f"_UNIMPLEMENTED_OBSERVABILITY_ASSERTIONS, or implement them." + ) + + +def test_every_unimplemented_key_is_a_real_model_field() -> None: + # A typo in the set would make the guard silently never match the key it was + # meant to catch, so the set is pinned against the model that declares them. + declared = set(ObservabilityExpected.model_fields) + unknown = sorted(langfuse_runner._UNIMPLEMENTED_OBSERVABILITY_ASSERTIONS - declared) + assert not unknown, ( + f"{unknown} are listed as unimplemented assertions but are not fields on " + f"ObservabilityExpected, so the guard would never fire for them. Fix the spelling or " + f"drop them from the set." + ) + + +def _inject(monkeypatch: pytest.MonkeyPatch, key: str) -> None: + """Graft `key` onto the first case of whatever fixture the runner loads.""" + original = langfuse_runner._load + + def _loader(path: Path) -> dict[str, Any]: + spec = original(path) + # A single-case fixture IS the case: it carries `expected` at top level + # with no `cases` key, which is the shape that reaches the runner's + # single-case dispatch path. + target = spec["cases"][0] if "cases" in spec else spec + target.setdefault("expected", {})[key] = True + return spec + + monkeypatch.setattr(langfuse_runner, "_load", _loader) + + +# One activated fixture per DISPATCH PATH through test_langfuse_fixture. Covering +# only the multi-case loop is how this guard shipped wired into one path of three: +# the tests exercised the path that already worked. +_PATHS = { + "multi_case_loop": "023-langfuse-generation-rendering", + "single_case": "022-langfuse-basic-trace", + "hand_built_134": "134-langfuse-nested-fan-out-parent-resolution", +} + + +@pytest.mark.parametrize("path", sorted(_PATHS)) +@pytest.mark.parametrize("key", sorted(langfuse_runner._UNIMPLEMENTED_OBSERVABILITY_ASSERTIONS)) +async def test_an_activated_case_reaching_an_unimplemented_assertion_fails( + monkeypatch: pytest.MonkeyPatch, key: str, path: str +) -> None: + # Every key against every path: a guard covering four of five keys, or two of + # three paths, reads identically to one that covers all of them. + _inject(monkeypatch, key) + with pytest.raises(AssertionError, match="does not implement"): + await langfuse_runner.test_langfuse_fixture(langfuse_runner.CONFORMANCE_DIR / f"{_PATHS[path]}.yaml") + + +@pytest.mark.parametrize("path", sorted(_PATHS)) +async def test_each_dispatch_path_passes_when_nothing_is_injected(path: str) -> None: + # Non-vacuity for the matrix above, per path: each fixture passes untouched, + # so the failures there come from the injected key rather than from the + # fixture or the dispatch path itself. + await langfuse_runner.test_langfuse_fixture(langfuse_runner.CONFORMANCE_DIR / f"{_PATHS[path]}.yaml") + + +def test_the_named_fixtures_still_take_the_paths_they_are_meant_to() -> None: + # The matrix is only three-path coverage while these fixtures keep their + # shapes. A spec edit that gives 022 a `cases:` block would silently collapse + # two of the three paths onto one, with everything still green. + single = langfuse_runner._load(langfuse_runner.CONFORMANCE_DIR / f"{_PATHS['single_case']}.yaml") + multi = langfuse_runner._load(langfuse_runner.CONFORMANCE_DIR / f"{_PATHS['multi_case_loop']}.yaml") + assert "cases" not in single, f"{_PATHS['single_case']} gained a `cases:` block; pick another" + assert "cases" in multi, f"{_PATHS['multi_case_loop']} lost its `cases:` block; pick another" + assert _PATHS["hand_built_134"] == langfuse_runner._FIXTURE_134 diff --git a/tests/test_smoke.py b/tests/test_smoke.py index 0c70f07..dd3ffaf 100644 --- a/tests/test_smoke.py +++ b/tests/test_smoke.py @@ -9,7 +9,7 @@ def test_package_versions() -> None: assert openarmature.__version__ == "0.16.0" - assert openarmature.__spec_version__ == "0.107.0" + assert openarmature.__spec_version__ == "0.112.0" def test_spec_version_matches_pyproject() -> None: