Skip to content

MonitorFormulaAndFunctionEventsDataSource missing llm_observability: multi-query llm-observability alert monitors cannot be created #4577

Description

@jrmils89

Describe the bug

MonitorFormulaAndFunctionEventsDataSource does not include llm_observability, so multi-query (formula) monitors of type llm-observability alert cannot be created through this client — or through anything generated from it, including the Terraform provider.

The monitor type itself was added in #4436. That PR extended model_monitor_type.go and the OpenAPI schema, but did not extend model_monitor_formula_and_function_events_data_source.go, and its test cassette only exercises the single-query form (llm-observability("*").rollup("count").last("2h") > 0). Single-query monitors of this type therefore work, and multi-query ones cannot be expressed at all.

The API supports the multi-query form. It is in fact the only accepted value for this monitor type — the API says so when given anything else:

400 Bad Request
{"errors":["Value of parameter 'data_source' for 'llm-observability alert' must be 'llm_observability' not 'spans'"]}

So the API requires a value the client refuses to serialize.

To Reproduce

  1. Attempt to create a monitor of type llm-observability alert with two variables entries using data_source: "llm_observability" and a formula(...) query.
  2. The client rejects it before issuing a request:
invalid value 'llm_observability' for MonitorFormulaAndFunctionEventsDataSource:
valid values are [rum ci_pipelines ci_tests audit events logs spans
database_queries network network_path]
  1. Substituting spans — the nearest available value — reaches the API and returns the 400 quoted above, naming llm_observability as the required value.

A payload the API accepts, which no generated client can currently produce:

{
  "name": "example",
  "type": "llm-observability alert",
  "query": "formula(\"query / query1\").last(\"1h\") > 0.05",
  "message": "example",
  "options": {
    "thresholds": { "critical": 0.05 },
    "variables": [
      {
        "name": "query",
        "indexes": ["llmobs"],
        "data_source": "llm_observability",
        "compute": { "aggregation": "count" },
        "search": { "query": "@event_type:span @status:error" }
      },
      {
        "name": "query1",
        "indexes": ["llmobs"],
        "data_source": "llm_observability",
        "compute": { "aggregation": "count" },
        "search": { "query": "@event_type:span" }
      }
    ]
  }
}

Expected behavior

llm_observability is accepted as a MonitorFormulaAndFunctionEventsDataSource value, so multi-query llm-observability alert monitors can be created through the client and the Terraform provider — matching what rum alert already supports and what the API already accepts.

Environment and Versions

Additional context

Impact is on ratio-based alerting. A burn-rate or error-budget monitor is bad / total, which needs two queries and a formula — the shape rum alert uses today. Without the enum value, that shape is unavailable for LLM Observability data, and since custom span tags are deliberately not exposed as ml_obs.* metric tags, there is no equivalent metric-based route for signals that live on span attributes.

The workaround is the Terraform provider's datadog_monitor_json, which passes raw JSON and so bypasses the enum. It works, but it forgoes the server-side validation the typed resource performs during plan — a monitor missing an org-required tag still plans clean under monitor_json, while the typed resource rejects it up front. So the workaround trades away exactly the pre-apply safety net that makes monitor changes reviewable.

Since the clients are generated, I assume the actual change belongs in the API spec rather than here; filing against the client as the visible surface. Happy to open a PR if it is useful, though a generated enum may not be the right place for an external contribution.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugBug related issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions