Skip to content

[Feat] Support external OTLP collector for trace runtime - #2041

Open
matrix72c wants to merge 5 commits into
InternLM:mainfrom
matrix72c:feat/external-otel-collector
Open

[Feat] Support external OTLP collector for trace runtime#2041
matrix72c wants to merge 5 commits into
InternLM:mainfrom
matrix72c:feat/external-otel-collector

Conversation

@matrix72c

@matrix72c matrix72c commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add TraceConfig.external_otlp_endpoint so all XTuner processes can export spans to an externally managed, network-reachable OTLP collector
  • keep the existing driver-local collector and per-run traces.jsonl as the default when no external endpoint is configured
  • propagate the external OTLP endpoint and trace run metadata to Ray child processes
  • optionally point the XTuner viewer at an external collector's shared JSONL output through external_trace_jsonl_path
  • support external Jaeger-only deployments when the XTuner viewer is disabled

Motivation

The XTuner viewer already reads traces.jsonl directly. It is a consumer of the trace file, however, not the component that receives spans or writes that file.

The default trace path introduced in #1946 is:

driver / Ray actors
        | OTLP/gRPC
        v
driver-local collector (127.0.0.1)
        |-- file exporter --> traces.jsonl --> XTuner viewer
        `-- OTLP exporter --> Jaeger

This works when every traced process can reach the driver's loopback collector. In a multi-pod deployment, each pod resolves 127.0.0.1 to itself, so a remote Ray actor cannot export spans to the collector running in the driver pod. The resulting JSONL and Jaeger trace therefore miss spans from remote actors.

This PR allows the producers to use a routable collector Service instead:

driver / Ray actors in multiple pods
        | OTLP/gRPC
        v
external collector Service
        |-- file exporter --> shared traces.jsonl --> XTuner viewer
        `-- exporter ------> Jaeger or another backend

Configuration and ownership

TraceConfig(
    enabled=True,
    external_otlp_endpoint="http://otel-collector.namespace.svc:4317",
    external_trace_jsonl_path="/shared/traces/traces.jsonl",
    xtuner_viewer_enabled=True,
)
  • external_otlp_endpoint is propagated to Ray child processes so every process exports to the same reachable collector.
  • The external collector is deployed, configured, started, and stopped outside XTuner.
  • external_trace_jsonl_path does not configure the collector and is not propagated to Ray child processes. It only tells the driver-side viewer where to read the collector-owned JSONL file.
  • When the viewer is enabled, that path must refer to the same underlying file written by the external collector and must be visible from the driver, for example through a shared volume.
  • The viewer filters a shared JSONL file by service.name and XTuner run.id, so the file may contain spans from multiple runs.
  • If the external collector only exports to Jaeger or another backend, leave xtuner_viewer_enabled=False; no JSONL path is required.
  • Viewer JSONL loading is deferred until the first request so the external collector may create the file after the viewer process starts.

Compatibility

When external_otlp_endpoint is unset, local collector startup, per-run traces.jsonl output, viewer startup, and shutdown behavior remain unchanged. External collectors are never started or stopped by XTuner.

Verification

  • 8 focused tests covering local defaults, external collector lifecycle, endpoint propagation to Ray children, collector ownership of JSONL, deferred viewer loading, shared JSONL viewer configuration, and invalid configuration combinations
  • ruff check
  • compileall
  • git diff --check

@matrix72c
matrix72c force-pushed the feat/external-otel-collector branch from d6f6738 to 7681fde Compare August 24, 2026 14:09
@windreamer
windreamer requested a review from YanhuiDua August 24, 2026 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant