Skip to content

fix: reuse default tracers without resetting warning deduplication - #73

Open
rhewett-nv wants to merge 1 commit into
mainfrom
rhewett/fix/tracer-warning-dedup
Open

rhewett-nv wants to merge 1 commit into
mainfrom
rhewett/fix/tracer-warning-dedup

Conversation

@rhewett-nv

Copy link
Copy Markdown
Collaborator

What does this PR do?

Reuse Lens default tracers to prevent repeated OpenTelemetry SDK lookups from resetting Python warning deduplication.

OpenTelemetry SDK 1.44 changes warning filters on every get_tracer() call, allowing previously emitted warnings to print again. This change caches default tracers by provider and
instrumentation scope, resetting the cache after provider changes or process forks.

Issues

No linked issue.

Usage

No usage changes are required. Existing instrumentation automatically reuses default tracers:

from nemo.lens import span_cm

for step in range(100):
with span_cm("training.step"):
train_step()

Explicitly supplied tracers and disabled span groups retain their existing behavior.

Before your PR is "Ready for review"

Pre checks:

  • I read the Contributor guidelines (/NVIDIA-NeMo/lens/blob/main/CONTRIBUTING.md).
  • All commits are signed off (git commit -s) per the DCO.
  • I added or updated tests under tests/ for any behavior change.
  • I ran pytest and pre-commit run --all-files locally; both pass.
  • I updated docs/ and README.md where behavior, configuration, or public API changed.
  • If this changes a lens public API symbol, I updated the cross-repo _fallbacks.py files in Megatron-LM / RL / Gym. Not applicable: public signatures are unchanged.

Additional information

  • All 460 tests pass, including regression coverage for warning deduplication, concurrent lookups, provider replacement, fork safety, explicit tracers, and disabled groups.
  • Updated the instrumentation documentation; no README changes were necessary.
  • Lens does not introduce global warning suppression or rank-specific policy. Persistent rank-zero warning deduplication is handled separately in Megatron.

Signed-off-by: Russell Hewett <rhewett@nvidia.com>

This branch was successfully deployed

1 active deployment
public 297df912 Deployed Sep 21, 2026 by copy-pr-bot[bot] via release / finalize / notify #144
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant