Skip to content

Add IdleTimeAnalyser: classify GPU idle gaps by root cause - #620

Open
ajassani wants to merge 8 commits into
mainfrom
ajassani/idle-time-classification
Open

Add IdleTimeAnalyser: classify GPU idle gaps by root cause#620
ajassani wants to merge 8 commits into
mainfrom
ajassani/idle-time-classification

Conversation

@ajassani

@ajassani ajassani commented May 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Adds an IdleTimeAnalyser module that classifies every gap between consecutive GPU events (kernels, memcpy, memset) by root cause. Instead of reporting GPU idle time as a single number, each idle interval is classified along two axes so engineers get actionable information:

  • drain_type — sync_drain (a sync call blocked the CPU while the GPU drained its queue) or starved (the CPU couldn't submit work fast enough)
  • cpu_during_gap — LAUNCH_ANOMALY, LAUNCH_OVERHEAD_ONLY, RUNTIME_DOMINATED, CPU_DOMINATED, or CPU_UNTRACED

Usage

# Add idle_overview / idle_summary / idle_intervals sheets to the PyTorch perf report
TraceLens_generate_perf_report_pytorch --profile_json_path trace.json --enable_idle_analysis

# Also write <trace>_idle_augmented.json.gz with idle annotation tracks for Perfetto
TraceLens_generate_perf_report_pytorch --profile_json_path trace.json --enable_idle_analysis --enable_augmented_trace

# Standalone: Excel + augmented trace without the full report
python -m TraceLens.Reporting.classify_idle_time trace.json
from TraceLens import TreePerfAnalyzer, IdleTimeAnalyser
dfs = IdleTimeAnalyser(TreePerfAnalyzer.from_file("trace.json").tree).get_dataframes()

What is new

  • TraceLens/IdleTimeAnalyser/ — classify.py (classifier), report.py (DataFrame builder), IdleTimeAnalyser wrapper exported from TraceLens
  • --enable_idle_analysis / --enable_augmented_trace on generate_perf_report_pytorch; --micro_idle_thresh_us doubles as the noise threshold (5 µs default)
  • TraceLens/Reporting/classify_idle_time.py — thin standalone CLI over the module
  • LAUNCH_ANOMALY for non-prequeued kernels requires launch_to_exec > 10 µs and > 25% of the gap (reduces false positives on long gaps)
  • cudaLaunchKernelExC recognized as a launch call
  • Docs: docs/how-to/analyze-idle-time.md and docs/reference/idle-time-columns.md, linked from the TOC, the PyTorch report page, and the API reference

Tests

tests/test_classify_idle_time.py (70 tests with copyright checks):

  • Unit tests for self-time attribution, sync / runtime classification, interval extraction, overlap queries
  • Synthetic scenarios for each classification path: launch anomaly (launched-during-gap and prequeued), the 25% guard, causal vs non-causal sync drain, runtime-dominated, CPU-dominated, CPU-untraced
  • Real-trace regression on a DDP ResNet-18 fixture (tests/traces/mi300/ddp_resnet18_no_pool/) and the existing FSDP rank-0 trace
  • End-to-end: standalone CLI outputs, and generate_perf_report_pytorch with / without --enable_idle_analysis

Coverage of the new code: classify.py 98%, report.py 95%, classify_idle_time.py 96%, __init__.py 100%. Existing PyTorch / inference perf-report regression tests pass unchanged (idle sheets are opt-in).

ajassani added 5 commits April 23, 2026 21:15
@ajassani
Add GPU idle time classification for PyTorch profiler traces
3aa6cec 
Classifies each GPU idle interval along two axes:
- drain_type: why the GPU queue became empty (sync_drain vs starved)
- cpu_during_gap: what the CPU was doing (LAUNCH_ANOMALY, LAUNCH_OVERHEAD_ONLY,
  RUNTIME_DOMINATED, CPU_DOMINATED, CPU_UNTRACED)

Includes:
- classify_idle_time.py: core classifier with OverlapIndex for fast queries,
  self-time based dominant op detection, augmented trace generation, Excel reports
- idle_time_extension.py: integration hook for generate_perf_report_pytorch
- docs/idle_time_classification.md: column documentation for all three sheets
  (idle_overview, idle_summary, idle_intervals)
- tests/test_classify_idle_time.py: 43 unit tests

Made-with: Cursor
@ajassani
Require launch_to_exec > 25% of gap for non-prequeued LAUNCH_ANOMALY
b29f25a 
Previously, any non-prequeued interval with launch_to_exec > 10µs was
classified as LAUNCH_ANOMALY regardless of gap duration. This caused
intervals like idle#136 in ddp_resnet18 (858µs gap, 10.2µs launch_to_exec)
to be labeled LAUNCH_ANOMALY when the real bottleneck was CPU/runtime
overhead consuming 99% of the gap.

Now non-prequeued LAUNCH_ANOMALY requires both:
  - launch_to_exec > 10µs (absolute threshold)
  - launch_to_exec > 25% of gap duration (significance check)

39 intervals across 3 traces reclassify (all had l2e/dur < 2%).
Zero impact on MI325X Llama/Mistral prequeued anomalies.

Made-with: Cursor
@ajassani
Add IdleTimeAnalyser submodule with built-in perf report integration
069d0c9 
Promote idle time classification from an external extension to a
first-class TraceLens feature:

- New TraceLens/IdleTimeAnalyser/ package (classify.py, report.py,
  __init__.py) wrapping the classification and DataFrame logic in a
  reusable IdleTimeAnalyser class.
- Built-in --enable_idle_analysis and --enable_augmented_trace flags
  on generate_perf_report_pytorch, adding idle_overview,
  idle_summary, and idle_intervals sheets without an extension file.
- User-facing guide (docs/idle_time_guide.md) with background
  concepts, quick start, report reading walkthrough, and overhead
  benchmarks.
- Updated docs/generate_perf_report.md and
  docs/idle_time_classification.md to reference the new flags.
- 7 new tests for the wrapper and report module (50 total, all
  passing). Added test_classify_idle_time.py to CI workflow.
- Deprecated idle_time_extension.py with a notice pointing to the
  built-in flag.

Made-with: Cursor
@ajassani
Add cudaLaunchKernelExC to LAUNCH_NAMES
ce7f43e 
This CUDA extended launch API variant was falling through to
OTHER_RUNTIME instead of LAUNCH_STALL, misclassifying 21 intervals
across H100 traces.

Made-with: Cursor
@ajassani
Add DDP resnet18 trace and real-trace regression tests
86d615a 
Add a DDP resnet18 trace (no CUDA memory pool) that exercises the
MEMORY_ALLOC runtime sub-type, the only classification branch not
covered by existing repo traces. Also add a regression test asserting
FSDP rank0 covers LAUNCH_ANOMALY, CPU_DOMINATED, LAUNCH_OVERHEAD_ONLY,
and CPU_UNTRACED categories.

Made-with: Cursor
This was referenced May 5, 2026
Open
Open
ajassani and others added 3 commits September 30, 2026 16:34
@ajassani
Merge remote-tracking branch 'github/main' into ajassani/idle-time-cl…
fe4f853 
…assification

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	.github/workflows/regression-tests.yml
#	TraceLens/Reporting/generate_perf_report_pytorch.py
#	docs/generate_perf_report.md
@ajassani @cursoragent
IdleTimeAnalyser: adapt to current main and deduplicate the standalon…
495af75 
…e CLI

- Use TraceLens.util.merge_intervals (moved out of GPUEventAnalyser on main).
- Make Reporting/classify_idle_time.py a thin CLI over IdleTimeAnalyser.classify/report
  instead of a full copy of the classifier and report builder.
- Drop the unreleased idle_time_extension.py; --enable_idle_analysis replaces it.
- Remove dead code (compute_median_launch_latency, is_memcpy_with_kind, detect_workload_tags,
  unused anomaly_multiplier parameter).
- Restore the json/gzip imports the augmented-trace path needs after the merge.
- Add copyright headers, apply black, and add tests for launch-anomaly / sync-drain /
  runtime / CPU classification, the CLI, and the perf-report integration.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ajassani @cursoragent
docs: move idle time analysis docs into the ROCm docs site
31861c1 
Port the idle time guide and column reference to how-to/analyze-idle-time.md and
reference/idle-time-columns.md, add them to the TOC, and document --enable_idle_analysis /
--enable_augmented_trace on the PyTorch report page and in the API reference.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ajassani
ajassani marked this pull request as ready for review September 30, 2026 20:58
gabeweisz
gabeweisz approved these changes Oct 1, 2026
View reviewed changes

@gabeweisz gabeweisz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find the multi-level process of classification a bit confusing here, but it seems like you went through this pretty carefully to figure out how to do it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gabeweisz gabeweisz gabeweisz approved these changes
@devalshahamd devalshahamd Awaiting requested review from devalshahamd devalshahamd is a code owner
@tsrikris tsrikris Awaiting requested review from tsrikris tsrikris is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ajassani @gabeweisz