[Bug] A5 2-card: intermittent GM-read stall wedges AICore under repeated dispatch (CSA kernels + cross-card MoE traffic); HEAP_RING_DEADLOCK at <=1GiB ring heap

[lwDavid]

Background

In pypto-lib, models/deepseek_v4_pro/prefill_layer.py on a5 (2 cards) has a runtime problem: sustained benchmark re-dispatch of the same prepared graph intermittently wedges the device — ~84 AIC/AIV cores hit the AICore watchdog at once (507015, subErrType 0x4, "timeout or trap error") at a random round (observed rounds 5–62 of a 100-round bench across many runs), and the device pair frequently stays unusable afterwards (the next run fails startup with chip worker(s) [N] did not become ready within 300.0s). This is the residual failure keeping prefill_layer red in the A5 nightly after pypto-lib#960 fixed the MoE window-epoch protocol — first-dispatch validation is stably green; only repeated dispatch dies.

Reproduce with (2 A5 cards):

PYPTO_BENCH=1 PYPTO_BENCH_ROUNDS=200 \
PTO2_RING_DEP_POOL=16384 PTO2_RING_TASK_WINDOW=16384 PTO2_RING_HEAP=1073741824 \
python models/deepseek_v4_pro/prefill_layer.py -p a5 -d <c0>,<c1> --layer-id 2

Repro case: models/deepseek_v4_pro/prefill_layer.py at pypto-lib main (any commit containing pypto-lib#960, e.g. 7a57215 or later; stock file, no local edits).

Built-in negative controls (same command, byte-identical otherwise): --layer-id 0 (SWA layer kind) and --layer-id 3 (HCA layer kind) both complete 200 rounds cleanly under identical conditions. --layer-id 2 selects the CSA layer kind and dies (observed at round 8 in the paired control run).

Reproduction environment:

Component	Version
pypto-lib	main with #960 (repro verified at the #960 merge commit content)
pypto	ed2aaa18 (branch: main; also reproduces on the nightly CI's newer pypto heads, 2026-08-13..08-17)
simpler	3165cc89 (detached; == pypto runtime gitlink)
ptoas	v0.57 (matches toolchain/versions.env)
pto-isa	83d01313 (matches runtime/pto_isa.pin)
CANN	9.2.0

Diagnosis: simpler — the terminal state is a device-runtime/memory-path wedge: cores parked at ptoas-inserted intra-core pipe wait_flags whose producing MTE2 TLOAD never completes (details below); with a small ring heap the same stall surfaces instead as orch_error_code=2 HEAP_RING_DEADLOCK. The involved kernels are provably victims (they run clean for 200+ rounds when the CSA kernel set is absent).

Description

Two-card repeated dispatch of the composed prefill layer (CSA attention kind + MoE cross-card dispatch/combine) intermittently stalls GM reads on AIC cores; the AICore watchdog then sweeps the device. Evidence chain, all from host-side observation plus build artifacts:

1. Trap dump decoding. Every dump's pc start values (0x…000278 for AIC cores, 0x…002f78 for AIV cores; low bits shift slightly across builds) are the persistent dispatcher entry points, not kernels — the ~75-84 cores parked there (current pc ≈ entry+0x3d8..0x43c) are idle collateral. The truly stuck cores' current PCs fall inside the uploaded ChipCallable image (single image, 92 children in the failing build).

2. Victim identification. Mapping those PCs through the image layout (code_addr = image_base + 0x38 + child_offset(i), offsets read via ChipCallable bindings; image base from the upload_chip_callable_buffer DEBUG log line) lands overwhelmingly in aic_kv_proj_matmul (a plain bf16 [T,7168]x[7168,512] split-K matmul with atomic Add GM accumulation), offsets +0x728..0x9f4, plus occasional aic_prefill_idx_qr_proj / aiv_prefill_idx_qr_rope / aiv_rope_cs.

3. Stuck instruction. Resolving those offsets through the kernel .o's .debug_line shows every parked core sits at a compiler-inserted intra-core pipe sync — wait_flag(PIPE_MTE2→PIPE_MTE1) / wait_flag(PIPE_M→PIPE_MTE1) immediately guarding a TLOAD consumer — i.e. the MTE2 pipe's GM read never completed. The dumps' mte error info / l1 error info fields are non-zero on these cores; vec/cube error info are clean.

4. Trigger isolation (layer-kind bisection, 200 rounds each, same devices/env):

   Runtime kernel set	Result
   SWA attention + MoE (--layer-id 0)	PASS (206 dispatches)
   HCA attention + MoE (--layer-id 3)	PASS
   CSA attention + MoE (--layer-id 2)	dies round ~8
   MoE standalone (l3_moe, nightly bench)	PASS (105 dispatches)
   CSA attention standalone (nightly bench)	PASS

   So the CSA-exclusive kernels (idx qr/hadamard/score, c4 softmax/pool, csa cache_write, compressors, inner-state) must run concurrently with cross-card MoE window traffic (remote_store + AtomicAdd notifies) to fire. aic_kv_proj_matmul itself is healthy without the CSA set present.

5. Ring-heap interaction (amplifier, not cause). scope_stats shows one dispatch flows ~1.2 GB through the ring heap, with the reclaim tail pinned at 0 for the whole first request (top reaches 696 MB before the first reclaim) — at the CI's PTO2_RING_HEAP=1GiB the ring must wrap intra-dispatch with minimal slack. Consequences, all verified:

   • 256 MiB heap → deterministic first-dispatch death with orch_error_code=2 HEAP_RING_DEADLOCK ("ran out of both task slots and heap bytes");
   • 1 GiB → dies rounds 5-14; at least one death recorded orch_error_code=2 before the core traps;
   • 2 GiB / 4 GiB (effective values confirmed via the resolve_arena_sizing "Ring buffer sizes" log) → still dies (rounds 62 / 7 / 35 observed) with pure 507015 and no orchestration error — the heap margin only changes how the underlying stall is reported.

6. Post-mortem device state. After the trap ("force reset will follow in finalize"), the pair often rejects the next run (chip worker(s) did not become ready within 300.0s) or produces corrupt validation output, until some later recovery — a secondary robustness gap worth its own attention.

Related: pypto-lib#858 (VEC UB out-of-bounds in the same CSA/sparse-attn family, surfaces as S1 stall), pypto-lib#696, pypto-lib#960 (protocol fix that exposed this residual), simpler#1816 (different signature: 507018 paged-attention stalls).

Steps to Reproduce

1. 2-card A5 host, environment pinned as in the Background table (stock pypto-lib main, pypto pins for simpler/ptoas/pto-isa).
2. Run the command from the Background (CSA layer kind, 200 bench rounds, CI-default ring env).
3. Typically dies within rounds 5–62; repeat once if a run survives. --layer-id 0 / 3 under the same command are the negative controls.
4. Optional: set PTO2_RING_HEAP=268435456 for a deterministic first-dispatch HEAP_RING_DEADLOCK, or PYPTO_RUNTIME_LOG=debug on a validate-only run to log the image base for PC mapping.

Expected Behavior

The CSA-layer benchmark completes all rounds like the SWA/HCA layer kinds do (first-dispatch validation already passes in all cases), and a failed run leaves the device pair usable.

Actual Behavior

Random-round device wedge: benchmark rounds interrupted: RuntimeError: ... chip_process dev=N: RuntimeError: run failed with code 507015 (or code -2 with orch_error_code=2 HEAP_RING_DEADLOCK at 1 GiB heap and below); trap dump shows dozens of cores on both dies with subErrType 0x4, the stuck ones at MTE2-guarding wait_flags inside aic_kv_proj_matmul et al.; the device pair often needs manual recovery afterwards.

---

Platform: a5 (Ascend 950 hardware)
Runtime Variant: tensormap_and_ringbuffer ("device orchestration mode")
Git Commit ID: 3165cc8
CANN Version: 9.2.0
Host Platform: Linux (x86_64)

[lwDavid]

Deep-dive update: root cause narrowed to an on-die stall in the dense CSA mixed-kernel phase; every runtime software layer above it exonerated by intervention experiments

Summary of a full day of on-device root-causing. Each candidate mechanism was tested by modifying the system and re-running the repro, not just by code reading. All experiments used the issue's repro (prefill_layer.py --layer-id 2, 2 cards, CI env) plus progressively stripped diagnostic variants (local uncommitted patches, described below).

Exclusion ladder (each row = a device experiment)

#	Hypothesis	Intervention	Result	Verdict
1	Corrupted index → unmapped-VA MTE access	Upper-bound clamps (pl.min) added to all 12 data-dependent GM address sites in the CSA+MoE kernels (gathers, cache writes, MoE routes/counts). Validation stayed green (clamps are semantics-neutral).	Died ×2 (507015, 84-core sweep)	Excluded
2	Cross-card payload DMA (7 KiB tput rows)	All routes forced to local experts (eid -> my_rank*N_LOCAL + eid%N_LOCAL); structural per-epoch notifies kept	Died (trap sweep)	Excluded
3	Any cross-card traffic at all	Additionally suppressed every cross-rank notify/wait/meta store (protocol degraded consistently on both sides)	Died (both ranks, 168 cores)	Excluded
4	MoE window ops (local TPUT/TSTORE, scalar atomics, TWait polling)	MoE dispatch/combine cut out entirely — shared expert only, zero DistributedTensor ops	Died, now deterministically: both independent ranks trap in the same second, during dispatch 2, ~12 s wall-clock, 3/3 runs	Excluded (and the failure got easier to hit)
5	Ring-heap wrap / use-after-reclaim	Mechanistic: allocator cursors are re-zeroed every dispatch (SmReset -> task_allocator.init), so ≥2 GiB/ring is wrap-free by construction; deaths persist at 2 GiB and 4 GiB	—	Excluded
6	In-flight throttling as mitigation	PTO2_RING_TASK_WINDOW=2048 (256 hits FLOW_CONTROL_DEADLOCK(-3), a config floor)	Died	Window depth does not bound running concurrency (cores do)
7	Scheduler completion-reaping (see below)	decide_slot_transition Case 3.1 hardened to complete-and-promote instead of skip; rebuilt AICPU; re-ran	Died identically	Case 3.1 is an amplifier in reporting, not the origin

The stripped variant (row 4) is the most useful new repro property: no communication of any kind, and the wedge becomes deterministic at dispatch 2 — two fully independent cards die in the same second. Whatever the mechanism is, it lives inside one die's compute/memory path plus the AICPU scheduler, and it needs one prior dispatch of state/warm-up plus the dense CSA task mix (SWA/HCA layer kinds and every component-in-isolation run clean for 200 rounds).

Scheduler's own snapshot of the wedge (new evidence)

With device logs at INFO, the S1 stall snapshot (scheduler_cold_path.cpp CLUSTER dump) at the freeze shows, simultaneously:

CLUSTER cluster_id=7  aic=core7(busy kernel=25 task=...379 cond_reg_state=fin ANOMALY) aiv0=core42(busy kernel=26 task=...379 cond_reg_state=fin ANOMALY) aiv1=core43(busy kernel=3 task=...806 cond_reg_state=fin ANOMALY)
CLUSTER cluster_id=1  aic=core1(busy kernel=12 task=...365 cond_reg_state=ack)         aiv0=core30(busy kernel=27 task=...926 cond_reg_state=ack) ...

i.e. several cores have posted FIN that the scheduler never consumed (fin ANOMALY) while the 4 scheduler threads spin idle (idle_iterations ~1.3–1.6 M), completions frozen at ~210/1356, running=5 ready=0 waiting=1141, and the same stuck task id reproduces across runs (ring 3, local id 67 = the first big qk_pv mixed task of dispatch 2; also ids 38/62 in sibling runs — always the early mixed-kernel phase). The unreaped FINs are explained by decide_slot_transition Case 3.1 (scheduler_completion.cpp: running FIN + non-gated pending staged → "skip (transient)", with no timeout): when a core's persistent dispatcher stalls picking up its staged pending task, the running task's FIN is withheld forever. Hardening 3.1 (complete + promote, like 3.3) did not unwedge the machine (row 7), which pins the origin one level down: the cores themselves stop making progress — trapped cores sit at ptoas-inserted wait_flag(PIPE_MTE2/M→PIPE_MTE1) inside ordinary kernels (kv_proj_matmul, idx_qr_hadamard_quant, rope_cs, qk_pv…) and at dispatcher task-fetch, i.e. GM/L1 supply to a whole group of clusters stalls and never recovers. Suggest still fixing 3.1 (unbounded "transient" is a diagnosability landmine), but it is not the root.

Dispatch-layer bookkeeping verified clean (DEBUG dispatch records)

From a full per-subtask dispatch log of a dying run (16 212 records): every mixed task's aic/aiv halves are dispatched with identical block sets (no pairing skew, block_idx consistent per side), block claims are unique (no double dispatch within a round; per-core reg_task_id sequence is continuous across rounds so no cond-reg aliasing), and the wedge onset is a cliff: all cores receive their last dispatch within ~7 ms early in dispatch 2 and nothing completes afterwards.

Static audit of the mixed-kernel FFTS sync protocol (all 6 aic/aiv pairs)

Set/wait counts are provably balanced on every reachable path (including destructor drains) for qr_hadamard_quant, qr_proj, idx_score, qk_pv, gate, weights_proj. Two hardening gaps worth fixing even though we could not prove either fires here:

1. prefill_idx_score derives its sync-bearing loop bound from two independent uninvalidated GM scalar reads of the same tensor cell (one per core half) — any coherency divergence yields cube-pushes ≠ vec-pops and a deterministic pair deadlock. The bound should be computed once (orchestration-level read passed as a scalar arg) or the read made coherent.
2. All six mixed kernels share physical FFTS flag ids 0–3 per cluster; a single diverged instance poisons the counters for the next mixed kernel on that cluster, which will present as the innocent epicenter.

Separate confirmed defect found on the way (comm path; independent of this wedge)

DWARF-level inspection of the compiled MoE kernels shows the writer-side cache maintenance after cross-card stores covers one 64 B line out of 112 (dispatch payload) / 224 (combine payload): pto.cmo.cacheinvalid <peer view> single_cache_line lowers to a single dcci(base), while the notify's own flag line gets a full dcci+st_atomic+dcci+dsb push and the receiver reliably sees the flag. The peer can therefore observe the signal before the payload lines are written back — and the 111/223 dirty peer-addressed lines are later evicted at arbitrary times, outside every protocol fence. The two small TSTOREs (aux/route) additionally issue their dcci with no pipe wait after the MTE3 store. A pypto-codegen candidate fix (per-64B-line dcci loop ordered behind PIPE_ALL, verified in the regenerated .pto/.cpp and validate-green on device) exists and can be PR'd; ptoas' single-line lowering of partition-view cacheinvalid is the underlying limitation. This defect produces stale-data reads (silent wrong values), consistent with the intermittent non-finite-value validation flakes seen in this model family, but eliminating it did not stop the trap-class wedge (arm B died as well).

Where this leaves the issue

• The wedge reproduces with zero communication, deterministically, at dispatch 2 of the dense CSA mixed-kernel phase, on both cards independently — the trigger is on-die (compute/memory-path service to a group of clusters stops; trap dumps' error-info words carry no latched fault, error code = 0 / subErrType 0x4 timeout).
• All simpler-level protocols audited/tested above are sound for this failure; the remaining depth (BIU/L2/translation state at the moment of stall) needs privileged tooling — msnpureport/bbox export requires root and could not be captured from an unprivileged account.
• Actionable simpler-side follow-ups regardless: (a) bound Case 3.1's transient skip and surface it in the stall snapshot; (b) extend the (very useful) CLUSTER snapshot with per-cluster FFTS counter values and the pending slot's doorbell/pickup state; (c) the two FFTS hardening items above.

Full arm-by-arm logs and the diagnostic patches are available on request.