Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/rules/running-onboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ for the signature that actually fired:**
| `FATAL: Task Allocator Deadlock` / `FATAL: Dependency Pool Deadlock` / `FATAL: Fanin Spill Pool Deadlock` | an allocator or pool could not reclaim enough space | This header identifies the blocked resource, not the root cause. Classify it using the structural/timeout line that follows. |
| `Provable head-of-line deadlock` | **proven open-scope structural deadlock** | TRB only: the reclaim head is the oldest task owned by an open scope on that ring. The blocked orchestrator cannot end that scope, so the head cannot become consumed. On A5, classification waits for at least 10 ms without reclaim progress and an exact-watermark publication acknowledgment. |
| `No reclaim progress for ~500 ms` / `cannot reclaim space after ~500 ms` | allocator/pool **reclaim timeout** | TRB only — the 500ms backstop (`ALLOC_DEADLOCK_TIMEOUT_CYCLES`) lives in its reclaiming allocators. It proves prolonged lack of reclaim progress, not why progress stopped; check capacity, the dumped head, consumers and scheduler state. |
| `Task Window Exhausted` / `Fanin Capacity Exhausted` / `TensorMap Entry Pool Exhausted` | **capacity**, HBG only | HBG is whole-graph-resident and reclaims nothing mid-run, so this is a sizing verdict reached immediately, not a stall. The line carries used/capacity and the requested amount; raise `runtime_env.ring_task_window` / `CHIP_TENSORMAP_POOL_SIZE` or shrink the graph. Inline fanin is hard-capped at `CHIP_MAX_FANIN=128`; HBG has no dependency spill pool. HBG's graph heap has no knob and cannot exhaust: it is committed after orchestration at the measured size, so a graph too large for the device fails host-side at that commit. |
| `Graph Too Large` / `Fanin Capacity Exhausted` / `TensorMap Entry Pool Exhausted` | **capacity**, HBG only | HBG is whole-graph-resident and reclaims nothing mid-run, so this is a sizing verdict reached immediately, not a stall. The line carries used/capacity and the requested amount; raise `runtime_env.ring_task_window` (any positive count — HBG indexes slots by task id and masks with nothing) / `CHIP_TENSORMAP_POOL_SIZE`, or shrink the graph. Inline fanin is hard-capped at `CHIP_MAX_FANIN=128`; HBG has no dependency spill pool. HBG's graph heap has no knob and cannot exhaust: it is committed after orchestration at the measured size, so a graph too large for the device fails host-side at that commit. |
| `Timeout (N cycles): producer/consumers ...` | **SPIN** wait on a specific producer/consumer | `runtime_core.cpp`. |
| `HandleTaskTimeout` / `kill aicpu-sd` | **OS op-execute timeout** | STARS/tsdaemon, default 45s (`PLATFORM_OP_EXECUTE_TIMEOUT_US`). **A 45s kill ≠ deadlock** — the op was merely long or stalled. Raise this constant to measure true on-device duration. |
| `log_stall_diagnostics` (cores idle + tasks `state=WAIT fanin 0/N` + `completed` frozen) | **forward-progress stall** | No dedicated detector — intermittent races (often contention-triggered) land here and are reaped only by the op-timeout above. |
Expand Down
6 changes: 3 additions & 3 deletions docs/troubleshooting/device-error-codes/capacity.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ resource and determines how strong that diagnosis is:
- `No reclaim progress for ~500 ms` or `cannot reclaim space after ~500 ms` is
the backstop. It proves that reclaim remained stalled, but not whether the root
cause is undersizing, a stuck consumer, or a stalled scheduler.
- `Task Window Exhausted` / `Fanin Capacity Exhausted` / `TensorMap Entry Pool
- `Graph Too Large` / `Fanin Capacity Exhausted` / `TensorMap Entry Pool
Exhausted` is HBG, and it is unambiguous: that runtime builds a
whole-graph-resident image on the host, so the graph simply does not fit. These
checks return immediately; there is no concurrent scheduler progress that could
Expand All @@ -42,10 +42,10 @@ that trips the code.

| Runtime | Bottleneck resource | Code | Fix |
| ------- | ------------------- | ---- | --- |
| HBG | task window | 3 | raise `runtime_env.ring_task_window`, or shrink the graph |
| HBG | task count | 3 | raise `runtime_env.ring_task_window` (any positive count), or shrink the graph |
| HBG | inline fanin | 4 | reduce distinct producers to `CHIP_MAX_FANIN` (currently 128) or less; HBG has no dependency spill pool |
| HBG | TensorMap entries | 11 | increase `CHIP_TENSORMAP_POOL_SIZE`, or reduce registered outputs |
| TRB | open-scope task window | 1 or 3 | raise `runtime_env.ring_task_window`, split the scope, or diagnose stalled reclaim |
| TRB | open-scope task window | 1 or 3 | raise `runtime_env.ring_task_window` (a power of two, >= 4), split the scope, or diagnose stalled reclaim |
| TRB | heap | 2 | raise `runtime_env.ring_heap`, shrink allocations, or diagnose stalled reclaim |
| TRB | dependency pool | 4 | raise `runtime_env.ring_dep_pool`, cut fanin, or diagnose stalled reclaim |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,14 +108,6 @@ class TestDeepseekV4FlashDecodeHostBuildGraph(SceneTestCase):
"config": {
"device_count": N_RANKS,
"num_sub_workers": 0,
# Ring sizing matches the TMR case: both runtimes now build the
# same static per-expert tile grid, so both allocate tile scratch
# for all 32 experts of every MoE layer.
"runtime_env": {
"ring_task_window": 16384,
"ring_heap": 2 << 30,
"ring_dep_pool": 16384,
},
},
"params": {"seed": 1234},
}
Expand Down
11 changes: 6 additions & 5 deletions src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) {
// and every cross-task reference it wrote is an offset from its own block, so
// the SM/arena this thread sees need no address fixup. This thread attaches
// the prebuilt arena, points the SM
// handle's ring-header pointers at the device SM WITHOUT resetting the
// handle's task-header pointers at the device SM WITHOUT resetting the
// host-populated data, hands the host-computed task count to the scheduler,
// and releases the other threads. It then falls through and schedules its own
// cores like every other thread — host_build_graph has no device-side
Expand Down Expand Up @@ -264,9 +264,10 @@ int32_t AicpuExecutor::run(Runtime *runtime) {

void *sm_ptr = runtime->get_gm_sm_ptr();
// The image the host shipped is pitched to the submitted task count,
// not to the ring capacity, and the device region holds exactly that
// image — so its size comes from the same pitch. attach_populated
// rejects a pitch outside (0, capacity] and a region too small for it.
// not to the count the table was dimensioned for, and the device region
// holds exactly that image — so its size comes from the same pitch.
// attach_populated rejects a pitch outside (0, task_capacity] and a
// region too small for it.
const uint64_t live_slots = sm_layout::live_slot_pitch(static_cast<uint64_t>(runtime->host_total_tasks));
const uint64_t sm_size = runtime->sm_image_bytes;
// sm_handle and the scheduler state are the device-only zone: their
Expand All @@ -276,7 +277,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) {
// attach_populated.
memset(rt->sm_handle, 0, sizeof(*rt->sm_handle));
if (!rt->sm_handle->attach_populated(
sm_ptr, sm_size, rt->prebuilt_layout.task_window_size, live_slots, runtime->sm_image_bytes
sm_ptr, sm_size, rt->prebuilt_layout.task_capacity, live_slots, runtime->sm_image_bytes
)) {
LOG_ERROR("Thread %d: host-orch: sm_handle->attach_populated failed", thread_idx);
rt = nullptr;
Expand Down
47 changes: 28 additions & 19 deletions src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ scheduler reads, the count of tasks completed inline during orchestration, is a
scalar `rt_orchestration_done` publishes into the runtime header.

**Why the scheduler state is device-written.** `SchedulerState` holds no
per-run content: `sm_header` and the ring pointer derive from a pooled SM base,
per-run content: `sm_header` and the task-header pointer derive from a pooled SM base,
queue capacities are compile-time constants, hbg never advances
`last_task_alive`, and it has no host-side entry point at all. So the host would
only be writing an initialization pattern — 203,392 bytes
Expand Down Expand Up @@ -172,52 +172,61 @@ pins the invariant that makes that safe.

### 3.2 Bounded H2D Upload

The shared-memory mirror is sized to ring capacity (task window) but a run only
The shared-memory mirror is dimensioned for the run's configured task count
(`runtime_env.ring_task_window`, default `CHIP_DEFAULT_GRAPH_TASKS`) but a run only
writes `[0, total_tasks)`, and the device boots scheduler-only and reads no SM slot
past `total_tasks`. So the SM H2D shipped each run is bounded, not capacity-sized —
the contract that keeps `bind` proportional to the workload.

The header is zeroed on the host; `descriptors`, `payloads`, `slot_states` and
`completion_flags` are each written per task at submit and H2D-uploaded bounded to
`[0, total_tasks)`. Per-slot reset is init-on-write in `orch::prepare_task` as each
slot is claimed — there is no window-wide reset. The four segments travel as four
copies rather than one because ring-sized tails separate their live prefixes.
`completion_flags` are each written per task at submit. Per-slot reset is
init-on-write in `orch::prepare_task` as each slot is claimed — there is no
table-wide reset. In the mirror those four live prefixes are a full reservation
apart, so `compact_live_image` restacks them (plus the three argument pools) into
an image pitched to `total_tasks`, where they are contiguous and travel as **one**
`copy_to_device`. The device attaches with the same pitch.

## 4. Whole-Graph Capacity

The runtime uses one task ring, one graph heap, and one TensorMap pool. They are
The runtime uses one task table, one graph heap, and one TensorMap pool. They are
capacity-bounded storage, not streaming flow-control buffers:

- the task ring and the graph heap are forward-only bump allocators;
- the task table and the graph heap are forward-only bump allocators;
- task slots and heap bytes are never recycled mid-run; and
- TensorMap entries are held for the whole run.

There is no reclaim channel from the scheduler back to the allocator, so the
allocators carry no reclaim pointer and no back-pressure wait.
allocators carry no reclaim pointer and no back-pressure wait. A task id is
therefore also its slot index: ids run `0..capacity-1`, never wrap, and every
segment is indexed by the id directly — there is no slot mask, so the capacity need
not be a power of two.

`completed_watermark` records the contiguous prefix of completed device tasks.
It supports completion/consumer metadata only; it does not reclaim the task ring
or heap.
It supports completion/consumer metadata only; it reclaims neither task slots
nor heap.

There is no post-run sweep that makes graph space reusable. Runtime destruction
releases the complete arena, and the next run starts from a newly initialized
image.

### 4.1 Allocation Failure

The graph must fit the configured task window, the fanin capacity, and the
TensorMap pool. Because nothing is reclaimed, a request that does not fit can
never become satisfiable — the allocator names the exhausted resource and fails
on the spot. There is no wait and no timeout.
The graph must fit the configured task count, the fanin capacity, and the TensorMap
pool. The task count comes from `runtime_env.ring_task_window` (default
`CHIP_DEFAULT_GRAPH_TASKS`); the host mirror is allocated at that size and
committed by first touch, so a run pays only for the slots it writes. Because
nothing is reclaimed, a request that does not fit can never become satisfiable —
the allocator names the exhausted resource and fails on the spot. There is no wait
and no timeout.

Representative allocator output is:

```text
FATAL: Task Window Exhausted!
FATAL: Graph Too Large!
The whole graph must fit at once; nothing is reclaimed mid-run.
Task window: used=.../...
Graph heap: used=.../..., available=...
Requested: ... bytes + 1 task slot
Tasks: used=.../...
Graph heap: used=.../..., available=...
Requested: ... bytes + 1 task slot
```

This is host-orchestration logging. The allocator records the corresponding
Expand Down
6 changes: 3 additions & 3 deletions src/a2a3/runtime/host_build_graph/docs/SCALAR_DATA_ACCESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,11 @@ it derives for itself, from a heap block whose prior contents it never reads.

Before a wait slot is used, the runtime verifies:

- the task ID is valid and belongs to the single HBG ring;
- the selected ring slot has a bound task descriptor; and
- the task ID is valid and carries ring 0, the only ring HBG places tasks on;
- the task table slot that ID indexes has a bound task descriptor; and
- the descriptor's full task ID matches the tensor's owner/producer ID.

The full-ID check prevents a masked ring-slot lookup from aliasing an unused or
The full-ID check prevents a slot lookup from aliasing an unused or
different task. A failure latches `SIMPLER_ERROR_INVALID_ARGS` and the run returns
status `-5`; reads return zero and writes stop only after that fatal status is
recorded.
Expand Down
6 changes: 3 additions & 3 deletions src/a2a3/runtime/host_build_graph/docs/SUBMIT_BY_CLUSTER.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,9 @@ the run.

`host_build_graph` is whole-graph-resident. Task slots, heap bytes, fanin IDs,
and TensorMap entries are not reclaimed while the graph is executing. The graph
must fit the configured task window and TensorMap pool before launch; its heap
takes no configuration, since the device region is committed after orchestration
at the size the graph turned out to need.
must fit the configured task count (`runtime_env.ring_task_window`) and the
TensorMap pool before launch; the heap is not a third capacity, since its device
region is committed after orchestration at the size the graph turned out to need.

## Validation

Expand Down
Loading
Loading