diff --git a/.claude/rules/running-onboard.md b/.claude/rules/running-onboard.md index 78fee2ffea..83fab1427e 100644 --- a/.claude/rules/running-onboard.md +++ b/.claude/rules/running-onboard.md @@ -175,7 +175,7 @@ for the signature that actually fired:** | `FATAL: Task Allocator Deadlock` / `FATAL: Dependency Pool Deadlock` / `FATAL: Fanin Spill Pool Deadlock` | an allocator or pool could not reclaim enough space | This header identifies the blocked resource, not the root cause. Classify it using the structural/timeout line that follows. | | `Provable head-of-line deadlock` | **proven open-scope structural deadlock** | TRB only: the reclaim head is the oldest task owned by an open scope on that ring. The blocked orchestrator cannot end that scope, so the head cannot become consumed. On A5, classification waits for at least 10 ms without reclaim progress and an exact-watermark publication acknowledgment. | | `No reclaim progress for ~500 ms` / `cannot reclaim space after ~500 ms` | allocator/pool **reclaim timeout** | TRB only — the 500ms backstop (`ALLOC_DEADLOCK_TIMEOUT_CYCLES`) lives in its reclaiming allocators. It proves prolonged lack of reclaim progress, not why progress stopped; check capacity, the dumped head, consumers and scheduler state. | -| `Task Window Exhausted` / `Fanin Capacity Exhausted` / `TensorMap Entry Pool Exhausted` | **capacity**, HBG only | HBG is whole-graph-resident and reclaims nothing mid-run, so this is a sizing verdict reached immediately, not a stall. The line carries used/capacity and the requested amount; raise `runtime_env.ring_task_window` / `CHIP_TENSORMAP_POOL_SIZE` or shrink the graph. Inline fanin is hard-capped at `CHIP_MAX_FANIN=128`; HBG has no dependency spill pool. HBG's graph heap has no knob and cannot exhaust: it is committed after orchestration at the measured size, so a graph too large for the device fails host-side at that commit. | +| `Graph Too Large` / `Fanin Capacity Exhausted` / `TensorMap Entry Pool Exhausted` | **capacity**, HBG only | HBG is whole-graph-resident and reclaims nothing mid-run, so this is a sizing verdict reached immediately, not a stall. The line carries used/capacity and the requested amount; raise `runtime_env.ring_task_window` (any positive count — HBG indexes slots by task id and masks with nothing) / `CHIP_TENSORMAP_POOL_SIZE`, or shrink the graph. Inline fanin is hard-capped at `CHIP_MAX_FANIN=128`; HBG has no dependency spill pool. HBG's graph heap has no knob and cannot exhaust: it is committed after orchestration at the measured size, so a graph too large for the device fails host-side at that commit. | | `Timeout (N cycles): producer/consumers ...` | **SPIN** wait on a specific producer/consumer | `runtime_core.cpp`. | | `HandleTaskTimeout` / `kill aicpu-sd` | **OS op-execute timeout** | STARS/tsdaemon, default 45s (`PLATFORM_OP_EXECUTE_TIMEOUT_US`). **A 45s kill ≠ deadlock** — the op was merely long or stalled. Raise this constant to measure true on-device duration. | | `log_stall_diagnostics` (cores idle + tasks `state=WAIT fanin 0/N` + `completed` frozen) | **forward-progress stall** | No dedicated detector — intermittent races (often contention-triggered) land here and are reaped only by the op-timeout above. | diff --git a/docs/troubleshooting/device-error-codes/capacity.md b/docs/troubleshooting/device-error-codes/capacity.md index 834c9af9c9..b4c751ad87 100644 --- a/docs/troubleshooting/device-error-codes/capacity.md +++ b/docs/troubleshooting/device-error-codes/capacity.md @@ -15,7 +15,7 @@ resource and determines how strong that diagnosis is: - `No reclaim progress for ~500 ms` or `cannot reclaim space after ~500 ms` is the backstop. It proves that reclaim remained stalled, but not whether the root cause is undersizing, a stuck consumer, or a stalled scheduler. -- `Task Window Exhausted` / `Fanin Capacity Exhausted` / `TensorMap Entry Pool +- `Graph Too Large` / `Fanin Capacity Exhausted` / `TensorMap Entry Pool Exhausted` is HBG, and it is unambiguous: that runtime builds a whole-graph-resident image on the host, so the graph simply does not fit. These checks return immediately; there is no concurrent scheduler progress that could @@ -42,10 +42,10 @@ that trips the code. | Runtime | Bottleneck resource | Code | Fix | | ------- | ------------------- | ---- | --- | -| HBG | task window | 3 | raise `runtime_env.ring_task_window`, or shrink the graph | +| HBG | task count | 3 | raise `runtime_env.ring_task_window` (any positive count), or shrink the graph | | HBG | inline fanin | 4 | reduce distinct producers to `CHIP_MAX_FANIN` (currently 128) or less; HBG has no dependency spill pool | | HBG | TensorMap entries | 11 | increase `CHIP_TENSORMAP_POOL_SIZE`, or reduce registered outputs | -| TRB | open-scope task window | 1 or 3 | raise `runtime_env.ring_task_window`, split the scope, or diagnose stalled reclaim | +| TRB | open-scope task window | 1 or 3 | raise `runtime_env.ring_task_window` (a power of two, >= 4), split the scope, or diagnose stalled reclaim | | TRB | heap | 2 | raise `runtime_env.ring_heap`, shrink allocations, or diagnose stalled reclaim | | TRB | dependency pool | 4 | raise `runtime_env.ring_dep_pool`, cut fanin, or diagnose stalled reclaim | diff --git a/examples/a2a3/host_build_graph/deepseek_v4_flash_decode/test_deepseek_v4_flash_decode.py b/examples/a2a3/host_build_graph/deepseek_v4_flash_decode/test_deepseek_v4_flash_decode.py index a7fdcc5377..ee138cb139 100644 --- a/examples/a2a3/host_build_graph/deepseek_v4_flash_decode/test_deepseek_v4_flash_decode.py +++ b/examples/a2a3/host_build_graph/deepseek_v4_flash_decode/test_deepseek_v4_flash_decode.py @@ -108,14 +108,6 @@ class TestDeepseekV4FlashDecodeHostBuildGraph(SceneTestCase): "config": { "device_count": N_RANKS, "num_sub_workers": 0, - # Ring sizing matches the TMR case: both runtimes now build the - # same static per-expert tile grid, so both allocate tile scratch - # for all 32 experts of every MoE layer. - "runtime_env": { - "ring_task_window": 16384, - "ring_heap": 2 << 30, - "ring_dep_pool": 16384, - }, }, "params": {"seed": 1234}, } diff --git a/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp b/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp index 15c2b6a72f..d3604b762f 100644 --- a/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp +++ b/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp @@ -231,7 +231,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) { // and every cross-task reference it wrote is an offset from its own block, so // the SM/arena this thread sees need no address fixup. This thread attaches // the prebuilt arena, points the SM - // handle's ring-header pointers at the device SM WITHOUT resetting the + // handle's task-header pointers at the device SM WITHOUT resetting the // host-populated data, hands the host-computed task count to the scheduler, // and releases the other threads. It then falls through and schedules its own // cores like every other thread — host_build_graph has no device-side @@ -264,9 +264,10 @@ int32_t AicpuExecutor::run(Runtime *runtime) { void *sm_ptr = runtime->get_gm_sm_ptr(); // The image the host shipped is pitched to the submitted task count, - // not to the ring capacity, and the device region holds exactly that - // image — so its size comes from the same pitch. attach_populated - // rejects a pitch outside (0, capacity] and a region too small for it. + // not to the count the table was dimensioned for, and the device region + // holds exactly that image — so its size comes from the same pitch. + // attach_populated rejects a pitch outside (0, task_capacity] and a + // region too small for it. const uint64_t live_slots = sm_layout::live_slot_pitch(static_cast(runtime->host_total_tasks)); const uint64_t sm_size = runtime->sm_image_bytes; // sm_handle and the scheduler state are the device-only zone: their @@ -276,7 +277,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) { // attach_populated. memset(rt->sm_handle, 0, sizeof(*rt->sm_handle)); if (!rt->sm_handle->attach_populated( - sm_ptr, sm_size, rt->prebuilt_layout.task_window_size, live_slots, runtime->sm_image_bytes + sm_ptr, sm_size, rt->prebuilt_layout.task_capacity, live_slots, runtime->sm_image_bytes )) { LOG_ERROR("Thread %d: host-orch: sm_handle->attach_populated failed", thread_idx); rt = nullptr; diff --git a/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md b/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md index 7ca208cabd..e3d16ecfe7 100644 --- a/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md +++ b/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md @@ -123,7 +123,7 @@ scheduler reads, the count of tasks completed inline during orchestration, is a scalar `rt_orchestration_done` publishes into the runtime header. **Why the scheduler state is device-written.** `SchedulerState` holds no -per-run content: `sm_header` and the ring pointer derive from a pooled SM base, +per-run content: `sm_header` and the task-header pointer derive from a pooled SM base, queue capacities are compile-time constants, hbg never advances `last_task_alive`, and it has no host-side entry point at all. So the host would only be writing an initialization pattern — 203,392 bytes @@ -172,32 +172,38 @@ pins the invariant that makes that safe. ### 3.2 Bounded H2D Upload -The shared-memory mirror is sized to ring capacity (task window) but a run only +The shared-memory mirror is dimensioned for the run's configured task count +(`runtime_env.ring_task_window`, default `CHIP_DEFAULT_GRAPH_TASKS`) but a run only writes `[0, total_tasks)`, and the device boots scheduler-only and reads no SM slot past `total_tasks`. So the SM H2D shipped each run is bounded, not capacity-sized — the contract that keeps `bind` proportional to the workload. The header is zeroed on the host; `descriptors`, `payloads`, `slot_states` and -`completion_flags` are each written per task at submit and H2D-uploaded bounded to -`[0, total_tasks)`. Per-slot reset is init-on-write in `orch::prepare_task` as each -slot is claimed — there is no window-wide reset. The four segments travel as four -copies rather than one because ring-sized tails separate their live prefixes. +`completion_flags` are each written per task at submit. Per-slot reset is +init-on-write in `orch::prepare_task` as each slot is claimed — there is no +table-wide reset. In the mirror those four live prefixes are a full reservation +apart, so `compact_live_image` restacks them (plus the three argument pools) into +an image pitched to `total_tasks`, where they are contiguous and travel as **one** +`copy_to_device`. The device attaches with the same pitch. ## 4. Whole-Graph Capacity -The runtime uses one task ring, one graph heap, and one TensorMap pool. They are +The runtime uses one task table, one graph heap, and one TensorMap pool. They are capacity-bounded storage, not streaming flow-control buffers: -- the task ring and the graph heap are forward-only bump allocators; +- the task table and the graph heap are forward-only bump allocators; - task slots and heap bytes are never recycled mid-run; and - TensorMap entries are held for the whole run. There is no reclaim channel from the scheduler back to the allocator, so the -allocators carry no reclaim pointer and no back-pressure wait. +allocators carry no reclaim pointer and no back-pressure wait. A task id is +therefore also its slot index: ids run `0..capacity-1`, never wrap, and every +segment is indexed by the id directly — there is no slot mask, so the capacity need +not be a power of two. `completed_watermark` records the contiguous prefix of completed device tasks. -It supports completion/consumer metadata only; it does not reclaim the task ring -or heap. +It supports completion/consumer metadata only; it reclaims neither task slots +nor heap. There is no post-run sweep that makes graph space reusable. Runtime destruction releases the complete arena, and the next run starts from a newly initialized @@ -205,19 +211,22 @@ image. ### 4.1 Allocation Failure -The graph must fit the configured task window, the fanin capacity, and the -TensorMap pool. Because nothing is reclaimed, a request that does not fit can -never become satisfiable — the allocator names the exhausted resource and fails -on the spot. There is no wait and no timeout. +The graph must fit the configured task count, the fanin capacity, and the TensorMap +pool. The task count comes from `runtime_env.ring_task_window` (default +`CHIP_DEFAULT_GRAPH_TASKS`); the host mirror is allocated at that size and +committed by first touch, so a run pays only for the slots it writes. Because +nothing is reclaimed, a request that does not fit can never become satisfiable — +the allocator names the exhausted resource and fails on the spot. There is no wait +and no timeout. Representative allocator output is: ```text -FATAL: Task Window Exhausted! +FATAL: Graph Too Large! The whole graph must fit at once; nothing is reclaimed mid-run. - Task window: used=.../... - Graph heap: used=.../..., available=... - Requested: ... bytes + 1 task slot + Tasks: used=.../... + Graph heap: used=.../..., available=... + Requested: ... bytes + 1 task slot ``` This is host-orchestration logging. The allocator records the corresponding diff --git a/src/a2a3/runtime/host_build_graph/docs/SCALAR_DATA_ACCESS.md b/src/a2a3/runtime/host_build_graph/docs/SCALAR_DATA_ACCESS.md index 0a436f7cca..39daa3d8d4 100644 --- a/src/a2a3/runtime/host_build_graph/docs/SCALAR_DATA_ACCESS.md +++ b/src/a2a3/runtime/host_build_graph/docs/SCALAR_DATA_ACCESS.md @@ -70,11 +70,11 @@ it derives for itself, from a heap block whose prior contents it never reads. Before a wait slot is used, the runtime verifies: -- the task ID is valid and belongs to the single HBG ring; -- the selected ring slot has a bound task descriptor; and +- the task ID is valid and carries ring 0, the only ring HBG places tasks on; +- the task table slot that ID indexes has a bound task descriptor; and - the descriptor's full task ID matches the tensor's owner/producer ID. -The full-ID check prevents a masked ring-slot lookup from aliasing an unused or +The full-ID check prevents a slot lookup from aliasing an unused or different task. A failure latches `SIMPLER_ERROR_INVALID_ARGS` and the run returns status `-5`; reads return zero and writes stop only after that fatal status is recorded. diff --git a/src/a2a3/runtime/host_build_graph/docs/SUBMIT_BY_CLUSTER.md b/src/a2a3/runtime/host_build_graph/docs/SUBMIT_BY_CLUSTER.md index 0839f42e9f..28835d7a59 100644 --- a/src/a2a3/runtime/host_build_graph/docs/SUBMIT_BY_CLUSTER.md +++ b/src/a2a3/runtime/host_build_graph/docs/SUBMIT_BY_CLUSTER.md @@ -111,9 +111,9 @@ the run. `host_build_graph` is whole-graph-resident. Task slots, heap bytes, fanin IDs, and TensorMap entries are not reclaimed while the graph is executing. The graph -must fit the configured task window and TensorMap pool before launch; its heap -takes no configuration, since the device region is committed after orchestration -at the size the graph turned out to need. +must fit the configured task count (`runtime_env.ring_task_window`) and the +TensorMap pool before launch; the heap is not a third capacity, since its device +region is committed after orchestration at the size the graph turned out to need. ## Validation diff --git a/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp b/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp index bd1df92e73..70f76dac78 100644 --- a/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp +++ b/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp @@ -120,11 +120,9 @@ extern "C" int concurrent_native_prepare_supported_impl(void) { // RuntimeEnv (call_config.h) is the cross-runtime ABI for per-ring config and // carries RUNTIME_ENV_RING_COUNT slots, shared with tensormap_and_ringbuffer. -// host_build_graph has one ring and reads slot 0, so it only needs the ABI to -// carry at least one. -static_assert(RUNTIME_ENV_RING_COUNT >= 1, "RuntimeEnv must carry the ring slot host_build_graph reads"); - -static bool is_power_of_2_u64(uint64_t value) { return value != 0 && (value & (value - 1)) == 0; } +// host_build_graph keeps one task table and reads slot 0, so it only needs the ABI +// to carry at least one. +static_assert(RUNTIME_ENV_RING_COUNT >= 1, "RuntimeEnv must carry the slot host_build_graph reads"); // Host monotonic clock, shared with the record pool so spans and records can be // read against each other. @@ -197,7 +195,7 @@ static void warn_on_retired_ring_env() { } } -// ring_task_window points into the #pragma pack(1) RuntimeEnv wire struct +// A RuntimeEnv knob array points into the #pragma pack(1) wire struct // (call_config.h), so its uint64_t entries are only byte-aligned — runtime_env // sits at offset 28 in CallConfig (after 7 int32_t), i.e. 4-byte but not 8-byte // aligned. Reading them as `base[idx]` is an unaligned 8-byte load: UB, and fatal @@ -213,38 +211,42 @@ static uint64_t read_ring_override(const uint64_t *base, int idx) { } // ring_task_window points at the first slot of a per-ring array in the RuntimeEnv -// wire struct (0 = unset); hbg has one ring and reads slot 0. A per-task entry -// wins over the compile-time default, and there is nothing between them. +// wire struct (0 = unset); hbg keeps one task table and reads slot 0. A per-task +// entry wins over the compile-time default, and there is nothing between them. // // The heap takes no configuration: its device region is committed after // orchestration, sized to what the graph turned out to need, so there is nothing // to resolve up front. RuntimeEnv::ring_heap stays the reclaiming runtime's knob; // this function does not resolve it, and the caller reads it only to warn that it // reaches nothing here. -static bool resolve_task_window_size(const uint64_t *ring_task_window, uint64_t *eff_task_window_size) { - *eff_task_window_size = CHIP_TASK_WINDOW_SIZE; +static bool resolve_graph_task_capacity(const uint64_t *ring_task_window, uint64_t *task_capacity) { + *task_capacity = CHIP_DEFAULT_GRAPH_TASKS; warn_on_retired_ring_env(); - const uint64_t task_window_override = read_ring_override(ring_task_window, 0); - if (task_window_override != 0) { - *eff_task_window_size = task_window_override; + const uint64_t override_value = read_ring_override(ring_task_window, 0); + if (override_value != 0) { + *task_capacity = override_value; } - if (*eff_task_window_size < 4 || *eff_task_window_size > static_cast(INT32_MAX) || - !is_power_of_2_u64(*eff_task_window_size)) { - LOG_ERROR("ring_task_window=%" PRIu64 " must be a power of 2 in [4, INT32_MAX]", *eff_task_window_size); + // Any positive count is usable: a task id indexes its slot directly, so + // nothing masks with this value. The power-of-two, >= 4 requirement belongs to + // tensormap_and_ringbuffer, which does mask, and is enforced in that runtime's + // own resolve; neither the RuntimeEnv setter nor Worker.run constrains the + // value, so this bound is the only one a ring_task_window passes through. + if (*task_capacity < 1 || *task_capacity > static_cast(INT32_MAX)) { + LOG_ERROR("ring_task_window=%" PRIu64 " must be in [1, INT32_MAX]", *task_capacity); return false; } // A slot state reaches its payload and descriptor through a 32-bit // self-relative delta, so every pair of addresses in the shared-memory // image must be within INT32_MAX of each other. - const uint64_t sm_bytes = sm_layout::ring_segment_offsets(*eff_task_window_size).end; + const uint64_t sm_bytes = sm_layout::segment_offsets(*task_capacity).end; if (sm_bytes > static_cast(INT32_MAX)) { LOG_ERROR( "ring_task_window=%" PRIu64 " needs a %" PRIu64 "-byte shared memory image, past the %d-byte limit " "a slot state's self-relative payload/descriptor delta can span", - *eff_task_window_size, sm_bytes, INT32_MAX + *task_capacity, sm_bytes, INT32_MAX ); return false; } @@ -490,7 +492,7 @@ struct GraphHostStateBinding { int32_t run_host_orchestration( Runtime *runtime, const HostApi *api, HostTensorAccessor &tensor_access, RuntimeContext *rt, - DeviceArena &host_arena, const RuntimeArenaLayout &layout, uint64_t sm_size, uint64_t eff_task_window_size, + DeviceArena &host_arena, const RuntimeArenaLayout &layout, uint64_t sm_size, uint64_t task_capacity, void *host_orch_func_ptr, const ChipTaskArgs &orch_l2 ) { dep_gen_host_graph_begin_capture(); @@ -499,7 +501,7 @@ int32_t run_host_orchestration( // each written per task at submit and read only for [0, total_tasks). Zero // only the fixed-size header here; the per-slot segments are initialized in // orch::prepare_task and shipped bounded to total_tasks below. - const sm_layout::ChipRingSegmentOffsets sm_segs = sm_layout::ring_segment_offsets(eff_task_window_size); + const sm_layout::SegmentOffsets sm_segs = sm_layout::segment_offsets(task_capacity); // Over-allocated and rounded up: every segment offset is a multiple of // CHIP_ALIGN_SIZE and ChipTaskSlotState is alignas(64), which a plain // new uint8_t[] does not guarantee. @@ -523,15 +525,14 @@ int32_t run_host_orchestration( // actually needs, and compact_live_image moves every address the orchestrator // wrote onto the real base before the image travels. if (!orchestrator.init( - host_sm, reinterpret_cast(HEAP_VIRTUAL_BASE), HEAP_VIRTUAL_CAPACITY, eff_task_window_size, - rt->scheduler + host_sm, reinterpret_cast(HEAP_VIRTUAL_BASE), HEAP_VIRTUAL_CAPACITY, task_capacity, rt->scheduler )) { LOG_ERROR("host-orch: orchestrator init against host SM failed"); return PTO_RUNTIME_ERR_INTERNAL; } SharedMemoryHandle host_sm_handle; - if (!host_sm_handle.init(host_sm, sm_size, eff_task_window_size)) { + if (!host_sm_handle.init(host_sm, sm_size, task_capacity)) { LOG_ERROR("host-orch: host SM init failed"); return PTO_RUNTIME_ERR_INTERNAL; } @@ -632,7 +633,7 @@ int32_t run_host_orchestration( return status; } - const int32_t total_tasks = sm_layout::ring_current_task_index_addr(host_sm)->load(std::memory_order_acquire); + const int32_t total_tasks = orchestrator.task_allocator.active_count(); { char attrs[160]; snprintf( @@ -668,23 +669,29 @@ int32_t run_host_orchestration( } // total_tasks sizes the bounded per-segment H2D copies below; a value outside - // [0, task_window] would make those copies read/write out of bounds. - if (total_tasks < 0 || static_cast(total_tasks) > eff_task_window_size) { - LOG_ERROR("host-orch: total_tasks %d out of range [0, %" PRIu64 "]", total_tasks, eff_task_window_size); + // [0, task_capacity] would make those copies read/write out of bounds. + if (total_tasks < 0 || static_cast(total_tasks) > task_capacity) { + LOG_ERROR("host-orch: total_tasks %d out of range [0, %" PRIu64 "]", total_tasks, task_capacity); return PTO_RUNTIME_ERR_INTERNAL; } host_phase_trace_note_submitted(static_cast(total_tasks)); - // The device reads no ring slot past total_tasks, so only that prefix of each + // The count travels inside the header the restack copies wholesale, which is + // what lets the device bound its completed_watermark walk without a second + // carrier. Written after the range check above, so the value the device reads + // is one the segments are actually pitched to. + reinterpret_cast(host_sm)->tasks.total_tasks = total_tasks; + + // The device reads no task slot past total_tasks, so only that prefix of each // segment has to travel. In the mirror the orchestrator wrote, the four - // prefixes are a ring capacity apart, which would make the upload four copies - // of a few hundred kilobytes each — and at these sizes a copy_to_device is - // priced by the call, not by the bytes. + // prefixes are a whole task_capacity apart, which would make the upload four + // copies of a few hundred kilobytes each — and at these sizes a copy_to_device + // is priced by the call, not by the bytes. // // So the prefixes are restacked into an image pitched to total_tasks, where // they are contiguous, and that image goes up as one copy. The device attaches - // with the same pitch. The ring capacity and mask are untouched: `local_id & - // mask` is `local_id`, which is below the pitch for every ring task. + // with the same pitch, which is sound because a task id is its own slot index: + // every id is below total_tasks and indexes the image directly. const uint64_t nt = static_cast(total_tasks); // What this bind actually put in the pools. The orchestrator's cursors are the // exact populated extent of each one — no scan of the mirror is needed, and the @@ -696,7 +703,7 @@ int32_t run_host_orchestration( static_cast(orch_state.tensor_pool_cursor), static_cast(orch_state.scalar_pool_cursor), }; - const uint64_t image_bytes = sm_layout::ring_segment_offsets(sm_layout::image_extents(bind_usage)).end; + const uint64_t image_bytes = sm_layout::segment_offsets(sm_layout::image_extents(bind_usage)).end; runtime->sm_image_bytes = image_bytes; // Only now are both sizes known, so this is where the two device regions are @@ -798,7 +805,7 @@ int32_t run_host_orchestration( rt->orchestrator = nullptr; std::memcpy(upload_base, static_cast(host_arena.base()) + layout.off_copied_begin, copied_bytes); const uint64_t compacted = sm_layout::compact_live_image( - static_cast(host_sm), eff_task_window_size, bind_usage, heap_rebase, upload_base + copied_bytes + static_cast(host_sm), task_capacity, bind_usage, heap_rebase, upload_base + copied_bytes ); always_assert(compacted == image_bytes); @@ -989,8 +996,8 @@ extern "C" int bind_callable_to_runtime_impl( host_phase_trace_end(); }); - uint64_t eff_task_window_size = 0; - if (!resolve_task_window_size(ring_task_window, &eff_task_window_size)) { + uint64_t task_capacity = 0; + if (!resolve_graph_task_capacity(ring_task_window, &task_capacity)) { return PTO_RUNTIME_ERR_INTERNAL; } // The heap takes no configuration, so a set ring_heap reaches nothing in this @@ -1002,7 +1009,6 @@ extern "C" int bind_callable_to_runtime_impl( "size the graph turned out to need" ); } - LOG_INFO("Ring task window: %" PRIu64, eff_task_window_size); // Build device args: copy from input, replace host tensor pointers with device pointers ChipStorageTaskArgs device_args; @@ -1102,11 +1108,11 @@ extern "C" int bind_callable_to_runtime_impl( // runs — do NOT record in tensor_pairs_; the free is deferred to // DeviceRunner::finalize(). The runtime-arena size is determined by replaying // the reserve sequence on a host-side arena. - uint64_t sm_size = SharedMemoryHandle::calculate_size(eff_task_window_size); + uint64_t sm_size = SharedMemoryHandle::calculate_size(task_capacity); const int64_t t_arena_build_ns = bind_now_ns(); DeviceArena host_arena; - RuntimeArenaLayout layout = runtime_reserve_layout(host_arena, eff_task_window_size); + RuntimeArenaLayout layout = runtime_reserve_layout(host_arena, task_capacity); if (host_arena.commit(DeviceArena::kDefaultBaseAlign) == nullptr) { LOG_ERROR("Failed to commit host arena for prebuilt runtime image"); return PTO_RUNTIME_ERR_INTERNAL; @@ -1162,8 +1168,7 @@ extern "C" int bind_callable_to_runtime_impl( ChipTaskArgs orch_l2; orch_l2.create_from_entry_storage(runtime->get_orch_args()); int32_t total_tasks = run_host_orchestration( - runtime, api, tensor_access, rt, host_arena, layout, sm_size, eff_task_window_size, host_orch_func_ptr, - orch_l2 + runtime, api, tensor_access, rt, host_arena, layout, sm_size, task_capacity, host_orch_func_ptr, orch_l2 ); // The orchestrator is the only host-view reader; from here the device // owns these buffers, so drop the window on both exits. diff --git a/src/a2a3/runtime/host_build_graph/runtime/orchestrator.h b/src/a2a3/runtime/host_build_graph/runtime/orchestrator.h index 46fc583592..5cfa80b843 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/orchestrator.h +++ b/src/a2a3/runtime/host_build_graph/runtime/orchestrator.h @@ -30,7 +30,7 @@ #include #include "common/chip_swimlane_profiling.h" -#include "ring_buffer.h" +#include "task_allocator.h" #include "graph_cache.h" #include "runtime_types.h" #include "submit_types.h" @@ -106,8 +106,8 @@ struct OrchestratorState { // binds its region at the cursor and advances it by what the task uses, so the // pools stay packed and the bind path reads the cursors as the image's pool // extents. Nothing is ever returned, and the pools are dimensioned for the worst - // case (task_window tasks each at their full cap), so a bump cannot overflow. - // The bases live here, not in the ring header: nothing on the device resolves one. + // case (max_tasks tasks each at their full cap), so a bump cannot overflow. + // The bases live here, not in the task header: nothing on the device resolves one. // // The fanin cursor does not advance at bind time. FaninBuilder appends and // dedups producers afterwards, so the region's length is known only when the count @@ -135,11 +135,12 @@ struct OrchestratorState { // and bind this orchestrator to one shared-memory mirror, GM heap and // scheduler. sm_base is the base of the mirror this orchestrator writes; it // is dereferenced, so a host-orch pass passes its host mirror rather than a - // device address. task_window_size must be a power of two. + // device address. `max_tasks` is the slot count that mirror is dimensioned + // for — the bind's resolved ring_task_window — and the cap alloc() enforces. // // Returns false when an allocation fails; the caller then has no hazard map // and must not orchestrate. - bool init(void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t task_window_size, SchedulerState *scheduler); + bool init(void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t max_tasks, SchedulerState *scheduler); void set_scheduler(SchedulerState *scheduler); void report_fatal(int32_t error_code, const char *func, const char *fmt, ...); diff --git a/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp b/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp index 5a1bab7d5f..8d49941ecd 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp @@ -1126,8 +1126,7 @@ static uint32_t next_fanin_seen_epoch(OrchestratorState *orch) { uint32_t next = orch->fanin_seen_current_epoch + 1; if (next == 0) { memset( - orch->fanin_seen_epoch.get(), 0, - static_cast(orch->sm_header->ring.task_window_size) * sizeof(uint32_t) + orch->fanin_seen_epoch.get(), 0, static_cast(orch->task_allocator.capacity()) * sizeof(uint32_t) ); next = 1; } @@ -1158,18 +1157,18 @@ struct FaninBuilder { // re-resolve the delta per producer. Requires the regions bound before construction. int32_t *slots{nullptr}; - bool mark_seen(uint8_t prod_ring, int32_t prod_slot) { - // Dedup only: a producer this orchestrator did not place has no slot in the + bool mark_seen(TaskId producer_task_id) { + // Dedup only: a producer this orchestrator did not place has no entry in the // epoch table, so it is reported as not-yet-seen and the caller appends it - // rather than rejecting it. hbg places every task on its one ring, so the - // assert is the invariant; enforcing it on the caller's side would be a new - // fatal path. - debug_assert(prod_ring == 0 && "hbg places every task on its one ring"); - if (prod_ring != 0 || prod_slot < 0) { + // rather than rejecting it. hbg places every task on ring 0, so the assert is + // the invariant; enforcing it on the caller's side would be a new fatal path. + debug_assert(producer_task_id.ring() == 0 && "hbg places every task on ring 0"); + const int32_t prod_local = static_cast(producer_task_id.local()); + if (producer_task_id.ring() != 0 || prod_local < 0) { return false; } uint32_t *seen = orch->fanin_seen_epoch.get(); - uint32_t slot = static_cast(prod_slot); + uint32_t slot = static_cast(prod_local); if (seen[slot] == seen_epoch) { return true; } @@ -1179,8 +1178,7 @@ struct FaninBuilder { }; static bool append_fanin_or_fail( - OrchestratorState *orch, uint8_t prod_ring, int32_t prod_slot, ChipTaskSlotState *prod_state, - TaskId producer_task_id, FaninBuilder *fanin_builder + OrchestratorState *orch, ChipTaskSlotState *prod_state, TaskId producer_task_id, FaninBuilder *fanin_builder ) { // Skip a stale/reused producer slot: the cached owner id no longer resolves // to this producer (defensive — whole-graph-resident hbg does not reuse slots @@ -1189,8 +1187,8 @@ static bool append_fanin_or_fail( if (prod_state->task == nullptr || prod_state->task->task_id.local() != producer_task_id.local()) { return true; } - // Dedup by (ring, slot). Single-ring hbg: prod_ring is always 0. - if (fanin_builder->mark_seen(prod_ring, prod_slot)) { + // Dedup by producer local id, which is also its task-table slot. + if (fanin_builder->mark_seen(producer_task_id)) { return true; } if (fanin_builder->count >= CHIP_MAX_FANIN) { @@ -1219,7 +1217,7 @@ static bool append_fanin_or_fail( struct PreparedTask { TaskId task_id = TaskId::invalid(); - TaskAllocResult alloc_result = {-1, 0, nullptr, nullptr}; + TaskAllocResult alloc_result = {-1, nullptr, nullptr}; TaskDescriptor *task = nullptr; TaskPayload *payload = nullptr; ChipTaskSlotState *slot_state = nullptr; @@ -1243,7 +1241,6 @@ static bool prepare_task( TaskAttrs task_attrs, PreparedTask *out ) { always_assert(orch->scope_stack_top >= 0 && "Cannot submit task outside a scope"); - uint8_t ring_id = 0; auto &allocator = orch->task_allocator; int16_t block_num = args.launch_spec.block_num(); @@ -1264,10 +1261,10 @@ static bool prepare_task( return false; } - out->task_id = TaskId::make(ring_id, static_cast(out->alloc_result.task_id)); - out->slot_state = &orch->sm_header->ring.get_slot_state_by_slot(out->alloc_result.slot); - out->task = &orch->sm_header->ring.task_descriptors[out->alloc_result.slot]; - out->payload = &orch->sm_header->ring.task_payloads[out->alloc_result.slot]; + out->task_id = TaskId::make(0, static_cast(out->alloc_result.task_id)); + out->slot_state = &orch->sm_header->tasks.get_slot_state_by_task_id(out->alloc_result.task_id); + out->task = &orch->sm_header->tasks.task_descriptors[out->alloc_result.task_id]; + out->payload = &orch->sm_header->tasks.task_payloads[out->alloc_result.task_id]; // Bind the three argument regions before prefetch() and init(), both of which // dereference them. The scalar cursor advances in whole cache lines because init() @@ -1275,11 +1272,11 @@ static bool prepare_task( // write into the next task's region. A tensor region is aligned for any count, // simpler::hbg::Tensor being two cache lines. The fanin cursor advances at publish, not // here — see the comment where it does. - const uint64_t window = orch->sm_header->ring.task_window_size; + const uint64_t max_tasks = static_cast(orch->task_allocator.capacity()); const int32_t scalar_span = CHIP_ALIGN_UP(args.scalar_count(), ARG_POOL_ALIGN / (int32_t)sizeof(uint64_t)); - debug_assert(static_cast(orch->tensor_pool_cursor) + args.tensor_count() <= window * MAX_TENSOR_ARGS); - debug_assert(static_cast(orch->scalar_pool_cursor) + scalar_span <= window * MAX_SCALAR_ARGS); - debug_assert(static_cast(orch->fanin_pool_cursor) + CHIP_MAX_FANIN <= window * CHIP_MAX_FANIN); + debug_assert(static_cast(orch->tensor_pool_cursor) + args.tensor_count() <= max_tasks * MAX_TENSOR_ARGS); + debug_assert(static_cast(orch->scalar_pool_cursor) + scalar_span <= max_tasks * MAX_SCALAR_ARGS); + debug_assert(static_cast(orch->fanin_pool_cursor) + CHIP_MAX_FANIN <= max_tasks * CHIP_MAX_FANIN); out->payload->bind_regions( orch->tensor_pool + orch->tensor_pool_cursor, orch->scalar_pool + orch->scalar_pool_cursor, orch->fanin_pool + orch->fanin_pool_cursor @@ -1293,13 +1290,13 @@ static bool prepare_task( // past total_tasks, so this claim-time write is the only per-slot SM reset and // the unclaimed tail is neither initialized nor read. out->slot_state->reset_for_reuse(); - orch->sm_header->ring.completion_flags[out->alloc_result.slot].store(0, std::memory_order_relaxed); + orch->sm_header->tasks.completion_flags[out->alloc_result.task_id].store(0, std::memory_order_relaxed); out->payload->prefetch(args.tensor_count(), args.scalar_count()); // Re-bind payload/task pointers each submit. Value is per-slot constant // (same as &task_payloads[slot] / &task_descriptors[slot]), but writing - // here lets RingSchedState::init() skip the O(window_size) bind loop. + // here lets TaskHeaderView::init() skip the O(max_tasks) bind loop. // Both writes hit the same 64B slot_state cache line we're about to // dirty below, so the extra cost is two stores on an already-hot line. // Must precede the Orch-side wiring publish at the end of @@ -1314,8 +1311,6 @@ static bool prepare_task( // Fields already zeroed by the reset_for_reuse() above: // wake_list_head=nullptr, next_in_wake_list=nullptr, // any_subtask_deferred=false, completed_subtasks=0, next_block_idx=0 - // Fields immutable after RingSchedState::init(): - // ring_id // task_state is set to PENDING here as the orchestrator populates the slot // (host_build_graph does not recycle slots at runtime, so there is no // post-CONSUMED reset path). @@ -1506,12 +1501,10 @@ static TaskOutputTensors submit_task_common( if (capture_dep_graph) { dep_gen_host_graph_add_explicit_edge(dep_task_id.raw); } - uint8_t dep_ring_id = dep_task_id.ring(); - SharedMemoryRingHeader &dep_ring = orch->sm_header->ring; + SharedMemoryTaskHeader &dep_tasks = orch->sm_header->tasks; int32_t dep_local_task_id = static_cast(dep_task_id.local()); - int32_t dep_slot = dep_ring.get_slot_by_task_id(dep_local_task_id); - ChipTaskSlotState *producer_slot_state = &dep_ring.get_slot_state_by_slot(dep_slot); - if (!append_fanin_or_fail(orch, dep_ring_id, dep_slot, producer_slot_state, dep_task_id, &fanin_builder)) { + ChipTaskSlotState *producer_slot_state = &dep_tasks.get_slot_state_by_task_id(dep_local_task_id); + if (!append_fanin_or_fail(orch, producer_slot_state, dep_task_id, &fanin_builder)) { return result; } } @@ -1523,11 +1516,10 @@ static TaskOutputTensors submit_task_common( }; auto runtime_emit = [&](TaskId producer_task_id) -> bool { - uint8_t prod_ring = producer_task_id.ring(); - SharedMemoryRingHeader &producer_ring = orch->sm_header->ring; - int32_t prod_slot = producer_ring.get_slot_by_task_id(static_cast(producer_task_id.local())); - ChipTaskSlotState *prod_state = &producer_ring.get_slot_state_by_slot(prod_slot); - return append_fanin_or_fail(orch, prod_ring, prod_slot, prod_state, producer_task_id, &fanin_builder); + SharedMemoryTaskHeader &producer_tasks = orch->sm_header->tasks; + ChipTaskSlotState *prod_state = + &producer_tasks.get_slot_state_by_task_id(static_cast(producer_task_id.local())); + return append_fanin_or_fail(orch, prod_state, producer_task_id, &fanin_builder); }; // The capture branch instantiates compute_task_fanin with a live Annotate; @@ -1760,26 +1752,26 @@ bool graph_submit_outer( ) { always_assert(orch->scope_stack_top >= 0 && "Cannot submit Graph outside a scope"); auto &allocator = orch->task_allocator; - if (allocator.active_count() >= allocator.window_size() || + if (allocator.active_count() >= allocator.capacity() || (!defer_heap && static_cast(owned_heap) > allocator.heap_available())) { - LOG_WARN("%s", "[GraphExecution] task-window/heap preflight failed; using ordinary path"); + LOG_WARN("%s", "[GraphExecution] task-capacity/heap preflight failed; using ordinary path"); return false; } // The argument pools hold MAX_TENSOR_ARGS ChipTensors and MAX_SCALAR_ARGS scalars - // per window slot, a budget no CoreTaskArgs task can exceed. A Graph boundary is + // per task slot, a budget no CoreTaskArgs task can exceed. A Graph boundary is // GraphTaskArgs-wide, so a wide one draws more than the single slot it occupies is // worth — GraphBoundaryPool.WidestBoundaryExceedsOneSlotBudget pins how much. The // cursors bump through a fixed mirror whose last segment is the scalar pool, so an // overdraw writes past that mirror rather than merely exhausting a quota. Test it // ahead of the slot claim and decline the Graph path, which leaves the caller to // replay the block as ordinary tasks. - const uint64_t task_window = orch->sm_header->ring.task_window_size; + const uint64_t max_tasks = static_cast(orch->task_allocator.capacity()); const int32_t tensor_slots = static_cast(graph_boundary_tensor_pool_slots(static_cast(args.tensor_count()))); const int32_t scalar_span = CHIP_ALIGN_UP(args.scalar_count(), ARG_POOL_ALIGN / (int32_t)sizeof(uint64_t)); - if (static_cast(orch->tensor_pool_cursor) + tensor_slots > task_window * MAX_TENSOR_ARGS || - static_cast(orch->scalar_pool_cursor) + scalar_span > task_window * MAX_SCALAR_ARGS) { + if (static_cast(orch->tensor_pool_cursor) + tensor_slots > max_tasks * MAX_TENSOR_ARGS || + static_cast(orch->scalar_pool_cursor) + scalar_span > max_tasks * MAX_SCALAR_ARGS) { LOG_WARN("%s", "[GraphExecution] boundary exceeds the argument pools; using ordinary path"); return false; } @@ -1800,10 +1792,10 @@ bool graph_submit_outer( return false; } const TaskId task_id = TaskId::make(0, static_cast(allocation.task_id)); - SharedMemoryRingHeader &ring = orch->sm_header->ring; - TaskDescriptor &task = ring.task_descriptors[allocation.slot]; - TaskPayload &payload = ring.task_payloads[allocation.slot]; - ChipTaskSlotState &slot = ring.get_slot_state_by_slot(allocation.slot); + SharedMemoryTaskHeader &tasks = orch->sm_header->tasks; + TaskDescriptor &task = tasks.task_descriptors[allocation.task_id]; + TaskPayload &payload = tasks.task_payloads[allocation.task_id]; + ChipTaskSlotState &slot = tasks.get_slot_state_by_task_id(allocation.task_id); // Init-on-write, as in prepare_task: this slot's dynamic scheduling fields and // completion flag are established here, at the claim, because nothing else @@ -1811,7 +1803,7 @@ bool graph_submit_outer( // list against every consumer, and a stale completion flag would report the // Graph done before it ran. slot.reset_for_reuse(); - ring.completion_flags[allocation.slot].store(0, std::memory_order_relaxed); + tasks.completion_flags[allocation.task_id].store(0, std::memory_order_relaxed); slot.bind_buffers(&payload, &task); // Graph boundaries use the same compact argument pools as ordinary tasks. The @@ -1851,16 +1843,14 @@ bool graph_submit_outer( FaninBuilder fanin_builder(orch, &payload, static_cast(task_id.local()), next_fanin_seen_epoch(orch)); auto emit = [&](TaskId producer_id) -> bool { - const int32_t producer_local = static_cast(producer_id.local()); - const int32_t producer_slot = ring.get_slot_by_task_id(producer_local); - ChipTaskSlotState *producer = &ring.get_slot_state_by_slot(producer_slot); - return append_fanin_or_fail(orch, producer_id.ring(), producer_slot, producer, producer_id, &fanin_builder); + ChipTaskSlotState *producer = &tasks.get_slot_state_by_task_id(static_cast(producer_id.local())); + return append_fanin_or_fail(orch, producer, producer_id, &fanin_builder); }; - // An outer GRAPH task is a ring task like any other, so the dependency graph + // An outer GRAPH task is an ordinary task, so the dependency graph // has to carry it: without this the whole Graph — and every edge into it — // is absent from deps.json, leaving a run of 40 replays described by only its // handful of non-Graph tasks. It dispatches no kernel of its own and the - // sub-DAG it replays owns no ring slots, so what is captured is its boundary: + // sub-DAG it replays owns no task slots, so what is captured is its boundary: // the args it consumes and the edges those produce. const bool capture_dep_graph = dep_gen_host_graph_enabled(); if (capture_dep_graph) { @@ -1956,10 +1946,10 @@ bool graph_finalize_pending_submissions(OrchestratorState *orch, GraphHostState } // Record one internal Graph node while recording, without consuming a -// ring task-window slot. Builds the node's metadata and materialized outputs +// task-table slot. Builds the node's metadata and materialized outputs // exactly as submit_task_common would, but assigns output buffers from the // bit-63 virtual address range and derives internal fanins from tensor-source -// classification — so no ring slot, tensormap entry, fanin-pool entry, or upload +// classification — so no task slot, tensormap entry, fanin-pool entry, or upload // is produced for the node. The resulting Definition is later attached to the // outer GRAPH shells already submitted by the main thread. The returned // TaskOutputTensors borrow the node's own tensor storage; moving the node into @@ -2753,9 +2743,9 @@ TaskOutputTensors OrchestratorState::alloc_tensors(const CoreTaskArgs &args) { // every consumer register_wakes on a producer that never runs on device and // the run hangs. (The device watermark walk transparently steps past this // pre-set flag when a later on-device task completes.) - SharedMemoryRingHeader &done_ring = orch->sm_header->ring; + SharedMemoryTaskHeader &done_tasks = orch->sm_header->tasks; int32_t done_local = static_cast(prepared.task_id.local()); - done_ring.set_completion_flag(done_local); + done_tasks.set_completion_flag(done_local); } orch->inline_completed_tasks++; diff --git a/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/ring_buffer.cpp b/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/ring_buffer.cpp deleted file mode 100644 index 26b277c8a3..0000000000 --- a/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/ring_buffer.cpp +++ /dev/null @@ -1,17 +0,0 @@ -/* - * Copyright (c) PyPTO Contributors. - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of - * CANN Open Software License Agreement Version 2.0 (the "License"). - * Please refer to the License for details. You may not use this file except in compliance with the License. - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. - * See LICENSE in the root of the software repository for the full text of the License. - * ----------------------------------------------------------------------------------------------------------- - */ - -// The polling completion scheduler has no dep-pool / fanin-spill pool: producer -// dependencies are position-independent local-id integers on the payload and -// readiness is derived from the ring completion_flags. The pool ring buffers -// (FaninPool / DepListPool) that once lived here are gone, so this -// translation unit carries no definitions. TaskAllocator's methods are -// defined inline in ring_buffer.h. diff --git a/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/runtime_core.cpp b/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/runtime_core.cpp index 235d090ac6..698f3bdaaa 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/runtime_core.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/runtime_core.cpp @@ -195,15 +195,14 @@ static bool wait_for_tensor_ready( return nullptr; } - auto &ring = orch.sm_header->ring; + auto &tasks = orch.sm_header->tasks; int32_t local_id = static_cast(producer.local()); - int32_t slot_index = ring.get_slot_by_task_id(local_id); - auto &slot = ring.get_slot_state_by_slot(slot_index); + auto &slot = tasks.get_slot_state_by_task_id(local_id); if (slot.task == nullptr || slot.task->task_id != producer) { orch.report_fatal( SIMPLER_ERROR_INVALID_ARGS, caller, "tensor producer task %#llx does not match the descriptor bound to slot %d", - static_cast(producer.raw), slot_index + static_cast(producer.raw), local_id ); failed = true; return nullptr; @@ -212,7 +211,6 @@ static bool wait_for_tensor_ready( }; auto wait_one_producer = [&](const ChipTaskSlotState &slot) { - uint8_t ring_id = 0; int32_t local_id = static_cast(slot.task->task_id.local()); uint64_t t0 = get_sys_cnt_aicpu(); int32_t spin_count = 0; @@ -228,8 +226,8 @@ static bool wait_for_tensor_ready( if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) { orch.report_fatal( SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller, - "Timeout (%llu cycles): producer (ring=%d, local=%d) not completed", - (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id + "Timeout (%llu cycles): producer (local=%d) not completed", + (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, local_id ); failed = true; return; @@ -239,7 +237,6 @@ static bool wait_for_tensor_ready( }; auto wait_one_consumers = [&](const ChipTaskSlotState &slot) { - uint8_t ring_id = 0; int32_t local_id = slot.task->task_id.local(); uint64_t t0 = get_sys_cnt_aicpu(); int32_t spin_count = 0; @@ -247,8 +244,8 @@ static bool wait_for_tensor_ready( // completed_watermark reaches the producer's highest consumer id (set at // submit in append_fanin_or_fail). Replaces the fanout_refcount == // fanout_count wiring check, which polling removes. - SharedMemoryRingHeader &cons_ring = orch.sm_header->ring; - while (cons_ring.completed_watermark.load(std::memory_order_acquire) < slot.last_consumer_local_id) { + SharedMemoryTaskHeader &cons_tasks = orch.sm_header->tasks; + while (cons_tasks.completed_watermark.load(std::memory_order_acquire) < slot.last_consumer_local_id) { SPIN_WAIT_HINT(); if ((++spin_count & 1023) == 0) { // A fatal latched elsewhere (e.g. the scheduler-side wiring @@ -260,8 +257,8 @@ static bool wait_for_tensor_ready( if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) { orch.report_fatal( SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller, - "Timeout (%llu cycles): consumers of producer (ring=%d, local=%d) not done", - (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id + "Timeout (%llu cycles): consumers of producer (local=%d) not done", + (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, local_id ); failed = true; return; diff --git a/src/a2a3/runtime/host_build_graph/runtime/runtime.h b/src/a2a3/runtime/host_build_graph/runtime/runtime.h index a84e5424e6..cfb3e05785 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/runtime.h +++ b/src/a2a3/runtime/host_build_graph/runtime/runtime.h @@ -169,10 +169,10 @@ class Runtime { // NOTE: Made public for direct access from aicore code uint64_t func_id_to_addr_[RUNTIME_MAX_FUNC_ID]; - // Total tasks submitted by the host orchestrator — handed to the scheduler - // (SchedulerContext::on_orchestration_done) in place of latching the SM ring - // head on device. host_build_graph builds the whole graph on the host, so - // the boot thread reads this instead of counting SM ring heads. + // Total tasks the host orchestrator submitted, handed to the scheduler by + // SchedulerContext::on_orchestration_done. host_build_graph builds the whole + // graph on the host, so this scalar is the count's only carrier: the shared + // memory header holds no task counter for the boot thread to read. int32_t host_total_tasks; // Size of the shipped shared-memory image, argument pools included. Set by the diff --git a/src/a2a3/runtime/host_build_graph/runtime/runtime_core.h b/src/a2a3/runtime/host_build_graph/runtime/runtime_core.h index 6e17dfd18b..e035ab171f 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/runtime_core.h +++ b/src/a2a3/runtime/host_build_graph/runtime/runtime_core.h @@ -15,7 +15,7 @@ * It provides a unified API for task graph construction and execution. * * Key Features: - * - Ring buffer based memory management (zero allocation overhead) + * - Bump-allocated task table and graph heap (zero allocation overhead) * - Lazy invalidation TensorMap for dependency discovery * - Manual scopes that bypass TensorMap discovery for explicit dependencies * - Per-task spinlocks for concurrent fanout updates @@ -41,7 +41,7 @@ #include "graph_cache.h" #include "submit_types.h" #include "shared_memory.h" -#include "ring_buffer.h" +#include "task_allocator.h" #include "tensormap.h" #include "scheduler/scheduler.h" #include "orchestrator.h" @@ -150,9 +150,10 @@ struct RuntimeArenaLayout { size_t off_copied_begin{0}; size_t off_copied_end{0}; - // Cached parameter (re-used by init_data + wire stages). hbg is single-ring, - // so this is the one ring's. - uint64_t task_window_size{0}; + // The task-table slot count this image was built for, resolved per bind from + // runtime_env.ring_task_window. The device needs it to bound the pitch it + // attaches with. + uint64_t task_capacity{0}; // Total arena byte size post-commit. Used by host to size the prebuilt // image buffer and as the rtMemcpy length, and requested of the device as @@ -233,7 +234,7 @@ static_assert( * device memory and may run on host. Returns the layout descriptor; caller * commits/attaches the arena before Phase 2/3. */ -RuntimeArenaLayout runtime_reserve_layout(DeviceArena &arena, uint64_t task_window_size); +RuntimeArenaLayout runtime_reserve_layout(DeviceArena &arena, uint64_t task_capacity); /** * Phase 2 — write the data half of the runtime arena: standalone fields, @@ -306,7 +307,7 @@ void rt_scope_begin(RuntimeContext *rt); * * Closes the innermost scope, and when that is the outermost MANUAL one, returns * later submits to TensorMap discovery. A scope bounds no task or buffer lifetime - * here: the ring is whole-graph-resident, so no task slot and no heap byte is + * here: the task table is whole-graph-resident, so no task slot and no heap byte is * reclaimed before the run ends. */ void rt_scope_end(RuntimeContext *rt); diff --git a/src/a2a3/runtime/host_build_graph/runtime/runtime_types.h b/src/a2a3/runtime/host_build_graph/runtime/runtime_types.h index b3c1168bef..d78be43d3b 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/runtime_types.h +++ b/src/a2a3/runtime/host_build_graph/runtime/runtime_types.h @@ -70,20 +70,29 @@ // ============================================================================= // Task management -// Actual window size is passed at runtime to runtime_create_from_sm(). -// The slot is local_id masked by the window size, which is why the window is a power of two. -#define CHIP_TASK_WINDOW_SIZE 16384 // Default task window size (power of 2) - -// host_build_graph has one ring, and carries no per-ring dimension anywhere: -// host-orch builds the whole graph on the host, it fits that ring, and the -// device runs it once without reclaim. tmr's multi-ring design existed only to -// let inner scopes reclaim independently under small rings; with no reclaim and -// a whole-graph-resident ring, per-depth isolation is moot, so every scope depth -// uses the same ring. The RuntimeEnv ABI still carries RUNTIME_ENV_RING_COUNT -// slots because it is shared with tmr; this runtime reads slot 0 (see -// bind_callable_to_runtime_impl). - -// Memory pools (total = value, single ring) +// +// The task table is a flat array of slots, indexed directly by local task id: +// ids start at 0, are never recycled, and alloc() caps them at the table's size, +// so there is no wrap and no slot mask. The size need not be a power of two — +// nothing masks with it. +// +// This is the default; `CallConfig.runtime_env.ring_task_window` overrides it per +// task. The host mirror is allocated at whatever size is in effect and committed +// by first touch, so a run pays only for the slots and argument-pool bytes it +// actually writes. Raising it costs virtual address space, bounded by the int32 +// reach of a payload's self-relative region deltas (checked in +// SharedMemoryHandle::init). +#define CHIP_DEFAULT_GRAPH_TASKS 16384 + +// host_build_graph carries no per-scope-depth task partition: host-orch builds +// the whole graph on the host and the device runs it once without reclaim. tmr's +// multi-ring design existed only to let inner scopes reclaim independently under +// small rings; with no reclaim and a whole-graph-resident task table, per-depth +// isolation is moot. The RuntimeEnv ABI still carries RUNTIME_ENV_RING_COUNT +// slots because it is shared with tmr; this runtime reads none of them and warns +// when one is set (see bind_callable_to_runtime_impl). + +// Memory pools (total = value) #define CHIP_TENSORMAP_POOL_SIZE (65536) // TensorMap entry pool #define CHIP_TENSORMAP_NUM_BUCKETS 4096 // Power of 2 for fast hash (4096×8B=32KB fits L1) @@ -190,10 +199,11 @@ typedef enum { /** * Result of a unified task allocation. + * + * There is no separate slot: a task id indexes the task table directly. */ struct TaskAllocResult { - int32_t task_id; // Absolute task ID (not wrapped) - int32_t slot; // task_id & (window_size - 1) + int32_t task_id; // Task id, which is also its task-table index void *packed_base; // Heap allocation result (nullptr if failure) void *packed_end; // packed_base + aligned output_size @@ -462,7 +472,7 @@ struct TaskPayload { // The ring's payload storage is reused raw memory that no constructor runs // over, so an unset predicate reads back as whatever the slot last held — // and compact_live_image translates predicate.addr as a graph-heap address - // for every submitted slot. An ordinary ring task overwrites this right + // for every submitted slot. An ordinary task overwrites this right // after init(); a hidden-alloc task has nothing following it, so this is // the only value its predicate ever gets. predicate = DispatchPredicate{}; @@ -595,7 +605,7 @@ struct alignas(64) ChipTaskSlotState { // the slot state at one cache line and preserving the 40-byte descriptor // ABI consumed by AICore. Readiness uses the shared intrusive wake-list // fields above; this index identifies the node in the saved fanin CSR. - // Ordinary ring tasks leave both Graph fields -1/null. + // Ordinary tasks leave both Graph fields -1/null. int32_t graph_node_index{-1}; void *graph_context{nullptr}; diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h index 6f4e50fdb5..65cf34b583 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h @@ -18,7 +18,7 @@ * producer named in its inline fanin has set its completion_flags byte; * a producer publishes completion + drains its wake list on finish * 3. Publishing the host-visible task_state mirror (PENDING -> COMPLETED) and - * advancing the per-ring completed_watermark (consumer-retirement signal) + * advancing the completed_watermark (consumer-retirement signal) * 4. Two-stage mixed-task completion (subtask done bits -> mixed-task complete) * * The Scheduler runs on Device AI_CPU. host_build_graph is scheduler-only (the @@ -39,7 +39,7 @@ #include "aicpu/platform_regs.h" // get_reg_ptr / RegId for the early-dispatch doorbell #include "async_wait.h" #include "graph_execution.h" -#include "ring_buffer.h" +#include "task_allocator.h" #include "runtime_types.h" #include "shared_memory.h" @@ -474,21 +474,19 @@ struct SchedulerState { // Shared memory access SharedMemoryHeader *sm_header; - // Per-ring state - struct alignas(64) RingSchedState { - // --- Cache Line 0: ring pointer (read-only) + hot path (read-write) --- - SharedMemoryRingHeader *ring; - std::atomic advance_lock; // multi-thread CAS + // The task table's header, as a device address + struct alignas(64) TaskHeaderView { + SharedMemoryTaskHeader *tasks; - // Polling: no per-ring dep_pool. Readiness is derived from the SM ring's + // Polling: no dep_pool. Readiness is derived from the task table's // completion_flags; there is no arena-side wiring pool to reserve or wire. - // The `ring` field stores the device address of the SM ring header — + // The `tasks` field stores the device address of the SM task header — // computed via offset arithmetic, no SM dereference. bool init_data_from_layout(void *sm_dev_base); void destroy(); - } ring_sched_state; + } task_view; - // Ready queues remain global (scheduling is ring-agnostic) + // Ready queues are global: scheduling does not partition by task id ChipReadyQueue ready_queues[NUM_RESOURCE_SHAPES]; // Ready sync_start queues, one per shape. A ready sync_start cohort parks here @@ -570,7 +568,7 @@ struct SchedulerState { } } - // ---- Polling completion primitives (single-ring hbg) ---------------------- + // ---- Polling completion primitives --------------------------------------- // Readiness: a task is ready iff every producer named in its inline fanin has // set its completion_flags byte. Single-ring: all producers are ring 0, so // there is no per-edge ring indirection. @@ -586,10 +584,10 @@ struct SchedulerState { // completion re-scans its waiters via on_mixed_task_complete's wake drain. int classify_fanin_state(const ChipTaskSlotState *s) const { const TaskPayload &p = *s->payload; - const SharedMemoryRingHeader &ring = *ring_sched_state.ring; + const SharedMemoryTaskHeader &tasks = *task_view.tasks; const int32_t *fanin = p.fanin_data(); for (int32_t i = p.fanin_count - 1; i >= 0; i--) { - if (!ring.is_completion_flag_set(fanin[i])) return i; + if (!tasks.is_completion_flag_set(fanin[i])) return i; } return -1; } @@ -599,7 +597,7 @@ struct SchedulerState { // ready only when every fanin is met, else re-target the next unmet producer // and retry. Monotonic completion_flags guarantee termination. void register_wake(ChipTaskSlotState *producer, ChipTaskSlotState *consumer) { - SharedMemoryRingHeader &ring = *ring_sched_state.ring; + SharedMemoryTaskHeader &tasks = *task_view.tasks; while (true) { ChipTaskSlotState *expected = producer->wake_list_head.load(std::memory_order_relaxed); while (expected != WAKE_LIST_SENTINEL) { @@ -615,7 +613,7 @@ struct SchedulerState { push_ready_routed(consumer); return; } - producer = &ring.get_slot_state_by_task_id(consumer->payload->fanin_data()[state]); + producer = &tasks.get_slot_state_by_task_id(consumer->payload->fanin_data()[state]); } } @@ -624,13 +622,13 @@ struct SchedulerState { // (route/re-register each waiter), then CAS-advance the monotonic // completed_watermark (load-bearing: the host wait_for_consumers gates on // watermark >= producer.last_consumer_local_id). Whole-graph-resident hbg - // has no device slot reclaim, so no advance_ring_pointers here. + // has no device slot reclaim, so nothing advances a reclaim cursor here. void on_mixed_task_complete(ChipTaskSlotState &slot_state) { const int32_t task_id = static_cast(slot_state.task->task_id.local()); - SharedMemoryRingHeader &ring = *ring_sched_state.ring; + SharedMemoryTaskHeader &tasks = *task_view.tasks; slot_state.mark_completed(); // host-visible mirror (task_state = COMPLETED) - ring.set_completion_flag(task_id); + tasks.set_completion_flag(task_id); ChipTaskSlotState *waiter = slot_state.wake_list_head.exchange(WAKE_LIST_SENTINEL, std::memory_order_acq_rel); while (waiter != nullptr && waiter != WAKE_LIST_SENTINEL) { @@ -644,7 +642,7 @@ struct SchedulerState { if (state < 0) { push_ready_routed(waiter); } else { - register_wake(&ring.get_slot_state_by_task_id(waiter->payload->fanin_data()[state]), waiter); + register_wake(&tasks.get_slot_state_by_task_id(waiter->payload->fanin_data()[state]), waiter); } waiter = next; } @@ -656,7 +654,7 @@ struct SchedulerState { // makes the final value order-dependent: a low-id task completing after a // higher one would leave the watermark stuck below the true prefix, hanging // any wait_for_consumers whose last_consumer sits in the gap. - ring.update_completed_watermark(); + tasks.update_completed_watermark(); } // Polling: there is no ready-claim CAS (a producer routes each waiter exactly @@ -1091,7 +1089,7 @@ struct SchedulerState { if (!graph_completed) return outcome; // Internal nodes count as zero stream tasks. The final node publishes - // the outer ring task exactly once, waking external consumers and + // the outer task exactly once, waking external consumers and // contributing the one task the host actually submitted. if (execution->outer_slot != nullptr) { on_mixed_task_complete(*execution->outer_slot); @@ -1170,10 +1168,9 @@ struct SchedulerState { // Phase 3a: write everything *except* arena-internal pointer fields. // `sm_dev_base` is the device address of the SM (only stored, never // dereferenced here). Safe to call on a host arena that holds the - // prebuilt image buffer. (The orchestrator counterpart takes - // task_window_size for ring task_descriptors address arithmetic; the - // scheduler only needs the SM header / ring header base addresses, - // both window-size-independent.) + // prebuilt image buffer. (The orchestrator counterpart takes task_capacity + // for its task_descriptors address arithmetic; the scheduler only needs the + // SM header and task header base addresses, both capacity-independent.) bool init_data_from_layout(const SchedulerLayout &layout, DeviceArena &arena, void *sm_dev_base); // Phase 3b: write the arena-internal pointer fields diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp index 703b18b835..6fc25dd78a 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp @@ -226,25 +226,26 @@ void SchedulerContext::log_stall_diagnostics( ) { CoreTracker &tracker = core_trackers_[thread_idx]; - // T0 owns the shared-ring scan; printing it from other threads would + // T0 owns the task-table scan; printing it from other threads would // produce identical TASK lines once per scheduler thread. if (thread_idx == 0) { - int32_t cnt_ready = 0, cnt_waiting = 0, cnt_running = 0, submitted_in_ring = 0; - SharedMemoryRingHeader &ring = *sched_->ring_sched_state.ring; - int32_t ring_task_count = ring.fc.current_task_index.load(std::memory_order_relaxed); - submitted_in_ring += ring_task_count; - for (int32_t si = 0; si < ring_task_count; si++) { - ChipTaskSlotState &slot_state = ring.get_slot_state_by_task_id(si); + int32_t cnt_ready = 0, cnt_waiting = 0, cnt_running = 0; + SharedMemoryTaskHeader &tasks = *sched_->task_view.tasks; + // `task_count` is the run's task total, which both callers pass from + // total_tasks_. It bounds the scan as well as the SUMMARY line: every slot + // below it was claimed by the host orchestrator, and no slot above it was. + for (int32_t si = 0; si < task_count; si++) { + ChipTaskSlotState &slot_state = tasks.get_slot_state_by_task_id(si); ChipTaskState st = slot_state.task_state.load(std::memory_order_relaxed); // Polling: no fanin_refcount. Recompute met/total from the inline - // fanin ids vs the ring completion_flags (rc = satisfied producers, + // fanin ids vs the completion_flags (rc = satisfied producers, // fi = raw producer count) so the stall dump still shows readiness. int32_t fi = slot_state.payload != nullptr ? slot_state.payload->fanin_count : 0; int32_t rc = 0; if (slot_state.payload != nullptr) { const int32_t *fanin = slot_state.payload->fanin_data(); for (int32_t k = 0; k < fi; k++) { - if (ring.is_completion_flag_set(fanin[k], std::memory_order_relaxed)) rc++; + if (tasks.is_completion_flag_set(fanin[k], std::memory_order_relaxed)) rc++; } } int32_t kid_aic = slot_state.task->kernel_id[0]; @@ -302,12 +303,11 @@ void SchedulerContext::log_stall_diagnostics( thread_idx, idle_iterations, 0, task_id, rc, fi, kid_aic, kid_aiv0, kid_aiv1, fi - rc ); } - int32_t effective_total = task_count > 0 ? task_count : submitted_in_ring; int32_t c = completed_tasks_.load(std::memory_order_relaxed); LOG_INFO( "[STALL thread=%d idle_iterations=%d] SUMMARY completed=%d/%d last_progress_iteration=%d " "scan_ready=%d scan_waiting=%d scan_running=%d", - thread_idx, idle_iterations, c, effective_total, last_progress_count, cnt_ready, cnt_waiting, cnt_running + thread_idx, idle_iterations, c, task_count, last_progress_count, cnt_ready, cnt_waiting, cnt_running ); } @@ -905,21 +905,10 @@ int32_t SchedulerContext::post_handshake_init(Runtime *runtime) { #endif // Initialize task counters. Task count comes from shared memory. - if (runtime->get_gm_sm_ptr()) { - auto *header = static_cast(runtime->get_gm_sm_ptr()); - // Read at one-time boot init, before the SM is reset for the run, so a ring - // not yet written holds uninitialized memory (0xbe... under ASAN's - // malloc-fill). Only a plausible count is taken — (0, - // CHIP_TASK_WINDOW_SIZE], since the ring cannot hold more than its task - // window — so any garbage pattern, negative or positive, leaves the count - // at 0, which is the correct value at boot. - int32_t window_tasks = 0; - int32_t ring_tasks = header->ring.fc.current_task_index.load(std::memory_order_acquire); - if (ring_tasks > 0 && ring_tasks <= CHIP_TASK_WINDOW_SIZE) window_tasks = ring_tasks; - total_tasks_ = window_tasks; - } else { - total_tasks_ = 0; - } + // 0 is the correct count at boot: the graph is not attached yet, and + // on_orchestration_done latches the host-built total before releasing any + // scheduler thread. + total_tasks_ = 0; completed_tasks_.store(0, std::memory_order_release); // prepare_subtask_to_core fully writes a per-core payload / deferred-slab slot @@ -1041,10 +1030,9 @@ void SchedulerContext::bind_runtime(RuntimeContext *rt) { // ============================================================================= // Post-orchestration bookkeeping. Runs once on the boot leader after the // host-built image is attached; latches total_tasks_ and folds inline-completed -// tasks (or shuts down on a fatal orchestration error). The caller publishes -// runtime_init_ready_ (release) after this returns — that store is what makes -// total_tasks_ visible to the scheduler threads, which acquire it before -// dispatching. +// tasks (or shuts down on a fatal orchestration error). classify_ready_ is +// released after this call and is what publishes total_tasks_ to the peer threads, +// which acquire it before classify_partition reads the count. // ============================================================================= void SchedulerContext::on_orchestration_done( Runtime *runtime, RuntimeContext *rt, [[maybe_unused]] int32_t thread_idx, int32_t total_tasks @@ -1053,18 +1041,12 @@ void SchedulerContext::on_orchestration_done( // Allocate the per-S CompletedTaskQueues here on the boot leader, before it // releases runtime_init_ready_ — no scheduler thread can push until then. - // Completed-but-unresolved tasks in flight are bounded by BOTH the total task - // count and the ring's task window (a task must occupy a ring slot to run and - // complete), so size to the tighter of the two, rounded up to a power of two - // and floored at 256. The window already caps this, so there is no artificial - // ceiling and a producer never has to spin on a full queue. + // Completed-but-unresolved tasks in flight are bounded by the run's task count + // (a task must have been submitted to run and complete), so size to that, + // rounded up to a power of two and floored at 256. That bound is exact, so + // there is no artificial ceiling and a producer never has to spin on a full + // queue. uint64_t sp_bound = static_cast(total_tasks); - if (sched_->ring_sched_state.ring != nullptr) { - uint64_t window = static_cast(sched_->ring_sched_state.ring->task_window_mask) + 1; - if (window < sp_bound) { - sp_bound = window; - } - } uint64_t sp_cap = 256; while (sp_cap < sp_bound) { sp_cap <<= 1; @@ -1120,20 +1102,20 @@ void SchedulerContext::on_orchestration_done( // their external fanin follows the same ready/wake classification as any other // outer task. void SchedulerContext::classify_partition(int32_t thread_idx, int32_t nthreads) { - if (completed_.load(std::memory_order_acquire) || sched_->ring_sched_state.ring == nullptr) { + if (completed_.load(std::memory_order_acquire) || sched_->task_view.tasks == nullptr) { return; } - SharedMemoryRingHeader &ring = *sched_->ring_sched_state.ring; - const int32_t submitted = ring.fc.current_task_index.load(std::memory_order_acquire); + SharedMemoryTaskHeader &tasks = *sched_->task_view.tasks; + const int32_t submitted = total_tasks_; // Disjoint contiguous slices covering [0, submitted): thread t owns // [submitted*t/nthreads, submitted*(t+1)/nthreads). int64 math avoids overflow. const int32_t lo = static_cast((static_cast(submitted) * thread_idx) / nthreads); const int32_t hi = static_cast((static_cast(submitted) * (thread_idx + 1)) / nthreads); for (int32_t id = lo; id < hi; id++) { - if (ring.is_completion_flag_set(id)) { + if (tasks.is_completion_flag_set(id)) { continue; // completed on the host (hidden alloc); nothing to dispatch } - ChipTaskSlotState &slot = ring.get_slot_state_by_task_id(id); + ChipTaskSlotState &slot = tasks.get_slot_state_by_task_id(id); if (slot.task_kind == TaskKind::GRAPH) { if (graph_execution_localize(slot) == nullptr) slot.graph_context = nullptr; if (!sched_->push_graph_prepare(&slot, slot.task->task_id.raw, thread_idx)) return; @@ -1143,7 +1125,7 @@ void SchedulerContext::classify_partition(int32_t thread_idx, int32_t nthreads) sched_->push_ready_routed(&slot); } else { int32_t prod_local = slot.payload->fanin_data()[state]; - sched_->register_wake(&ring.get_slot_state_by_task_id(prod_local), &slot); + sched_->register_wake(&tasks.get_slot_state_by_task_id(prod_local), &slot); } } } diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_context.h b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_context.h index 52eb8f0855..d67d2ade98 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_context.h +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_context.h @@ -41,8 +41,9 @@ struct RuntimeContext; // reclaims a slot, so the queued ChipTaskSlotState* stays valid until P reads it. // Single producer (the owning S thread) / single consumer (P): plain // acquire/release on head/tail, no CAS. Capacity is a power of two sized to the -// in-flight bound (min of total tasks and the ring window), so a producer does -// not spin on a full queue in practice; the spin is a correctness backstop only +// run's task count, which bounds the completed-but-unresolved tasks in flight +// exactly, so a producer does not spin on a full queue in practice; the spin is a +// correctness backstop only // (P drains independently, so it always makes progress). The std::atomic cursors // make this type non-copyable / non-movable, so `buf` cannot be double-freed // through an accidental copy. diff --git a/src/a2a3/runtime/host_build_graph/runtime/shared/runtime_init.cpp b/src/a2a3/runtime/host_build_graph/runtime/shared/runtime_init.cpp index 88c84f1a99..75b7a1f3bb 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/shared/runtime_init.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/shared/runtime_init.cpp @@ -24,7 +24,7 @@ #include "orchestrator.h" #include "runtime_core.h" -#include "ring_buffer.h" +#include "task_allocator.h" #include "shared_memory.h" #include "tensormap.h" #include "scheduler/scheduler.h" @@ -63,11 +63,10 @@ void ready_queue_destroy(ChipReadyQueue *queue) { // Scheduler // ============================================================================= -bool SchedulerState::RingSchedState::init_data_from_layout(void *sm_dev_base) { - // ring stores the device address of the SM ring header — pure offset +bool SchedulerState::TaskHeaderView::init_data_from_layout(void *sm_dev_base) { + // `tasks` is the device address of the SM task header — pure offset // arithmetic, no SM load. - ring = sm_layout::ring_header_addr(sm_dev_base); - advance_lock.store(0, std::memory_order_relaxed); + tasks = sm_layout::task_header_addr(sm_dev_base); // Per-slot SM-side initialization (reset_for_reuse + active_mask, and clearing // the completion flag) happens init-on-write in orch::prepare_task as each slot @@ -76,7 +75,7 @@ bool SchedulerState::RingSchedState::init_data_from_layout(void *sm_dev_base) { return true; } -void SchedulerState::RingSchedState::destroy() { ring = nullptr; } +void SchedulerState::TaskHeaderView::destroy() { tasks = nullptr; } SchedulerLayout SchedulerState::reserve_layout(DeviceArena &arena) { SchedulerLayout layout{}; @@ -111,7 +110,7 @@ bool SchedulerState::init_data_from_layout(const SchedulerLayout &layout, Device sched->tasks_consumed.store(0, std::memory_order_relaxed); #endif - if (!sched->ring_sched_state.init_data_from_layout(sm_dev_base)) { + if (!sched->task_view.init_data_from_layout(sm_dev_base)) { return false; } @@ -177,7 +176,7 @@ void SchedulerState::wire_arena_pointers(const SchedulerLayout &layout, DeviceAr void SchedulerState::destroy() { SchedulerState *sched = this; - sched->ring_sched_state.destroy(); + sched->task_view.destroy(); for (int i = 0; i < NUM_RESOURCE_SHAPES; i++) { ready_queue_destroy(&sched->ready_queues[i]); } @@ -198,42 +197,40 @@ void SchedulerState::destroy() { // ============================================================================= bool OrchestratorState::init( - void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t task_window_size, SchedulerState *scheduler_arg + void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t max_tasks, SchedulerState *scheduler_arg ) { auto *orch = this; *orch = OrchestratorState{}; - // A power-of-two window lets the slot index mask instead of dividing. - always_assert(task_window_size > 0 && (task_window_size & (task_window_size - 1)) == 0); + always_assert(max_tasks > 0); orch->sm_header = reinterpret_cast(sm_base); orch->fatal = false; orch->scheduler = scheduler_arg; auto *orch_err = sm_layout::orch_error_code_addr(sm_base); - auto *cur_idx_dev = sm_layout::ring_current_task_index_addr(sm_base); - orch->task_allocator.init(static_cast(task_window_size), cur_idx_dev, gm_heap, heap_size, orch_err); + orch->task_allocator.init(static_cast(max_tasks), gm_heap, heap_size, orch_err); // The mirror's argument pools. Offset arithmetic on the same base as sm_header, // so it holds for whichever SM this orchestrator was pointed at. The cursors // reset with the rest of the state above. auto *sm_bytes = static_cast(sm_base); - const auto pools = sm_layout::ring_segment_offsets(sm_layout::mirror_extents(task_window_size)); + const auto pools = sm_layout::segment_offsets(sm_layout::mirror_extents(max_tasks)); orch->fanin_pool = reinterpret_cast(sm_bytes + pools.fanin_pool); orch->tensor_pool = reinterpret_cast(sm_bytes + pools.tensor_pool); orch->scalar_pool = reinterpret_cast(sm_bytes + pools.scalar_pool); // Polling: no fanin-spill pool — producer ids are inline on the payload. - const auto window = static_cast(task_window_size); - orch->fanin_seen_epoch.reset(new (std::nothrow) uint32_t[window]); + const auto slots = static_cast(max_tasks); + orch->fanin_seen_epoch.reset(new (std::nothrow) uint32_t[slots]); if (orch->fanin_seen_epoch == nullptr) { - LOG_ERROR("Orchestrator scratch allocation failed (task_window=%" PRIu64 ")", task_window_size); + LOG_ERROR("Orchestrator scratch allocation failed (max_tasks=%" PRIu64 ")", max_tasks); return false; } - memset(orch->fanin_seen_epoch.get(), 0, window * sizeof(uint32_t)); + memset(orch->fanin_seen_epoch.get(), 0, slots * sizeof(uint32_t)); - if (!orch->tensor_map.init_default(static_cast(task_window_size))) { + if (!orch->tensor_map.init_default(static_cast(max_tasks))) { return false; } @@ -249,10 +246,10 @@ void OrchestratorState::set_scheduler(SchedulerState *scheduler) { this->schedul // Top-level runtime arena // ============================================================================= -RuntimeArenaLayout runtime_reserve_layout(DeviceArena &arena, uint64_t task_window_size) { +RuntimeArenaLayout runtime_reserve_layout(DeviceArena &arena, uint64_t task_capacity) { RuntimeArenaLayout layout{}; - layout.task_window_size = task_window_size; + layout.task_capacity = task_capacity; // Reservation order is the zone partition (see RuntimeArenaLayout): // everything the device initializes itself, then the one copied range. Each diff --git a/src/a2a3/runtime/host_build_graph/runtime/shared/shared_memory.cpp b/src/a2a3/runtime/host_build_graph/runtime/shared/shared_memory.cpp index 35e3e044fe..a497349736 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/shared/shared_memory.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/shared/shared_memory.cpp @@ -27,10 +27,10 @@ // Size Calculation // ============================================================================= -uint64_t SharedMemoryHandle::calculate_size(uint64_t task_window_size) { - // Total SM size = offset just past the ring's slot_states, from the single - // source of truth for the layout (sm_layout::ring_segment_offsets). - return sm_layout::ring_segment_offsets(task_window_size).end; +uint64_t SharedMemoryHandle::calculate_size(uint64_t max_tasks) { + // Total SM size = offset just past the last segment, from the single + // source of truth for the layout (sm_layout::segment_offsets). + return sm_layout::segment_offsets(max_tasks).end; } // ============================================================================= @@ -41,60 +41,59 @@ void SharedMemoryHandle::setup_pointers(uint64_t pitch) { char *base = (char *)sm_base; header = (SharedMemoryHeader *)base; - // Per-ring descriptors / payloads / slot_states — offsets from the single source - // of truth (sm_layout::ring_segment_offsets), so this setup and the + // descriptors / payloads / slot_states — offsets from the single source + // of truth (sm_layout::segment_offsets), so this setup and the // device-address helpers cannot drift. // // Only the four slot-pitched segments, and no argument pool: the pool extents // differ between the mirror and the image, while these four depend on the pitch // alone. The orchestrator holds the mirror's pool bases; the device resolves an // argument region through the payload's delta and needs no base at all. - auto off = sm_layout::ring_segment_offsets(sm_layout::image_extents({pitch, 0, 0, 0})); - auto &ring = header->ring; - ring.task_descriptors = (TaskDescriptor *)(base + off.descriptors); - ring.task_payloads = (TaskPayload *)(base + off.payloads); - ring.slot_states = (ChipTaskSlotState *)(base + off.slot_states); - ring.completion_flags = (std::atomic *)(base + off.completion_flags); + auto off = sm_layout::segment_offsets(sm_layout::image_extents({pitch, 0, 0, 0})); + auto &tasks = header->tasks; + tasks.task_descriptors = (TaskDescriptor *)(base + off.descriptors); + tasks.task_payloads = (TaskPayload *)(base + off.payloads); + tasks.slot_states = (ChipTaskSlotState *)(base + off.slot_states); + tasks.completion_flags = (std::atomic *)(base + off.completion_flags); } -bool SharedMemoryHandle::init(void *sm_base_arg, uint64_t sm_size_arg, uint64_t task_window_size) { +bool SharedMemoryHandle::init(void *sm_base_arg, uint64_t sm_size_arg, uint64_t max_tasks) { if (!sm_base_arg || sm_size_arg == 0) return false; - const uint64_t mirror_bytes = calculate_size(task_window_size); + const uint64_t mirror_bytes = calculate_size(max_tasks); // The mirror has to stay inside an int32 delta's reach: a payload names its // argument regions that way, and set() leaves a field unbound rather than // truncating, so a pool out of reach would read back as null and be dereferenced // as one. attach_populated makes the same check on the shipped image. // - // Tested before the region's own size, because this rejects the ring capacity - // itself: no region, however large, makes such a window usable. + // Tested before the region's own size, because this rejects `max_tasks` itself: + // no region, however large, makes such a reservation usable. if (mirror_bytes > static_cast(INT32_MAX)) return false; if (sm_size_arg < mirror_bytes) return false; sm_base = sm_base_arg; sm_size = sm_size_arg; is_owner = false; - setup_pointers(task_window_size); - init_header(task_window_size); + setup_pointers(max_tasks); + init_header(); return true; } bool SharedMemoryHandle::attach_populated( - void *sm_base_arg, uint64_t sm_size_arg, uint64_t task_window_size, uint64_t live_slots, uint64_t image_bytes + void *sm_base_arg, uint64_t sm_size_arg, uint64_t max_tasks, uint64_t live_slots, uint64_t image_bytes ) { if (!sm_base_arg || sm_size_arg == 0) return false; - // A pitch above the capacity would name a slot the ring cannot address, and one + // A pitch above `max_tasks` names a slot the host could not have built, and one // below what the host used would place every segment past the descriptors // short. This is the whole contract between the two sides, checked once at // attach rather than trusted. - if (live_slots == 0 || live_slots > task_window_size) return false; + if (live_slots == 0 || live_slots > max_tasks) return false; // The image must at least hold the four slot-pitched segments; anything beyond // that is its argument pools, whose extents are the bind's cursors and are not // recomputable here. The first pool's offset is exactly where those four end. - const uint64_t slots_end = - sm_layout::ring_segment_offsets(sm_layout::image_extents({live_slots, 0, 0, 0})).fanin_pool; + const uint64_t slots_end = sm_layout::segment_offsets(sm_layout::image_extents({live_slots, 0, 0, 0})).fanin_pool; if (image_bytes < slots_end) return false; - // The region holds the compacted image, so that is what has to fit — the ring - // capacity is no longer its size. + // The region holds the compacted image, so that is what has to fit — the + // dimensioned size is no longer its size. if (sm_size_arg < image_bytes) return false; // A slot state names its payload and descriptor by an int32 delta from its own // address, and a payload names its argument regions the same way, so no two @@ -106,12 +105,12 @@ bool SharedMemoryHandle::attach_populated( is_owner = false; setup_pointers(live_slots); // Deliberately NO init_header: the SM already holds the host orchestrator's - // task graph (descriptors, slot states, ring counters). + // task graph (descriptors, slot states, completion flags). return true; } SharedMemoryHandle *SharedMemoryHandle::create_and_init_default(DeviceArena &arena) { - const uint64_t buffer_size = calculate_size(CHIP_TASK_WINDOW_SIZE); + const uint64_t buffer_size = calculate_size(CHIP_DEFAULT_GRAPH_TASKS); const size_t off_handle = arena.reserve(sizeof(SharedMemoryHandle), alignof(SharedMemoryHandle)); const size_t off_buffer = arena.reserve(static_cast(buffer_size), CHIP_ALIGN_SIZE); if (arena.commit() == nullptr) return nullptr; @@ -120,7 +119,7 @@ SharedMemoryHandle *SharedMemoryHandle::create_and_init_default(DeviceArena &are memset(handle, 0, sizeof(*handle)); void *buffer = arena.region_ptr(off_buffer); memset(buffer, 0, static_cast(buffer_size)); - if (!handle->init(buffer, buffer_size, CHIP_TASK_WINDOW_SIZE)) return nullptr; + if (!handle->init(buffer, buffer_size, CHIP_DEFAULT_GRAPH_TASKS)) return nullptr; return handle; } @@ -138,24 +137,21 @@ void SharedMemoryHandle::destroy() { // ============================================================================= // // no need init data in pool, init pool data when used -void SharedMemoryHandle::init_header(uint64_t task_window_size) { - // Flow control (starts at 0) - header->ring.fc.init(); - +void SharedMemoryHandle::init_header() { // Polling completion: -1 = "no task completed yet"; the first task to // complete (local_id 0) advances the watermark to 0. - header->ring.completed_watermark.store(-1, std::memory_order_relaxed); + header->tasks.completed_watermark.store(-1, std::memory_order_relaxed); + + // 0 until run_host_orchestration writes the run's count; init runs before + // orchestration, so the real value is not known here yet. + header->tasks.total_tasks = 0; header->orchestrator_done.store(0, std::memory_order_relaxed); - // Ring layout info - uint64_t offset = CHIP_ALIGN_UP(sizeof(SharedMemoryHeader), CHIP_ALIGN_SIZE); - header->ring.task_window_size = task_window_size; - header->ring.task_window_mask = static_cast(task_window_size - 1); - header->ring.task_descriptors_offset = offset; - offset += CHIP_ALIGN_UP(task_window_size * sizeof(TaskDescriptor), CHIP_ALIGN_SIZE); - offset += CHIP_ALIGN_UP(task_window_size * sizeof(TaskPayload), CHIP_ALIGN_SIZE); - offset += CHIP_ALIGN_UP(task_window_size * sizeof(ChipTaskSlotState), CHIP_ALIGN_SIZE); + // Layout info. The descriptors are the first segment, so their offset is where + // the header's own padded size ends — pitch-independent, unlike every segment + // after them. + header->tasks.task_descriptors_offset = CHIP_ALIGN_UP(sizeof(SharedMemoryHeader), CHIP_ALIGN_SIZE); header->total_size = sm_size; @@ -168,8 +164,8 @@ void SharedMemoryHandle::init_header(uint64_t task_window_size) { // Per-slot init (slot_states.reset_for_reuse() + active_mask, and clearing the // completion flag) happens init-on-write in orch::prepare_task as each slot // [0, total_tasks) is claimed, so the SM init/upload cost tracks the task - // count, not ring capacity. The device reads no slot past total_tasks, so the - // unclaimed tail is left uninitialized. + // count, not the size the table was dimensioned for. The device reads no slot + // past total_tasks, so the unclaimed tail is left uninitialized. } // ============================================================================= @@ -184,13 +180,12 @@ void SharedMemoryHandle::print_layout() { LOG_DEBUG("=== Shared Memory Layout ==="); LOG_DEBUG("Base address: %p", sm_base); LOG_DEBUG("Total size: %" PRIu64 " bytes", h->total_size); - LOG_DEBUG("Ring:"); - LOG_DEBUG(" task_window_size: %" PRIu64, h->ring.task_window_size); + LOG_DEBUG("Task table:"); LOG_DEBUG( - " descriptors_off: %" PRIu64 " (0x%" PRIx64 ")", h->ring.task_descriptors_offset, - h->ring.task_descriptors_offset + " descriptors_off: %" PRIu64 " (0x%" PRIx64 ")", h->tasks.task_descriptors_offset, + h->tasks.task_descriptors_offset ); - LOG_DEBUG(" current_task_idx: %d", h->ring.fc.current_task_index.load(std::memory_order_acquire)); + LOG_DEBUG(" completed_wm: %d", h->tasks.completed_watermark.load(std::memory_order_acquire)); LOG_DEBUG("orchestrator_done: %d", h->orchestrator_done.load(std::memory_order_acquire)); LOG_DEBUG("Error state:"); LOG_DEBUG(" orch_error_code: %d", h->orch_error_code.load(std::memory_order_relaxed)); @@ -204,28 +199,13 @@ bool SharedMemoryHandle::validate() { if (!sm_base) return false; if (!header) return false; - SharedMemoryHeader *h = header; - - if (!h->ring.fc.validate(this)) return false; - - return true; -} - -bool ChipRingFlowControl::validate(SharedMemoryHandle *handle) const { - if (!handle) return false; - if (!handle->header) return false; - - const SharedMemoryHeader *h = handle->header; + const SharedMemoryHeader *h = header; // Check that offsets are within bounds - if (h->ring.task_descriptors_offset >= h->total_size) return false; + if (h->tasks.task_descriptors_offset >= h->total_size) return false; // Check pointer alignment - if ((uintptr_t)h->ring.task_descriptors % CHIP_ALIGN_SIZE != 0) return false; - - // Check flow control pointer sanity - int32_t current = current_task_index.load(std::memory_order_acquire); - if (current < 0) return false; + if ((uintptr_t)h->tasks.task_descriptors % CHIP_ALIGN_SIZE != 0) return false; return true; } diff --git a/src/a2a3/runtime/host_build_graph/runtime/shared/tensormap.cpp b/src/a2a3/runtime/host_build_graph/runtime/shared/tensormap.cpp index 82799bff20..f75320d63d 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/shared/tensormap.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/shared/tensormap.cpp @@ -54,40 +54,39 @@ uint64_t g_insert_count = 0; * Allocate the four arrays and reset the map to empty. Clears the bucket heads * and the per-task entry heads and resets the pool cursors (next_entry_idx / * free_num); the entry pool is left uninitialized and initialized on write by - * new_entry(), so this is O(num_buckets + task_window_size), not O(pool_size). + * new_entry(), so this is O(num_buckets + max_tasks), not O(pool_size). * * `new (std::nothrow) T[n]` rather than a vector: a vector would value-initialize * the whole entry pool, which is megabytes of zeroing the init-on-write path * exists to avoid, and nothrow keeps the failure reportable to a caller that * still has an alternative path. */ -bool ChipTensorMap::init(int32_t new_num_buckets, int32_t new_pool_size, int32_t new_task_window_size) { - // num_buckets must be a power of two for the hash truncation to work, and - // task_window_size for the task-chain slot mask. +bool ChipTensorMap::init(int32_t new_num_buckets, int32_t new_pool_size, int32_t new_max_tasks) { + // num_buckets must be a power of two for the hash truncation to work. The + // task-chain count needs no such property: a task id indexes its chain directly. always_assert(new_num_buckets > 0 && (new_num_buckets & (new_num_buckets - 1)) == 0); - always_assert(new_task_window_size > 0 && (new_task_window_size & (new_task_window_size - 1)) == 0); + always_assert(new_max_tasks > 0); always_assert(new_pool_size > 0); buckets.reset(new (std::nothrow) ChipTensorMapEntry *[new_num_buckets]); entry_pool.reset(new (std::nothrow) ChipTensorMapEntry[new_pool_size]); free_entry_list.reset(new (std::nothrow) ChipTensorMapEntry *[new_pool_size]); - task_entry_heads.reset(new (std::nothrow) ChipTensorMapEntry *[new_task_window_size]); + task_entry_heads.reset(new (std::nothrow) ChipTensorMapEntry *[new_max_tasks]); if (buckets == nullptr || entry_pool == nullptr || free_entry_list == nullptr || task_entry_heads == nullptr) { LOG_ERROR( - "TensorMap init failed (buckets=%d, pool=%d, window=%d)", new_num_buckets, new_pool_size, - new_task_window_size + "TensorMap init failed (buckets=%d, pool=%d, max_tasks=%d)", new_num_buckets, new_pool_size, new_max_tasks ); return false; } num_buckets = new_num_buckets; pool_size = new_pool_size; - task_window_size = new_task_window_size; + max_tasks = new_max_tasks; for (int32_t i = 0; i < num_buckets; i++) { buckets[i] = nullptr; } - for (int32_t i = 0; i < task_window_size; i++) { + for (int32_t i = 0; i < max_tasks; i++) { task_entry_heads[i] = nullptr; } @@ -111,7 +110,7 @@ void ChipTensorMap::reset() { for (int32_t i = 0; i < num_buckets; i++) { buckets[i] = nullptr; } - for (int32_t i = 0; i < task_window_size; i++) { + for (int32_t i = 0; i < max_tasks; i++) { task_entry_heads[i] = nullptr; } // Entries are reached only through a bucket chain, and every chain is now empty, so @@ -121,8 +120,8 @@ void ChipTensorMap::reset() { free_num = 0; } -bool ChipTensorMap::init_default(int32_t new_task_window_size) { - return init(CHIP_TENSORMAP_NUM_BUCKETS, CHIP_TENSORMAP_POOL_SIZE, new_task_window_size); +bool ChipTensorMap::init_default(int32_t new_max_tasks) { + return init(CHIP_TENSORMAP_NUM_BUCKETS, CHIP_TENSORMAP_POOL_SIZE, new_max_tasks); } // ============================================================================= diff --git a/src/a2a3/runtime/host_build_graph/runtime/shared_memory.h b/src/a2a3/runtime/host_build_graph/runtime/shared_memory.h index 4e441af4ce..13d5117930 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/shared_memory.h +++ b/src/a2a3/runtime/host_build_graph/runtime/shared_memory.h @@ -9,13 +9,13 @@ * ----------------------------------------------------------------------------------------------------------- */ /** - * PTO Runtime2 - Shared Memory Layout + * Shared Memory Layout * * Defines the shared memory structure for Orchestrator-Scheduler communication. * - * Memory Layout (single ring): + * Memory Layout: * +---------------------------+ - * | SharedMemoryHeader | (flow control + sync) + * | SharedMemoryHeader | (completion watermark + sync + error state) * +---------------------------+ * | TaskDescriptor[] | * | TaskPayload[] | @@ -24,8 +24,8 @@ * * Design principles: * - Only data needed for Orchestrator<->Scheduler communication is here - * - TensorMap, scope_stack, ready_queues, dep_pool are in private memory - * - Flow control via atomic counters/flags (no locks needed for single-word R/W) + * - TensorMap, scope_stack and ready_queues are in private memory + * - Synchronization via atomic counters/flags (no locks needed for single-word R/W) * * Based on: docs/RUNTIME_LOGIC.md */ @@ -47,37 +47,19 @@ struct SharedMemoryHandle; /** - * Per-ring flow control state in shared memory. - * Written/read by Orchestrator and Scheduler for synchronization. - */ -struct alignas(64) ChipRingFlowControl { - // Written by Orchestrator, read by Scheduler. There is no reverse channel: - // the ring is whole-graph-resident, so the scheduler never reclaims task - // slots and has nothing to publish back. - alignas(64) std::atomic current_task_index; // Task ring head (next to allocate) - - // Per-boot SM reset. TaskAllocator::init() seeds its private - // local_task_id_ to 0 *without* dereferencing current_task_index — it - // relies on this reset running on every AICPU boot so 0 stays in sync. If - // you ever change the initial fc value or the boot ordering, update - // TaskAllocator::init (ring_buffer.h) in the same change, or - // submit IDs will be off by the divergence. - void init() { current_task_index.store(0, std::memory_order_relaxed); } - - bool validate(SharedMemoryHandle *handle) const; -}; - -static_assert(sizeof(ChipRingFlowControl) == 64, "ChipRingFlowControl must be exactly one cache line (64B)"); - -/** - * Per-ring shared memory header section. + * The task table's header in shared memory. * - * Groups flow-control, layout info, and per-ring data pointers for a single ring. - * Pointers are host-side only (set by setup_pointers, invalid on device). + * Groups the completion watermark, layout info, and the pointers to the four + * slot-pitched segments. Pointers are host-side only (set by setup_pointers, + * invalid on device). + * + * The run's task total sits here too, as a plain scalar. The graph is complete + * before the device starts, so the host writes it once into the mirror after + * orchestration and the restack ships it with the rest of the header; the device + * only ever reads it. Nothing publishes it incrementally, so it needs neither an + * atomic nor a cache line of its own. */ -struct alignas(64) SharedMemoryRingHeader { - ChipRingFlowControl fc; - +struct alignas(64) SharedMemoryTaskHeader { // Highest task_id such that every task with id in [0, completed_watermark] // has its completion_flags byte set. Advanced over the full contiguous // completed prefix at task-completion time (on_mixed_task_complete). The host @@ -87,11 +69,9 @@ struct alignas(64) SharedMemoryRingHeader { alignas(64) std::atomic completed_watermark; // Layout metadata (set once at init) - alignas(64) uint64_t task_window_size; - int32_t task_window_mask; - uint64_t task_descriptors_offset; // Offset from SM base, in bytes + alignas(64) uint64_t task_descriptors_offset; // Offset from SM base, in bytes - // Per-ring data pointers (host-side, set by setup_pointers) + // Segment pointers (host-side, set by setup_pointers) TaskDescriptor *task_descriptors; TaskPayload *task_payloads; ChipTaskSlotState *slot_states; @@ -100,25 +80,31 @@ struct alignas(64) SharedMemoryRingHeader { // 0 = pending, 1 = task fully COMPLETED. Writer = the task's completer at // on_mixed_task_complete; reader = consumer fanin polling (is_completion_flag_set). // Cleared per-slot in orch::prepare_task as each slot is claimed. Indexed by - // local_id & task_window_mask. + // local task id, like every other segment. std::atomic *completion_flags; + // Tasks this run submitted, i.e. the slot count the four segments above are + // pitched to. Written once by the host after orchestration (run_host_orchestration) + // and read-only from then on, so it packs into the padding rather than taking a + // line of its own. Bounds the completed_watermark walk: no slot at or above it was + // claimed, and the bytes past completion_flags[total_tasks - 1] are not flags. + int32_t total_tasks; + bool is_completion_flag_set(int32_t local_id, std::memory_order order = std::memory_order_acquire) const { - return completion_flags[local_id & task_window_mask].load(order) != 0; + return completion_flags[local_id].load(order) != 0; } void set_completion_flag(int32_t local_id, std::memory_order order = std::memory_order_release) const { - completion_flags[local_id & task_window_mask].store(1, order); + completion_flags[local_id].store(1, order); } // set completion flag first before updating the watermark (logic requirement) void update_completed_watermark() { int32_t curr_watermark = completed_watermark.load(std::memory_order_acquire); - const int32_t submitted = fc.current_task_index.load(std::memory_order_acquire); int32_t next = curr_watermark; while (true) { - while (next + 1 < submitted && is_completion_flag_set(next + 1)) { + while (next + 1 < total_tasks && is_completion_flag_set(next + 1)) { ++next; } if (next == curr_watermark) { @@ -137,37 +123,30 @@ struct alignas(64) SharedMemoryRingHeader { } } - int32_t get_slot_by_task_id(int32_t local_task_id) { return local_task_id & task_window_mask; } - - TaskDescriptor &get_task_by_slot(int32_t slot) { return task_descriptors[slot]; } + // A task id is its own slot index, so every segment is indexed directly. + TaskDescriptor &get_task_by_task_id(int32_t local_id) { return task_descriptors[local_id]; } - TaskDescriptor &get_task_by_task_id(int32_t local_id) { return task_descriptors[get_slot_by_task_id(local_id)]; } + // No get_payload_by_task_id here: a payload is reached through its slot + // state's `payload` delta, which the image's restack rebinds to the real + // address. - // No get_payload_by_slot / get_payload_by_task_id here: a payload is reached - // through its slot state's `payload` delta, which the image's restack rebinds to - // the real address. - - ChipTaskSlotState &get_slot_state_by_slot(int32_t slot) { return slot_states[slot]; } - - ChipTaskSlotState &get_slot_state_by_task_id(int32_t local_id) { - return slot_states[get_slot_by_task_id(local_id)]; - } + ChipTaskSlotState &get_slot_state_by_task_id(int32_t local_id) { return slot_states[local_id]; } }; -static_assert(sizeof(SharedMemoryRingHeader) == 192, "SharedMemoryRingHeader layout drift"); +static_assert(sizeof(SharedMemoryTaskHeader) == 128, "SharedMemoryTaskHeader layout drift"); static_assert( - offsetof(SharedMemoryRingHeader, task_descriptors_offset) == 144, - "SharedMemoryRingHeader task_descriptors_offset layout drift" + offsetof(SharedMemoryTaskHeader, task_descriptors_offset) == 64, + "SharedMemoryTaskHeader task_descriptors_offset layout drift" ); /** * Shared memory header structure * - * Contains per-ring flow control and global layout information. + * Contains the task table's header plus the run's global sync and error state. */ struct alignas(CHIP_ALIGN_SIZE) SharedMemoryHeader { - // === RING FLOW CONTROL + LAYOUT INFO (single ring, set once at init) === - SharedMemoryRingHeader ring; + // === TASK TABLE HEADER (set once at init) === + SharedMemoryTaskHeader tasks; // === GLOBAL FIELDS === std::atomic orchestrator_done; // Flag: orchestration complete @@ -188,9 +167,9 @@ struct alignas(CHIP_ALIGN_SIZE) SharedMemoryHeader { std::atomic sched_error_thread; // Thread index of last error writer }; -static_assert(sizeof(SharedMemoryHeader) == 256, "SharedMemoryHeader layout drift"); -static_assert(offsetof(SharedMemoryHeader, total_size) == 200, "SharedMemoryHeader total_size layout drift"); -static_assert(offsetof(SharedMemoryHeader, orch_error_code) == 208, "SharedMemoryHeader orch_error_code layout drift"); +static_assert(sizeof(SharedMemoryHeader) == 192, "SharedMemoryHeader layout drift"); +static_assert(offsetof(SharedMemoryHeader, total_size) == 136, "SharedMemoryHeader total_size layout drift"); +static_assert(offsetof(SharedMemoryHeader, orch_error_code) == 144, "SharedMemoryHeader orch_error_code layout drift"); // ============================================================================= // Shared Memory Handle @@ -211,10 +190,12 @@ struct SharedMemoryHandle { // === Static helpers === - static uint64_t calculate_size(uint64_t task_window_size); + // Bytes an SM image spans when dimensioned for `max_tasks` slots — the count + // the bind resolved from runtime_env.ring_task_window. + static uint64_t calculate_size(uint64_t max_tasks); // UT convenience: reserve wrapper + sm_base on `arena`, commit, and init using - // default CHIP_TASK_WINDOW_SIZE. Only valid when the arena is otherwise empty + // default CHIP_DEFAULT_GRAPH_TASKS. Only valid when the arena is otherwise empty // (the call performs the single commit). All memory is owned by the arena — // caller must not call destroy(). static SharedMemoryHandle *create_and_init_default(DeviceArena &arena); @@ -223,41 +204,39 @@ struct SharedMemoryHandle { // In-place init for caller-provided wrapper storage (e.g. a region carved // out of a DeviceArena). Sets is_owner = false, calls setup_pointers and - // init_header. Returns false when `sm_size` is too small for the requested - // `task_window_size`. - bool init(void *sm_base, uint64_t sm_size, uint64_t task_window_size); + // init_header. Returns false when `sm_size` is too small for `max_tasks`. + bool init(void *sm_base, uint64_t sm_size, uint64_t max_tasks); // Attach to an ALREADY-populated shared memory region: point the handle and - // every ring header's data pointers (descriptors / payloads / slot_states) - // at `sm_base`, but do NOT reset the flow-control counters / slot states. + // the task header's segment pointers (descriptors / payloads / slot_states) + // at `sm_base`, but do NOT reset the watermark / slot states. // Used by host_build_graph host-orch, where the host orchestrator populated // the SM and H2D'd it; the device must re-point at its own SM base without // wiping the contents (unlike init, which also resets the header). // // `live_slots` is the pitch the uploaded arrays were laid out with — the - // number of slots the host actually submitted, not the ring capacity. It must - // match what the host used or every segment past the descriptors resolves to - // the wrong address, so both sides derive it from the same submitted count. - // The capacity and mask in the header are unchanged, and `local_id & mask` - // yields `local_id`, which is below `live_slots` for every ring task. + // number of slots the host actually submitted, not the `max_tasks` the mirror + // was dimensioned for. It must match what the host used or every segment past + // the descriptors resolves to the wrong address, so both sides derive it from + // the same count. + // Every task id is below it, and indexes its slot directly. // // `image_bytes` is what the host shipped, pools included. The device cannot // recompute it — the pool extents are the bind's cursors, which only the host saw // — and it does not need to: a payload names its argument regions by delta, so no // pool base is resolved here. The value bounds the region and checks the int32 // delta reach. - bool attach_populated( - void *sm_base, uint64_t sm_size, uint64_t task_window_size, uint64_t live_slots, uint64_t image_bytes - ); + bool + attach_populated(void *sm_base, uint64_t sm_size, uint64_t max_tasks, uint64_t live_slots, uint64_t image_bytes); void destroy(); void print_layout(); bool validate(); private: - void init_header(uint64_t task_window_size); + void init_header(); // `pitch` is the slot count the arrays are dimensioned for. init passes the - // ring capacity (the mirror the orchestrator writes into); attach_populated + // mirror's reservation (what the orchestrator writes into); attach_populated // passes the submitted count (the compacted image that shipped). void setup_pointers(uint64_t pitch); }; @@ -267,7 +246,7 @@ struct SharedMemoryHandle { // ============================================================================= // // When the host pre-builds a runtime-arena image, it needs the device-side -// addresses of several SM sub-fields (ring flow-control counters, +// addresses of several SM sub-fields (the task header, // task_descriptors arrays, orch_error_code) so it can wire them into the // orchestrator / scheduler init_data path without dereferencing the SM — // the SM lives in device memory and cannot be touched from host. @@ -284,29 +263,22 @@ inline std::atomic *orch_error_code_addr(void *sm_dev_base) noexcept { ); } -inline SharedMemoryRingHeader *ring_header_addr(void *sm_dev_base) noexcept { - return reinterpret_cast( - static_cast(sm_dev_base) + offsetof(SharedMemoryHeader, ring) - ); -} - -inline std::atomic *ring_current_task_index_addr(void *sm_dev_base) noexcept { - return reinterpret_cast *>( - reinterpret_cast(ring_header_addr(sm_dev_base)) + offsetof(SharedMemoryRingHeader, fc) + - offsetof(ChipRingFlowControl, current_task_index) +inline SharedMemoryTaskHeader *task_header_addr(void *sm_dev_base) noexcept { + return reinterpret_cast( + static_cast(sm_dev_base) + offsetof(SharedMemoryHeader, tasks) ); } -// Byte offsets (from the SM base) of the ring's segments. The layout is: header, then +// Byte offsets (from the SM base) of the image's segments. The layout is: header, then // descriptors -> payloads -> slot_states -> completion_flags -> the three argument -// pools, every segment CHIP_ALIGN_UP-padded. RingImageExtents dimensions them: the +// pools, every segment CHIP_ALIGN_UP-padded. ImageExtents dimensions them: the // mirror for the worst case the API allows, the image for what this bind holds, which // is what makes the live prefixes contiguous and the upload one copy. // // The pools sit last because nothing on the device resolves a segment past // completion_flags: a payload names its argument regions by delta, so the four // slot-pitched offsets are all the attach path computes. -struct ChipRingSegmentOffsets { +struct SegmentOffsets { uint64_t descriptors; uint64_t payloads; uint64_t slot_states; @@ -318,22 +290,22 @@ struct ChipRingSegmentOffsets { }; // How many slots and how many pool elements a layout is dimensioned for. -struct RingImageExtents { +struct ImageExtents { uint64_t slots; uint64_t fanin_elems; uint64_t tensor_elems; uint64_t scalar_elems; }; -// The mirror the orchestrator writes into: the ring capacity, every pool sized so the -// worst case cannot overflow a bump — task_window tasks each at their full cap. That +// The mirror the orchestrator writes into: `max_tasks` slots, every pool sized so the +// worst case cannot overflow a bump — max_tasks tasks each at their full cap. That // bound is what lets prepare_task advance a cursor with no capacity check. -inline RingImageExtents mirror_extents(uint64_t task_window_size) noexcept { - return RingImageExtents{ - task_window_size, - task_window_size * CHIP_MAX_FANIN, - task_window_size * MAX_TENSOR_ARGS, - task_window_size * MAX_SCALAR_ARGS, +inline ImageExtents mirror_extents(uint64_t max_tasks) noexcept { + return ImageExtents{ + max_tasks, + max_tasks * CHIP_MAX_FANIN, + max_tasks * MAX_TENSOR_ARGS, + max_tasks * MAX_SCALAR_ARGS, }; } @@ -342,9 +314,9 @@ inline RingImageExtents mirror_extents(uint64_t task_window_size) noexcept { // `sm_base`) and the device-address helpers below (which add `sm_dev_base`). Adding // or reordering a segment is a one-line edit here; every consumer follows // automatically, so the layout walk can never silently disagree across call sites. -inline ChipRingSegmentOffsets ring_segment_offsets(const RingImageExtents &e) noexcept { +inline SegmentOffsets segment_offsets(const ImageExtents &e) noexcept { uint64_t off = CHIP_ALIGN_UP(sizeof(SharedMemoryHeader), CHIP_ALIGN_SIZE); - ChipRingSegmentOffsets o{}; + SegmentOffsets o{}; o.descriptors = off; off += CHIP_ALIGN_UP(e.slots * sizeof(TaskDescriptor), CHIP_ALIGN_SIZE); o.payloads = off; @@ -363,8 +335,8 @@ inline ChipRingSegmentOffsets ring_segment_offsets(const RingImageExtents &e) no return o; } -inline ChipRingSegmentOffsets ring_segment_offsets(uint64_t task_window_size) noexcept { - return ring_segment_offsets(mirror_extents(task_window_size)); +inline SegmentOffsets segment_offsets(uint64_t max_tasks) noexcept { + return segment_offsets(mirror_extents(max_tasks)); } // Every per-task region starts on a cache line, which TaskPayload::init's @@ -396,8 +368,8 @@ struct BindUsage { uint64_t scalar_elems; }; -inline RingImageExtents image_extents(const BindUsage &used) noexcept { - return RingImageExtents{ +inline ImageExtents image_extents(const BindUsage &used) noexcept { + return ImageExtents{ live_slot_pitch(used.submitted_tasks), used.fanin_elems, used.tensor_elems, @@ -432,16 +404,16 @@ inline uint64_t rebased_heap_addr(uint64_t addr, const HeapRebase &rebase) noexc return rebase.real_base + offset; } -// Restack the live prefix of every ring segment from the ring-pitched mirror the +// Restack the live prefix of every segment from the mirror the // orchestrator wrote into an image dimensioned for what this bind holds, where the // prefixes are contiguous and can travel as one copy. // // `out_base` must be CHIP_ALIGN_SIZE-aligned and hold -// `ring_segment_offsets(image_extents(used)).end` bytes. Returns that byte count. +// `segment_offsets(image_extents(used)).end` bytes. Returns that byte count. // // Three things the restack has to fix up, all because the image is not the mirror: // -// - the ring header's data pointers name the mirror's arrays, so they leave as +// - the task header's segment pointers name the mirror's arrays, so they leave as // null rather than carrying host addresses into device memory (the device // resolves them in attach_populated); // - a slot state names its payload and descriptor, and a payload names its three @@ -463,27 +435,27 @@ inline uint64_t rebased_heap_addr(uint64_t addr, const HeapRebase &rebase) noexc // falls inside the window. Orchestration must therefore pass a runtime-created // buffer as the tensor it got back, never as a scalar carrying its address. inline uint64_t compact_live_image( - const char *mirror_base, uint64_t task_window_size, const BindUsage &used, const HeapRebase &rebase, char *out_base + const char *mirror_base, uint64_t max_tasks, const BindUsage &used, const HeapRebase &rebase, char *out_base ) noexcept { // The mirror is dimensioned for the worst case, so a live count or a cursor past // it reads beyond the segment it is copying from and ships a corrupt image. // attach_populated tests the slot bound again on the device side. - const RingImageExtents mirror = mirror_extents(task_window_size); + const ImageExtents mirror = mirror_extents(max_tasks); always_assert(used.submitted_tasks <= mirror.slots); always_assert(used.fanin_elems <= mirror.fanin_elems); always_assert(used.tensor_elems <= mirror.tensor_elems); always_assert(used.scalar_elems <= mirror.scalar_elems); - const ChipRingSegmentOffsets from = ring_segment_offsets(mirror); - const ChipRingSegmentOffsets to = ring_segment_offsets(image_extents(used)); + const SegmentOffsets from = segment_offsets(mirror); + const SegmentOffsets to = segment_offsets(image_extents(used)); // The header and the descriptors offset are pitch-independent, so the header // lands where it already was. std::memcpy(out_base, mirror_base, to.descriptors); - auto &out_ring = reinterpret_cast(out_base)->ring; - out_ring.task_descriptors = nullptr; - out_ring.task_payloads = nullptr; - out_ring.slot_states = nullptr; - out_ring.completion_flags = nullptr; + auto &out_tasks = reinterpret_cast(out_base)->tasks; + out_tasks.task_descriptors = nullptr; + out_tasks.task_payloads = nullptr; + out_tasks.slot_states = nullptr; + out_tasks.completion_flags = nullptr; const uint64_t nt = used.submitted_tasks; std::memcpy(out_base + to.descriptors, mirror_base + from.descriptors, nt * sizeof(TaskDescriptor)); diff --git a/src/a2a3/runtime/host_build_graph/runtime/ring_buffer.h b/src/a2a3/runtime/host_build_graph/runtime/task_allocator.h similarity index 68% rename from src/a2a3/runtime/host_build_graph/runtime/ring_buffer.h rename to src/a2a3/runtime/host_build_graph/runtime/task_allocator.h index 6a2fe3f0ac..8a6c332f9d 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/ring_buffer.h +++ b/src/a2a3/runtime/host_build_graph/runtime/task_allocator.h @@ -9,11 +9,9 @@ * ----------------------------------------------------------------------------------------------------------- */ /** - * PTO Runtime2 - Ring Buffer Data Structures - * * TaskAllocator - Unified task slot + output buffer allocation - * - Combines task ring (slot allocation) and heap ring (output buffer allocation) - * - O(1) forward bump allocation for both task slots and heap buffers + * - Combines task-table slot allocation and heap output-buffer allocation + * - O(1) forward bump allocation for both * - Neither resource is reclaimed during a run, so exhaustion of either is a * capacity error reported on the spot, never back-pressure to wait on * @@ -37,41 +35,41 @@ /** * Unified task slot + heap buffer allocator. * - * Since task and heap are always allocated together and the orchestrator is - * single-threaded, both pointers (task index, heap top) are tracked locally - * and published to shared memory via plain store — no fetch_add or CAS needed. + * Task ids and heap bytes are handed out by two local bump counters. The + * orchestrator is single-threaded and nothing outside it observes either + * counter, so neither needs an atomic or a published copy: the run's task total + * is read back through active_count() once orchestration ends. * * The alloc() method checks both resources BEFORE committing to either, * eliminating the need for rollback on partial failure. * * host_build_graph is whole-graph-resident: the device runs only after the host * has built the entire graph, so no task slot or heap byte is ever reclaimed - * while allocation is in progress. Both rings are therefore forward-only, and a - * request that does not fit can never become satisfiable by waiting — alloc() + * while allocation is in progress. Both counters are therefore forward-only, and + * a request that does not fit can never become satisfiable by waiting — alloc() * reports the exhausted resource and fails on the spot. + * + * A task id is also its slot index: ids are capped at `capacity` and never + * recycled, so the task table is a flat array indexed by id, with no wrap. */ class TaskAllocator { public: /** - * Initialize the allocator with task ring and heap ring resources. + * Initialize the allocator with its task capacity and heap resources. * * All pointer arguments are device addresses (live in SM / GM heap); this * function only stores them, no dereferences, so it is safe to invoke * from host code that constructs a prebuilt arena image. * - * The ring starts at task id 0, matching the SM flow-control counter that - * current_index_ptr points at (ChipRingFlowControl::init() runs on the AICPU - * during SM reset), so local_task_id_ stays in sync without reading the SM. - * Because ids are never reclaimed, alloc() caps them at window_size — they - * cannot run away toward INT32_MAX. + * `capacity` is the number of task slots the caller's task table holds — what + * the bind resolved from runtime_env.ring_task_window, defaulting to + * CHIP_DEFAULT_GRAPH_TASKS. It need not be a power of two: a task id indexes + * its slot directly, so nothing masks with it. Because ids are never + * reclaimed, alloc() caps them at `capacity` — they cannot run away toward + * INT32_MAX. */ - void init( - int32_t window_size, std::atomic *current_index_ptr, void *heap_base, uint64_t heap_size, - std::atomic *error_code_ptr - ) { - window_size_ = window_size; - window_mask_ = window_size - 1; - current_index_ptr_ = current_index_ptr; + void init(int32_t capacity, void *heap_base, uint64_t heap_size, std::atomic *error_code_ptr) { + capacity_ = capacity; heap_base_ = heap_base; heap_size_ = heap_size; error_code_ptr_ = error_code_ptr; @@ -90,9 +88,8 @@ class TaskAllocator { /** * Allocate a task slot and its associated output buffer in one call. * - * Both task index and heap top are maintained as local counters and - * published to shared memory only on success. Since the orchestrator is - * single-threaded, no CAS or fetch_add is needed — just check-then-commit. + * Both the task id and the heap top are local counters, so this is a plain + * check-then-commit with no atomic and no rollback. * * A fatal latched elsewhere short-circuits the allocation: the caller maps * the failed result to orch_mark_fatal without overwriting the first code. @@ -105,21 +102,21 @@ class TaskAllocator { output_size > 0 ? CHIP_ALIGN_UP(static_cast(output_size), CHIP_ALIGN_SIZE) : 0; if (error_code_ptr_ != nullptr && error_code_ptr_->load(std::memory_order_acquire) != SIMPLER_ERROR_NONE) { - return {-1, -1, nullptr, nullptr}; + return {-1, nullptr, nullptr}; } // Check both resources; commit only if both are available. - if (local_task_id_ >= window_size_) { + if (local_task_id_ >= capacity_) { report_capacity_exhausted(/*heap_blocked=*/false, aligned_size); - return {-1, -1, nullptr, nullptr}; + return {-1, nullptr, nullptr}; } void *heap_ptr = try_bump_heap(aligned_size); if (heap_ptr == nullptr) { report_capacity_exhausted(/*heap_blocked=*/true, aligned_size); - return {-1, -1, nullptr, nullptr}; + return {-1, nullptr, nullptr}; } - int32_t task_id = commit_task(); - return {task_id, task_id & window_mask_, heap_ptr, static_cast(heap_ptr) + aligned_size}; + int32_t task_id = local_task_id_++; + return {task_id, heap_ptr, static_cast(heap_ptr) + aligned_size}; } bool reserve_deferred_heap(int32_t output_size, void **packed_base, void **packed_end) { @@ -140,13 +137,13 @@ class TaskAllocator { // State queries // ========================================================================= - // Nothing retires during a run, so every task allocated so far is still - // live and the ring's head doubles as its occupancy. + // Nothing retires during a run, so every task allocated so far is still live + // and the next id doubles as the occupancy. Once orchestration has ended this + // is therefore also the run's task total, and the only place it is read from: + // no copy of the count is published anywhere else. int32_t active_count() const { return local_task_id_; } - int32_t task_head() const { return local_task_id_; } - - int32_t window_size() const { return window_size_; } + int32_t capacity() const { return capacity_; } uint64_t heap_available() const { return heap_size_ - heap_top_; } @@ -155,10 +152,8 @@ class TaskAllocator { uint64_t heap_used_bytes() const { return heap_top_; } private: - // --- Task Ring --- - int32_t window_size_ = 0; - int32_t window_mask_ = 0; - std::atomic *current_index_ptr_ = nullptr; + // --- Task table --- + int32_t capacity_ = 0; // --- Heap --- void *heap_base_ = nullptr; @@ -175,16 +170,6 @@ class TaskAllocator { // Internal helpers // ========================================================================= - /** - * Commit a task slot: bump local counter and publish to shared memory. - * Must only be called after space check has passed. - */ - int32_t commit_task() { - int32_t task_id = local_task_id_++; - current_index_ptr_->store(local_task_id_, std::memory_order_release); - return task_id; - } - /** * Bump the heap pointer for the given allocation size. * Returns the allocated pointer, or nullptr if insufficient space. @@ -212,34 +197,36 @@ class TaskAllocator { * Nothing is reclaimed during a run, so this is a sizing verdict with no wait * that could still succeed. * - * The task window is a configured capacity, so its branch is the one a real - * bind reaches. The heap is not configured: a bind hands this allocator the - * whole HEAP_VIRTUAL_CAPACITY span and commits the device region afterwards, so - * a graph that does not fit the device fails at that commit and not here. The - * heap branch stays because the allocator is also constructed directly, against - * a small heap, by the unit tests that cover this report. + * The task capacity is a configured size, so its branch is the one a real bind + * reaches. The heap is not configured: a bind hands + * this allocator the whole HEAP_VIRTUAL_CAPACITY span and commits the device + * region afterwards, so a graph that does not fit the device fails at that + * commit and not here. The heap branch stays because the allocator is also + * constructed directly, against a small heap, by the unit tests that cover + * this report. */ void report_capacity_exhausted(bool heap_blocked, uint64_t requested_bytes) { LOG_ERROR("========================================"); if (heap_blocked) { LOG_ERROR("FATAL: Graph Heap Exhausted!"); } else { - LOG_ERROR("FATAL: Task Window Exhausted!"); + LOG_ERROR("FATAL: Graph Too Large!"); } LOG_ERROR("========================================"); LOG_ERROR("The whole graph must fit at once; nothing is reclaimed mid-run."); - LOG_ERROR(" Task window: used=%d/%d", local_task_id_, window_size_); + LOG_ERROR(" Tasks: used=%d/%d", local_task_id_, capacity_); LOG_ERROR( - " Graph heap: used=%" PRIu64 "/%" PRIu64 ", available=%" PRIu64, heap_top_, heap_size_, heap_available() + " Graph heap: used=%" PRIu64 "/%" PRIu64 ", available=%" PRIu64, heap_top_, heap_size_, heap_available() ); - LOG_ERROR(" Requested: %" PRIu64 " bytes + 1 task slot", requested_bytes); + LOG_ERROR(" Requested: %" PRIu64 " bytes + 1 task slot", requested_bytes); LOG_ERROR("Solution:"); if (heap_blocked) { LOG_ERROR(" Shrink the graph's intermediate tensors; this heap has no configuration knob"); } else { LOG_ERROR( - " Increase task window (current: %d); CallConfig.runtime_env.ring_task_window= (e.g. %d)", - window_size_, window_size_ * 2 + " Raise the task capacity (current: %d) via CallConfig.runtime_env.ring_task_window, or shrink " + "the graph", + capacity_ ); } LOG_ERROR("========================================"); diff --git a/src/a2a3/runtime/host_build_graph/runtime/tensormap.h b/src/a2a3/runtime/host_build_graph/runtime/tensormap.h index 187194131e..0ad8d89a84 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/tensormap.h +++ b/src/a2a3/runtime/host_build_graph/runtime/tensormap.h @@ -361,12 +361,10 @@ struct ChipTensorMap { int32_t next_entry_idx{0}; // id when next entry insert int32_t free_num{0}; // free entry number in entry pool - // Per-task entry tracking for O(1) unlinking of covered producers. - // Indexed by [local_id & (task_window_size - 1)] + // Per-task entry tracking for O(1) unlinking of covered producers. A task id + // is its own slot index, so this is indexed by local_id directly. std::unique_ptr task_entry_heads; - int32_t task_window_size{0}; // Task window size (for slot masking) - - uint32_t get_task_local_id_slot(uint32_t task_local_id) const { return task_local_id & (task_window_size - 1); } + int32_t max_tasks{0}; // Slots task_entry_heads is dimensioned for // Pool occupancy, read by the tensormap-exhaustion diagnostic and by the // Graph recording's capacity precheck. @@ -429,32 +427,32 @@ struct ChipTensorMap { /** * Allocate the four arrays and leave the map empty. num_buckets must be a - * power of two; task_window_size must be a power of two, since a producer's - * task chain is selected by `local_id & (task_window_size - 1)`. + * power of two; `new_max_tasks` is the number of task chains to reserve, one + * per task id the caller can place. * * Returns false when an allocation fails, so a caller that still has an * alternative path can take it rather than proceed without a hazard map. * - * Clearing is O(num_buckets + task_window_size), not O(pool_size): the entry + * Clearing is O(num_buckets + max_tasks), not O(pool_size): the entry * pool is left uninitialized and new_entry() puts each slot into the clean * unlinked state on first use, and free_entry_list is a stack meaningful only * below free_num. */ - bool init(int32_t new_num_buckets, int32_t new_pool_size, int32_t new_task_window_size); + bool init(int32_t new_num_buckets, int32_t new_pool_size, int32_t new_max_tasks); /** * Empty an already-initialized map without touching its allocations. * * Same post-state as init(): every bucket and task-chain head null, both pool * cursors at zero, the entry pool left to init-on-write. Costs - * O(num_buckets + task_window_size) stores against pages that are already + * O(num_buckets + max_tasks) stores against pages that are already * resident, where init() pays the allocation and the first touch of each. A * caller that records one body after another on the same map uses this. * - * Keeps the bucket count and task window init() reserved. Taking a new window - * here would have to be checked against the reserved length rather than the - * current one, and nothing tracks the reserved length once a smaller window has - * been set -- so the sizes stay init()'s and this takes no argument. + * Keeps the bucket count and task-chain count init() reserved. Taking a new + * count here would have to be checked against the reserved length rather than + * the current one, and nothing tracks the reserved length once a smaller count + * has been set -- so the sizes stay init()'s and this takes no argument. */ void reset(); @@ -462,7 +460,7 @@ struct ChipTensorMap { * Same as init() with default sizes (CHIP_TENSORMAP_NUM_BUCKETS, * CHIP_TENSORMAP_POOL_SIZE). */ - bool init_default(int32_t new_task_window_size); + bool init_default(int32_t new_max_tasks); /** * Lookup producer for a tensor region @@ -567,7 +565,7 @@ struct ChipTensorMap { #endif uint32_t bucket_index = hash(addr); auto local_id = producer_task_id.local(); - int32_t task_slot = local_id & (task_window_size - 1); + const int32_t task_slot = local_id; entry->producer_task_id = producer_task_id; @@ -605,7 +603,7 @@ struct ChipTensorMap { if (entry.prev_in_task == nullptr) { // Entry is the head of its task chain, update task_entry_heads int32_t local_id = static_cast(entry.producer_task_id.local()); - int32_t task_slot = local_id & (task_window_size - 1); + const int32_t task_slot = local_id; task_entry_heads[task_slot] = entry.next_in_task; } else { entry.prev_in_task->next_in_task = entry.next_in_task; diff --git a/src/a2a3/runtime/host_build_graph/runtime/types.h b/src/a2a3/runtime/host_build_graph/runtime/types.h index 89487dca0f..f110967f21 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/types.h +++ b/src/a2a3/runtime/host_build_graph/runtime/types.h @@ -386,8 +386,10 @@ struct Arg : TaskArgsTpl { * * count == 0 is a valid "set empty" — it clears any previously stored deps * and returns. This lets callers that build the dep set conditionally pass - * the result through unguarded, including in the no-dep branch: - * TaskId deps[3]; + * the result through unguarded, including in the no-dep branch. Fill with + * TaskId::invalid() so an entry the branches never write is rejected by the + * orchestrator rather than read as a task id: + * TaskId deps[3] = {TaskId::invalid(), TaskId::invalid(), TaskId::invalid()}; * uint32_t n = 0; * if (have_prev) deps[n++] = prev; * if (is_last) deps[n++] = alloc; diff --git a/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp b/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp index e9d98e6e86..151a392b3c 100644 --- a/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp +++ b/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp @@ -231,7 +231,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) { // and every cross-task reference it wrote is an offset from its own block, so // the SM/arena this thread sees need no address fixup. This thread attaches // the prebuilt arena, points the SM - // handle's ring-header pointers at the device SM WITHOUT resetting the + // handle's task-header pointers at the device SM WITHOUT resetting the // host-populated data, hands the host-computed task count to the scheduler, // and releases the other threads. It then falls through and schedules its own // cores like every other thread — host_build_graph has no device-side @@ -264,9 +264,10 @@ int32_t AicpuExecutor::run(Runtime *runtime) { void *sm_ptr = runtime->get_gm_sm_ptr(); // The image the host shipped is pitched to the submitted task count, - // not to the ring capacity, and the device region holds exactly that - // image — so its size comes from the same pitch. attach_populated - // rejects a pitch outside (0, capacity] and a region too small for it. + // not to the count the table was dimensioned for, and the device region + // holds exactly that image — so its size comes from the same pitch. + // attach_populated rejects a pitch outside (0, task_capacity] and a + // region too small for it. const uint64_t live_slots = sm_layout::live_slot_pitch(static_cast(runtime->host_total_tasks)); const uint64_t sm_size = runtime->sm_image_bytes; // sm_handle and the scheduler state are the device-only zone: their @@ -276,7 +277,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) { // attach_populated. memset(rt->sm_handle, 0, sizeof(*rt->sm_handle)); if (!rt->sm_handle->attach_populated( - sm_ptr, sm_size, rt->prebuilt_layout.task_window_size, live_slots, runtime->sm_image_bytes + sm_ptr, sm_size, rt->prebuilt_layout.task_capacity, live_slots, runtime->sm_image_bytes )) { LOG_ERROR("Thread %d: host-orch: sm_handle->attach_populated failed", thread_idx); rt = nullptr; diff --git a/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md b/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md index 7ca208cabd..e3d16ecfe7 100644 --- a/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md +++ b/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md @@ -123,7 +123,7 @@ scheduler reads, the count of tasks completed inline during orchestration, is a scalar `rt_orchestration_done` publishes into the runtime header. **Why the scheduler state is device-written.** `SchedulerState` holds no -per-run content: `sm_header` and the ring pointer derive from a pooled SM base, +per-run content: `sm_header` and the task-header pointer derive from a pooled SM base, queue capacities are compile-time constants, hbg never advances `last_task_alive`, and it has no host-side entry point at all. So the host would only be writing an initialization pattern — 203,392 bytes @@ -172,32 +172,38 @@ pins the invariant that makes that safe. ### 3.2 Bounded H2D Upload -The shared-memory mirror is sized to ring capacity (task window) but a run only +The shared-memory mirror is dimensioned for the run's configured task count +(`runtime_env.ring_task_window`, default `CHIP_DEFAULT_GRAPH_TASKS`) but a run only writes `[0, total_tasks)`, and the device boots scheduler-only and reads no SM slot past `total_tasks`. So the SM H2D shipped each run is bounded, not capacity-sized — the contract that keeps `bind` proportional to the workload. The header is zeroed on the host; `descriptors`, `payloads`, `slot_states` and -`completion_flags` are each written per task at submit and H2D-uploaded bounded to -`[0, total_tasks)`. Per-slot reset is init-on-write in `orch::prepare_task` as each -slot is claimed — there is no window-wide reset. The four segments travel as four -copies rather than one because ring-sized tails separate their live prefixes. +`completion_flags` are each written per task at submit. Per-slot reset is +init-on-write in `orch::prepare_task` as each slot is claimed — there is no +table-wide reset. In the mirror those four live prefixes are a full reservation +apart, so `compact_live_image` restacks them (plus the three argument pools) into +an image pitched to `total_tasks`, where they are contiguous and travel as **one** +`copy_to_device`. The device attaches with the same pitch. ## 4. Whole-Graph Capacity -The runtime uses one task ring, one graph heap, and one TensorMap pool. They are +The runtime uses one task table, one graph heap, and one TensorMap pool. They are capacity-bounded storage, not streaming flow-control buffers: -- the task ring and the graph heap are forward-only bump allocators; +- the task table and the graph heap are forward-only bump allocators; - task slots and heap bytes are never recycled mid-run; and - TensorMap entries are held for the whole run. There is no reclaim channel from the scheduler back to the allocator, so the -allocators carry no reclaim pointer and no back-pressure wait. +allocators carry no reclaim pointer and no back-pressure wait. A task id is +therefore also its slot index: ids run `0..capacity-1`, never wrap, and every +segment is indexed by the id directly — there is no slot mask, so the capacity need +not be a power of two. `completed_watermark` records the contiguous prefix of completed device tasks. -It supports completion/consumer metadata only; it does not reclaim the task ring -or heap. +It supports completion/consumer metadata only; it reclaims neither task slots +nor heap. There is no post-run sweep that makes graph space reusable. Runtime destruction releases the complete arena, and the next run starts from a newly initialized @@ -205,19 +211,22 @@ image. ### 4.1 Allocation Failure -The graph must fit the configured task window, the fanin capacity, and the -TensorMap pool. Because nothing is reclaimed, a request that does not fit can -never become satisfiable — the allocator names the exhausted resource and fails -on the spot. There is no wait and no timeout. +The graph must fit the configured task count, the fanin capacity, and the TensorMap +pool. The task count comes from `runtime_env.ring_task_window` (default +`CHIP_DEFAULT_GRAPH_TASKS`); the host mirror is allocated at that size and +committed by first touch, so a run pays only for the slots it writes. Because +nothing is reclaimed, a request that does not fit can never become satisfiable — +the allocator names the exhausted resource and fails on the spot. There is no wait +and no timeout. Representative allocator output is: ```text -FATAL: Task Window Exhausted! +FATAL: Graph Too Large! The whole graph must fit at once; nothing is reclaimed mid-run. - Task window: used=.../... - Graph heap: used=.../..., available=... - Requested: ... bytes + 1 task slot + Tasks: used=.../... + Graph heap: used=.../..., available=... + Requested: ... bytes + 1 task slot ``` This is host-orchestration logging. The allocator records the corresponding diff --git a/src/a5/runtime/host_build_graph/docs/SCALAR_DATA_ACCESS.md b/src/a5/runtime/host_build_graph/docs/SCALAR_DATA_ACCESS.md index 0a436f7cca..39daa3d8d4 100644 --- a/src/a5/runtime/host_build_graph/docs/SCALAR_DATA_ACCESS.md +++ b/src/a5/runtime/host_build_graph/docs/SCALAR_DATA_ACCESS.md @@ -70,11 +70,11 @@ it derives for itself, from a heap block whose prior contents it never reads. Before a wait slot is used, the runtime verifies: -- the task ID is valid and belongs to the single HBG ring; -- the selected ring slot has a bound task descriptor; and +- the task ID is valid and carries ring 0, the only ring HBG places tasks on; +- the task table slot that ID indexes has a bound task descriptor; and - the descriptor's full task ID matches the tensor's owner/producer ID. -The full-ID check prevents a masked ring-slot lookup from aliasing an unused or +The full-ID check prevents a slot lookup from aliasing an unused or different task. A failure latches `SIMPLER_ERROR_INVALID_ARGS` and the run returns status `-5`; reads return zero and writes stop only after that fatal status is recorded. diff --git a/src/a5/runtime/host_build_graph/docs/SUBMIT_BY_CLUSTER.md b/src/a5/runtime/host_build_graph/docs/SUBMIT_BY_CLUSTER.md index 0839f42e9f..28835d7a59 100644 --- a/src/a5/runtime/host_build_graph/docs/SUBMIT_BY_CLUSTER.md +++ b/src/a5/runtime/host_build_graph/docs/SUBMIT_BY_CLUSTER.md @@ -111,9 +111,9 @@ the run. `host_build_graph` is whole-graph-resident. Task slots, heap bytes, fanin IDs, and TensorMap entries are not reclaimed while the graph is executing. The graph -must fit the configured task window and TensorMap pool before launch; its heap -takes no configuration, since the device region is committed after orchestration -at the size the graph turned out to need. +must fit the configured task count (`runtime_env.ring_task_window`) and the +TensorMap pool before launch; the heap is not a third capacity, since its device +region is committed after orchestration at the size the graph turned out to need. ## Validation diff --git a/src/a5/runtime/host_build_graph/host/runtime_maker.cpp b/src/a5/runtime/host_build_graph/host/runtime_maker.cpp index 1c3ef68c0c..e4e7f5456c 100644 --- a/src/a5/runtime/host_build_graph/host/runtime_maker.cpp +++ b/src/a5/runtime/host_build_graph/host/runtime_maker.cpp @@ -120,11 +120,9 @@ extern "C" int concurrent_native_prepare_supported_impl(void) { // RuntimeEnv (call_config.h) is the cross-runtime ABI for per-ring config and // carries RUNTIME_ENV_RING_COUNT slots, shared with tensormap_and_ringbuffer. -// host_build_graph has one ring and reads slot 0, so it only needs the ABI to -// carry at least one. -static_assert(RUNTIME_ENV_RING_COUNT >= 1, "RuntimeEnv must carry the ring slot host_build_graph reads"); - -static bool is_power_of_2_u64(uint64_t value) { return value != 0 && (value & (value - 1)) == 0; } +// host_build_graph keeps one task table and reads slot 0, so it only needs the ABI +// to carry at least one. +static_assert(RUNTIME_ENV_RING_COUNT >= 1, "RuntimeEnv must carry the slot host_build_graph reads"); // Host monotonic clock, shared with the record pool so spans and records can be // read against each other. @@ -197,7 +195,7 @@ static void warn_on_retired_ring_env() { } } -// ring_task_window points into the #pragma pack(1) RuntimeEnv wire struct +// A RuntimeEnv knob array points into the #pragma pack(1) wire struct // (call_config.h), so its uint64_t entries are only byte-aligned — runtime_env // sits at offset 28 in CallConfig (after 7 int32_t), i.e. 4-byte but not 8-byte // aligned. Reading them as `base[idx]` is an unaligned 8-byte load: UB, and fatal @@ -213,38 +211,42 @@ static uint64_t read_ring_override(const uint64_t *base, int idx) { } // ring_task_window points at the first slot of a per-ring array in the RuntimeEnv -// wire struct (0 = unset); hbg has one ring and reads slot 0. A per-task entry -// wins over the compile-time default, and there is nothing between them. +// wire struct (0 = unset); hbg keeps one task table and reads slot 0. A per-task +// entry wins over the compile-time default, and there is nothing between them. // // The heap takes no configuration: its device region is committed after // orchestration, sized to what the graph turned out to need, so there is nothing // to resolve up front. RuntimeEnv::ring_heap stays the reclaiming runtime's knob; // this function does not resolve it, and the caller reads it only to warn that it // reaches nothing here. -static bool resolve_task_window_size(const uint64_t *ring_task_window, uint64_t *eff_task_window_size) { - *eff_task_window_size = CHIP_TASK_WINDOW_SIZE; +static bool resolve_graph_task_capacity(const uint64_t *ring_task_window, uint64_t *task_capacity) { + *task_capacity = CHIP_DEFAULT_GRAPH_TASKS; warn_on_retired_ring_env(); - const uint64_t task_window_override = read_ring_override(ring_task_window, 0); - if (task_window_override != 0) { - *eff_task_window_size = task_window_override; + const uint64_t override_value = read_ring_override(ring_task_window, 0); + if (override_value != 0) { + *task_capacity = override_value; } - if (*eff_task_window_size < 4 || *eff_task_window_size > static_cast(INT32_MAX) || - !is_power_of_2_u64(*eff_task_window_size)) { - LOG_ERROR("ring_task_window=%" PRIu64 " must be a power of 2 in [4, INT32_MAX]", *eff_task_window_size); + // Any positive count is usable: a task id indexes its slot directly, so + // nothing masks with this value. The power-of-two, >= 4 requirement belongs to + // tensormap_and_ringbuffer, which does mask, and is enforced in that runtime's + // own resolve; neither the RuntimeEnv setter nor Worker.run constrains the + // value, so this bound is the only one a ring_task_window passes through. + if (*task_capacity < 1 || *task_capacity > static_cast(INT32_MAX)) { + LOG_ERROR("ring_task_window=%" PRIu64 " must be in [1, INT32_MAX]", *task_capacity); return false; } // A slot state reaches its payload and descriptor through a 32-bit // self-relative delta, so every pair of addresses in the shared-memory // image must be within INT32_MAX of each other. - const uint64_t sm_bytes = sm_layout::ring_segment_offsets(*eff_task_window_size).end; + const uint64_t sm_bytes = sm_layout::segment_offsets(*task_capacity).end; if (sm_bytes > static_cast(INT32_MAX)) { LOG_ERROR( "ring_task_window=%" PRIu64 " needs a %" PRIu64 "-byte shared memory image, past the %d-byte limit " "a slot state's self-relative payload/descriptor delta can span", - *eff_task_window_size, sm_bytes, INT32_MAX + *task_capacity, sm_bytes, INT32_MAX ); return false; } @@ -495,7 +497,7 @@ struct GraphHostStateBinding { int32_t run_host_orchestration( Runtime *runtime, const HostApi *api, HostTensorAccessor &tensor_access, RuntimeContext *rt, - DeviceArena &host_arena, const RuntimeArenaLayout &layout, uint64_t sm_size, uint64_t eff_task_window_size, + DeviceArena &host_arena, const RuntimeArenaLayout &layout, uint64_t sm_size, uint64_t task_capacity, void *host_orch_func_ptr, const ChipTaskArgs &orch_l2 ) { // The dep_gen graph belongs to the orchestration that is about to run. @@ -505,7 +507,7 @@ int32_t run_host_orchestration( // each written per task at submit and read only for [0, total_tasks). Zero // only the fixed-size header here; the per-slot segments are initialized in // orch::prepare_task and shipped bounded to total_tasks below. - const sm_layout::ChipRingSegmentOffsets sm_segs = sm_layout::ring_segment_offsets(eff_task_window_size); + const sm_layout::SegmentOffsets sm_segs = sm_layout::segment_offsets(task_capacity); // Over-allocated and rounded up: every segment offset is a multiple of // CHIP_ALIGN_SIZE and ChipTaskSlotState is alignas(64), which a plain // new uint8_t[] does not guarantee. @@ -529,8 +531,7 @@ int32_t run_host_orchestration( // actually needs, and compact_live_image moves every address the orchestrator // wrote onto the real base before the image travels. if (!orchestrator.init( - host_sm, reinterpret_cast(HEAP_VIRTUAL_BASE), HEAP_VIRTUAL_CAPACITY, eff_task_window_size, - rt->scheduler + host_sm, reinterpret_cast(HEAP_VIRTUAL_BASE), HEAP_VIRTUAL_CAPACITY, task_capacity, rt->scheduler )) { LOG_ERROR("host-orch: orchestrator init against host SM failed"); return PTO_RUNTIME_ERR_INTERNAL; @@ -538,7 +539,7 @@ int32_t run_host_orchestration( // Initialize the host SM header (ring flow control) so submit_task can run. SharedMemoryHandle host_sm_handle; - if (!host_sm_handle.init(host_sm, sm_size, eff_task_window_size)) { + if (!host_sm_handle.init(host_sm, sm_size, task_capacity)) { LOG_ERROR("host-orch: host SM init failed"); return PTO_RUNTIME_ERR_INTERNAL; } @@ -648,7 +649,7 @@ int32_t run_host_orchestration( return status; } - const int32_t total_tasks = sm_layout::ring_current_task_index_addr(host_sm)->load(std::memory_order_acquire); + const int32_t total_tasks = orchestrator.task_allocator.active_count(); { char attrs[160]; snprintf( @@ -684,23 +685,29 @@ int32_t run_host_orchestration( } // total_tasks sizes the bounded per-segment H2D copies below; a value outside - // [0, task_window] would make those copies read/write out of bounds. - if (total_tasks < 0 || static_cast(total_tasks) > eff_task_window_size) { - LOG_ERROR("host-orch: total_tasks %d out of range [0, %" PRIu64 "]", total_tasks, eff_task_window_size); + // [0, task_capacity] would make those copies read/write out of bounds. + if (total_tasks < 0 || static_cast(total_tasks) > task_capacity) { + LOG_ERROR("host-orch: total_tasks %d out of range [0, %" PRIu64 "]", total_tasks, task_capacity); return PTO_RUNTIME_ERR_INTERNAL; } host_phase_trace_note_submitted(static_cast(total_tasks)); - // The device reads no ring slot past total_tasks, so only that prefix of each + // The count travels inside the header the restack copies wholesale, which is + // what lets the device bound its completed_watermark walk without a second + // carrier. Written after the range check above, so the value the device reads + // is one the segments are actually pitched to. + reinterpret_cast(host_sm)->tasks.total_tasks = total_tasks; + + // The device reads no task slot past total_tasks, so only that prefix of each // segment has to travel. In the mirror the orchestrator wrote, the four - // prefixes are a ring capacity apart, which would make the upload four copies - // of a few hundred kilobytes each — and at these sizes a copy_to_device is - // priced by the call, not by the bytes. + // prefixes are a whole task_capacity apart, which would make the upload four + // copies of a few hundred kilobytes each — and at these sizes a copy_to_device + // is priced by the call, not by the bytes. // // So the prefixes are restacked into an image pitched to total_tasks, where // they are contiguous, and that image goes up as one copy. The device attaches - // with the same pitch. The ring capacity and mask are untouched: `local_id & - // mask` is `local_id`, which is below the pitch for every ring task. + // with the same pitch, which is sound because a task id is its own slot index: + // every id is below total_tasks and indexes the image directly. const uint64_t nt = static_cast(total_tasks); // What this bind actually put in the pools. The orchestrator's cursors are the // exact populated extent of each one — no scan of the mirror is needed, and the @@ -712,7 +719,7 @@ int32_t run_host_orchestration( static_cast(orch_state.tensor_pool_cursor), static_cast(orch_state.scalar_pool_cursor), }; - const uint64_t image_bytes = sm_layout::ring_segment_offsets(sm_layout::image_extents(bind_usage)).end; + const uint64_t image_bytes = sm_layout::segment_offsets(sm_layout::image_extents(bind_usage)).end; runtime->sm_image_bytes = image_bytes; // Only now are both sizes known, so this is where the two device regions are @@ -814,7 +821,7 @@ int32_t run_host_orchestration( rt->orchestrator = nullptr; std::memcpy(upload_base, static_cast(host_arena.base()) + layout.off_copied_begin, copied_bytes); const uint64_t compacted = sm_layout::compact_live_image( - static_cast(host_sm), eff_task_window_size, bind_usage, heap_rebase, upload_base + copied_bytes + static_cast(host_sm), task_capacity, bind_usage, heap_rebase, upload_base + copied_bytes ); always_assert(compacted == image_bytes); @@ -1005,8 +1012,8 @@ extern "C" int bind_callable_to_runtime_impl( host_phase_trace_end(); }); - uint64_t eff_task_window_size = 0; - if (!resolve_task_window_size(ring_task_window, &eff_task_window_size)) { + uint64_t task_capacity = 0; + if (!resolve_graph_task_capacity(ring_task_window, &task_capacity)) { return PTO_RUNTIME_ERR_INTERNAL; } // The heap takes no configuration, so a set ring_heap reaches nothing in this @@ -1018,7 +1025,6 @@ extern "C" int bind_callable_to_runtime_impl( "size the graph turned out to need" ); } - LOG_INFO("Ring task window: %" PRIu64, eff_task_window_size); // Build device args: copy from input, replace host tensor pointers with device pointers ChipStorageTaskArgs device_args; @@ -1118,11 +1124,11 @@ extern "C" int bind_callable_to_runtime_impl( // runs — do NOT record in tensor_pairs_; the free is deferred to // DeviceRunner::finalize(). The runtime-arena size is determined by replaying // the reserve sequence on a host-side arena. - uint64_t sm_size = SharedMemoryHandle::calculate_size(eff_task_window_size); + uint64_t sm_size = SharedMemoryHandle::calculate_size(task_capacity); const int64_t t_arena_build_ns = bind_now_ns(); DeviceArena host_arena; - RuntimeArenaLayout layout = runtime_reserve_layout(host_arena, eff_task_window_size); + RuntimeArenaLayout layout = runtime_reserve_layout(host_arena, task_capacity); if (host_arena.commit(DeviceArena::kDefaultBaseAlign) == nullptr) { LOG_ERROR("Failed to commit host arena for prebuilt runtime image"); return PTO_RUNTIME_ERR_INTERNAL; @@ -1185,8 +1191,7 @@ extern "C" int bind_callable_to_runtime_impl( ChipTaskArgs orch_l2; orch_l2.create_from_entry_storage(runtime->get_orch_args()); int32_t total_tasks = run_host_orchestration( - runtime, api, tensor_access, rt, host_arena, layout, sm_size, eff_task_window_size, host_orch_func_ptr, - orch_l2 + runtime, api, tensor_access, rt, host_arena, layout, sm_size, task_capacity, host_orch_func_ptr, orch_l2 ); // The orchestrator is the only host-view reader; from here the device // owns these buffers, so drop the window on both exits. diff --git a/src/a5/runtime/host_build_graph/runtime/orchestrator.h b/src/a5/runtime/host_build_graph/runtime/orchestrator.h index e3180208d7..a53a040d9a 100644 --- a/src/a5/runtime/host_build_graph/runtime/orchestrator.h +++ b/src/a5/runtime/host_build_graph/runtime/orchestrator.h @@ -30,7 +30,7 @@ #include #include "common/chip_swimlane_profiling.h" -#include "ring_buffer.h" +#include "task_allocator.h" #include "graph_cache.h" #include "runtime_types.h" #include "submit_types.h" @@ -106,8 +106,8 @@ struct OrchestratorState { // binds its region at the cursor and advances it by what the task uses, so the // pools stay packed and the bind path reads the cursors as the image's pool // extents. Nothing is ever returned, and the pools are dimensioned for the worst - // case (task_window tasks each at their full cap), so a bump cannot overflow. - // The bases live here, not in the ring header: nothing on the device resolves one. + // case (max_tasks tasks each at their full cap), so a bump cannot overflow. + // The bases live here, not in the task header: nothing on the device resolves one. // // The fanin cursor does not advance at bind time. FaninBuilder appends and // dedups producers afterwards, so the region's length is known only when the count @@ -134,11 +134,12 @@ struct OrchestratorState { // and bind this orchestrator to one shared-memory mirror, GM heap and // scheduler. sm_base is the base of the mirror this orchestrator writes; it // is dereferenced, so a host-orch pass passes its host mirror rather than a - // device address. task_window_size must be a power of two. + // device address. `max_tasks` is the slot count that mirror is dimensioned + // for — the bind's resolved ring_task_window — and the cap alloc() enforces. // // Returns false when an allocation fails; the caller then has no hazard map // and must not orchestrate. - bool init(void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t task_window_size, SchedulerState *scheduler); + bool init(void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t max_tasks, SchedulerState *scheduler); void set_scheduler(SchedulerState *scheduler); void report_fatal(int32_t error_code, const char *func, const char *fmt, ...); diff --git a/src/a5/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp b/src/a5/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp index 5a1bab7d5f..8d49941ecd 100644 --- a/src/a5/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp +++ b/src/a5/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp @@ -1126,8 +1126,7 @@ static uint32_t next_fanin_seen_epoch(OrchestratorState *orch) { uint32_t next = orch->fanin_seen_current_epoch + 1; if (next == 0) { memset( - orch->fanin_seen_epoch.get(), 0, - static_cast(orch->sm_header->ring.task_window_size) * sizeof(uint32_t) + orch->fanin_seen_epoch.get(), 0, static_cast(orch->task_allocator.capacity()) * sizeof(uint32_t) ); next = 1; } @@ -1158,18 +1157,18 @@ struct FaninBuilder { // re-resolve the delta per producer. Requires the regions bound before construction. int32_t *slots{nullptr}; - bool mark_seen(uint8_t prod_ring, int32_t prod_slot) { - // Dedup only: a producer this orchestrator did not place has no slot in the + bool mark_seen(TaskId producer_task_id) { + // Dedup only: a producer this orchestrator did not place has no entry in the // epoch table, so it is reported as not-yet-seen and the caller appends it - // rather than rejecting it. hbg places every task on its one ring, so the - // assert is the invariant; enforcing it on the caller's side would be a new - // fatal path. - debug_assert(prod_ring == 0 && "hbg places every task on its one ring"); - if (prod_ring != 0 || prod_slot < 0) { + // rather than rejecting it. hbg places every task on ring 0, so the assert is + // the invariant; enforcing it on the caller's side would be a new fatal path. + debug_assert(producer_task_id.ring() == 0 && "hbg places every task on ring 0"); + const int32_t prod_local = static_cast(producer_task_id.local()); + if (producer_task_id.ring() != 0 || prod_local < 0) { return false; } uint32_t *seen = orch->fanin_seen_epoch.get(); - uint32_t slot = static_cast(prod_slot); + uint32_t slot = static_cast(prod_local); if (seen[slot] == seen_epoch) { return true; } @@ -1179,8 +1178,7 @@ struct FaninBuilder { }; static bool append_fanin_or_fail( - OrchestratorState *orch, uint8_t prod_ring, int32_t prod_slot, ChipTaskSlotState *prod_state, - TaskId producer_task_id, FaninBuilder *fanin_builder + OrchestratorState *orch, ChipTaskSlotState *prod_state, TaskId producer_task_id, FaninBuilder *fanin_builder ) { // Skip a stale/reused producer slot: the cached owner id no longer resolves // to this producer (defensive — whole-graph-resident hbg does not reuse slots @@ -1189,8 +1187,8 @@ static bool append_fanin_or_fail( if (prod_state->task == nullptr || prod_state->task->task_id.local() != producer_task_id.local()) { return true; } - // Dedup by (ring, slot). Single-ring hbg: prod_ring is always 0. - if (fanin_builder->mark_seen(prod_ring, prod_slot)) { + // Dedup by producer local id, which is also its task-table slot. + if (fanin_builder->mark_seen(producer_task_id)) { return true; } if (fanin_builder->count >= CHIP_MAX_FANIN) { @@ -1219,7 +1217,7 @@ static bool append_fanin_or_fail( struct PreparedTask { TaskId task_id = TaskId::invalid(); - TaskAllocResult alloc_result = {-1, 0, nullptr, nullptr}; + TaskAllocResult alloc_result = {-1, nullptr, nullptr}; TaskDescriptor *task = nullptr; TaskPayload *payload = nullptr; ChipTaskSlotState *slot_state = nullptr; @@ -1243,7 +1241,6 @@ static bool prepare_task( TaskAttrs task_attrs, PreparedTask *out ) { always_assert(orch->scope_stack_top >= 0 && "Cannot submit task outside a scope"); - uint8_t ring_id = 0; auto &allocator = orch->task_allocator; int16_t block_num = args.launch_spec.block_num(); @@ -1264,10 +1261,10 @@ static bool prepare_task( return false; } - out->task_id = TaskId::make(ring_id, static_cast(out->alloc_result.task_id)); - out->slot_state = &orch->sm_header->ring.get_slot_state_by_slot(out->alloc_result.slot); - out->task = &orch->sm_header->ring.task_descriptors[out->alloc_result.slot]; - out->payload = &orch->sm_header->ring.task_payloads[out->alloc_result.slot]; + out->task_id = TaskId::make(0, static_cast(out->alloc_result.task_id)); + out->slot_state = &orch->sm_header->tasks.get_slot_state_by_task_id(out->alloc_result.task_id); + out->task = &orch->sm_header->tasks.task_descriptors[out->alloc_result.task_id]; + out->payload = &orch->sm_header->tasks.task_payloads[out->alloc_result.task_id]; // Bind the three argument regions before prefetch() and init(), both of which // dereference them. The scalar cursor advances in whole cache lines because init() @@ -1275,11 +1272,11 @@ static bool prepare_task( // write into the next task's region. A tensor region is aligned for any count, // simpler::hbg::Tensor being two cache lines. The fanin cursor advances at publish, not // here — see the comment where it does. - const uint64_t window = orch->sm_header->ring.task_window_size; + const uint64_t max_tasks = static_cast(orch->task_allocator.capacity()); const int32_t scalar_span = CHIP_ALIGN_UP(args.scalar_count(), ARG_POOL_ALIGN / (int32_t)sizeof(uint64_t)); - debug_assert(static_cast(orch->tensor_pool_cursor) + args.tensor_count() <= window * MAX_TENSOR_ARGS); - debug_assert(static_cast(orch->scalar_pool_cursor) + scalar_span <= window * MAX_SCALAR_ARGS); - debug_assert(static_cast(orch->fanin_pool_cursor) + CHIP_MAX_FANIN <= window * CHIP_MAX_FANIN); + debug_assert(static_cast(orch->tensor_pool_cursor) + args.tensor_count() <= max_tasks * MAX_TENSOR_ARGS); + debug_assert(static_cast(orch->scalar_pool_cursor) + scalar_span <= max_tasks * MAX_SCALAR_ARGS); + debug_assert(static_cast(orch->fanin_pool_cursor) + CHIP_MAX_FANIN <= max_tasks * CHIP_MAX_FANIN); out->payload->bind_regions( orch->tensor_pool + orch->tensor_pool_cursor, orch->scalar_pool + orch->scalar_pool_cursor, orch->fanin_pool + orch->fanin_pool_cursor @@ -1293,13 +1290,13 @@ static bool prepare_task( // past total_tasks, so this claim-time write is the only per-slot SM reset and // the unclaimed tail is neither initialized nor read. out->slot_state->reset_for_reuse(); - orch->sm_header->ring.completion_flags[out->alloc_result.slot].store(0, std::memory_order_relaxed); + orch->sm_header->tasks.completion_flags[out->alloc_result.task_id].store(0, std::memory_order_relaxed); out->payload->prefetch(args.tensor_count(), args.scalar_count()); // Re-bind payload/task pointers each submit. Value is per-slot constant // (same as &task_payloads[slot] / &task_descriptors[slot]), but writing - // here lets RingSchedState::init() skip the O(window_size) bind loop. + // here lets TaskHeaderView::init() skip the O(max_tasks) bind loop. // Both writes hit the same 64B slot_state cache line we're about to // dirty below, so the extra cost is two stores on an already-hot line. // Must precede the Orch-side wiring publish at the end of @@ -1314,8 +1311,6 @@ static bool prepare_task( // Fields already zeroed by the reset_for_reuse() above: // wake_list_head=nullptr, next_in_wake_list=nullptr, // any_subtask_deferred=false, completed_subtasks=0, next_block_idx=0 - // Fields immutable after RingSchedState::init(): - // ring_id // task_state is set to PENDING here as the orchestrator populates the slot // (host_build_graph does not recycle slots at runtime, so there is no // post-CONSUMED reset path). @@ -1506,12 +1501,10 @@ static TaskOutputTensors submit_task_common( if (capture_dep_graph) { dep_gen_host_graph_add_explicit_edge(dep_task_id.raw); } - uint8_t dep_ring_id = dep_task_id.ring(); - SharedMemoryRingHeader &dep_ring = orch->sm_header->ring; + SharedMemoryTaskHeader &dep_tasks = orch->sm_header->tasks; int32_t dep_local_task_id = static_cast(dep_task_id.local()); - int32_t dep_slot = dep_ring.get_slot_by_task_id(dep_local_task_id); - ChipTaskSlotState *producer_slot_state = &dep_ring.get_slot_state_by_slot(dep_slot); - if (!append_fanin_or_fail(orch, dep_ring_id, dep_slot, producer_slot_state, dep_task_id, &fanin_builder)) { + ChipTaskSlotState *producer_slot_state = &dep_tasks.get_slot_state_by_task_id(dep_local_task_id); + if (!append_fanin_or_fail(orch, producer_slot_state, dep_task_id, &fanin_builder)) { return result; } } @@ -1523,11 +1516,10 @@ static TaskOutputTensors submit_task_common( }; auto runtime_emit = [&](TaskId producer_task_id) -> bool { - uint8_t prod_ring = producer_task_id.ring(); - SharedMemoryRingHeader &producer_ring = orch->sm_header->ring; - int32_t prod_slot = producer_ring.get_slot_by_task_id(static_cast(producer_task_id.local())); - ChipTaskSlotState *prod_state = &producer_ring.get_slot_state_by_slot(prod_slot); - return append_fanin_or_fail(orch, prod_ring, prod_slot, prod_state, producer_task_id, &fanin_builder); + SharedMemoryTaskHeader &producer_tasks = orch->sm_header->tasks; + ChipTaskSlotState *prod_state = + &producer_tasks.get_slot_state_by_task_id(static_cast(producer_task_id.local())); + return append_fanin_or_fail(orch, prod_state, producer_task_id, &fanin_builder); }; // The capture branch instantiates compute_task_fanin with a live Annotate; @@ -1760,26 +1752,26 @@ bool graph_submit_outer( ) { always_assert(orch->scope_stack_top >= 0 && "Cannot submit Graph outside a scope"); auto &allocator = orch->task_allocator; - if (allocator.active_count() >= allocator.window_size() || + if (allocator.active_count() >= allocator.capacity() || (!defer_heap && static_cast(owned_heap) > allocator.heap_available())) { - LOG_WARN("%s", "[GraphExecution] task-window/heap preflight failed; using ordinary path"); + LOG_WARN("%s", "[GraphExecution] task-capacity/heap preflight failed; using ordinary path"); return false; } // The argument pools hold MAX_TENSOR_ARGS ChipTensors and MAX_SCALAR_ARGS scalars - // per window slot, a budget no CoreTaskArgs task can exceed. A Graph boundary is + // per task slot, a budget no CoreTaskArgs task can exceed. A Graph boundary is // GraphTaskArgs-wide, so a wide one draws more than the single slot it occupies is // worth — GraphBoundaryPool.WidestBoundaryExceedsOneSlotBudget pins how much. The // cursors bump through a fixed mirror whose last segment is the scalar pool, so an // overdraw writes past that mirror rather than merely exhausting a quota. Test it // ahead of the slot claim and decline the Graph path, which leaves the caller to // replay the block as ordinary tasks. - const uint64_t task_window = orch->sm_header->ring.task_window_size; + const uint64_t max_tasks = static_cast(orch->task_allocator.capacity()); const int32_t tensor_slots = static_cast(graph_boundary_tensor_pool_slots(static_cast(args.tensor_count()))); const int32_t scalar_span = CHIP_ALIGN_UP(args.scalar_count(), ARG_POOL_ALIGN / (int32_t)sizeof(uint64_t)); - if (static_cast(orch->tensor_pool_cursor) + tensor_slots > task_window * MAX_TENSOR_ARGS || - static_cast(orch->scalar_pool_cursor) + scalar_span > task_window * MAX_SCALAR_ARGS) { + if (static_cast(orch->tensor_pool_cursor) + tensor_slots > max_tasks * MAX_TENSOR_ARGS || + static_cast(orch->scalar_pool_cursor) + scalar_span > max_tasks * MAX_SCALAR_ARGS) { LOG_WARN("%s", "[GraphExecution] boundary exceeds the argument pools; using ordinary path"); return false; } @@ -1800,10 +1792,10 @@ bool graph_submit_outer( return false; } const TaskId task_id = TaskId::make(0, static_cast(allocation.task_id)); - SharedMemoryRingHeader &ring = orch->sm_header->ring; - TaskDescriptor &task = ring.task_descriptors[allocation.slot]; - TaskPayload &payload = ring.task_payloads[allocation.slot]; - ChipTaskSlotState &slot = ring.get_slot_state_by_slot(allocation.slot); + SharedMemoryTaskHeader &tasks = orch->sm_header->tasks; + TaskDescriptor &task = tasks.task_descriptors[allocation.task_id]; + TaskPayload &payload = tasks.task_payloads[allocation.task_id]; + ChipTaskSlotState &slot = tasks.get_slot_state_by_task_id(allocation.task_id); // Init-on-write, as in prepare_task: this slot's dynamic scheduling fields and // completion flag are established here, at the claim, because nothing else @@ -1811,7 +1803,7 @@ bool graph_submit_outer( // list against every consumer, and a stale completion flag would report the // Graph done before it ran. slot.reset_for_reuse(); - ring.completion_flags[allocation.slot].store(0, std::memory_order_relaxed); + tasks.completion_flags[allocation.task_id].store(0, std::memory_order_relaxed); slot.bind_buffers(&payload, &task); // Graph boundaries use the same compact argument pools as ordinary tasks. The @@ -1851,16 +1843,14 @@ bool graph_submit_outer( FaninBuilder fanin_builder(orch, &payload, static_cast(task_id.local()), next_fanin_seen_epoch(orch)); auto emit = [&](TaskId producer_id) -> bool { - const int32_t producer_local = static_cast(producer_id.local()); - const int32_t producer_slot = ring.get_slot_by_task_id(producer_local); - ChipTaskSlotState *producer = &ring.get_slot_state_by_slot(producer_slot); - return append_fanin_or_fail(orch, producer_id.ring(), producer_slot, producer, producer_id, &fanin_builder); + ChipTaskSlotState *producer = &tasks.get_slot_state_by_task_id(static_cast(producer_id.local())); + return append_fanin_or_fail(orch, producer, producer_id, &fanin_builder); }; - // An outer GRAPH task is a ring task like any other, so the dependency graph + // An outer GRAPH task is an ordinary task, so the dependency graph // has to carry it: without this the whole Graph — and every edge into it — // is absent from deps.json, leaving a run of 40 replays described by only its // handful of non-Graph tasks. It dispatches no kernel of its own and the - // sub-DAG it replays owns no ring slots, so what is captured is its boundary: + // sub-DAG it replays owns no task slots, so what is captured is its boundary: // the args it consumes and the edges those produce. const bool capture_dep_graph = dep_gen_host_graph_enabled(); if (capture_dep_graph) { @@ -1956,10 +1946,10 @@ bool graph_finalize_pending_submissions(OrchestratorState *orch, GraphHostState } // Record one internal Graph node while recording, without consuming a -// ring task-window slot. Builds the node's metadata and materialized outputs +// task-table slot. Builds the node's metadata and materialized outputs // exactly as submit_task_common would, but assigns output buffers from the // bit-63 virtual address range and derives internal fanins from tensor-source -// classification — so no ring slot, tensormap entry, fanin-pool entry, or upload +// classification — so no task slot, tensormap entry, fanin-pool entry, or upload // is produced for the node. The resulting Definition is later attached to the // outer GRAPH shells already submitted by the main thread. The returned // TaskOutputTensors borrow the node's own tensor storage; moving the node into @@ -2753,9 +2743,9 @@ TaskOutputTensors OrchestratorState::alloc_tensors(const CoreTaskArgs &args) { // every consumer register_wakes on a producer that never runs on device and // the run hangs. (The device watermark walk transparently steps past this // pre-set flag when a later on-device task completes.) - SharedMemoryRingHeader &done_ring = orch->sm_header->ring; + SharedMemoryTaskHeader &done_tasks = orch->sm_header->tasks; int32_t done_local = static_cast(prepared.task_id.local()); - done_ring.set_completion_flag(done_local); + done_tasks.set_completion_flag(done_local); } orch->inline_completed_tasks++; diff --git a/src/a5/runtime/host_build_graph/runtime/orchestrator_core/ring_buffer.cpp b/src/a5/runtime/host_build_graph/runtime/orchestrator_core/ring_buffer.cpp deleted file mode 100644 index 26b277c8a3..0000000000 --- a/src/a5/runtime/host_build_graph/runtime/orchestrator_core/ring_buffer.cpp +++ /dev/null @@ -1,17 +0,0 @@ -/* - * Copyright (c) PyPTO Contributors. - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of - * CANN Open Software License Agreement Version 2.0 (the "License"). - * Please refer to the License for details. You may not use this file except in compliance with the License. - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. - * See LICENSE in the root of the software repository for the full text of the License. - * ----------------------------------------------------------------------------------------------------------- - */ - -// The polling completion scheduler has no dep-pool / fanin-spill pool: producer -// dependencies are position-independent local-id integers on the payload and -// readiness is derived from the ring completion_flags. The pool ring buffers -// (FaninPool / DepListPool) that once lived here are gone, so this -// translation unit carries no definitions. TaskAllocator's methods are -// defined inline in ring_buffer.h. diff --git a/src/a5/runtime/host_build_graph/runtime/orchestrator_core/runtime_core.cpp b/src/a5/runtime/host_build_graph/runtime/orchestrator_core/runtime_core.cpp index 235d090ac6..698f3bdaaa 100644 --- a/src/a5/runtime/host_build_graph/runtime/orchestrator_core/runtime_core.cpp +++ b/src/a5/runtime/host_build_graph/runtime/orchestrator_core/runtime_core.cpp @@ -195,15 +195,14 @@ static bool wait_for_tensor_ready( return nullptr; } - auto &ring = orch.sm_header->ring; + auto &tasks = orch.sm_header->tasks; int32_t local_id = static_cast(producer.local()); - int32_t slot_index = ring.get_slot_by_task_id(local_id); - auto &slot = ring.get_slot_state_by_slot(slot_index); + auto &slot = tasks.get_slot_state_by_task_id(local_id); if (slot.task == nullptr || slot.task->task_id != producer) { orch.report_fatal( SIMPLER_ERROR_INVALID_ARGS, caller, "tensor producer task %#llx does not match the descriptor bound to slot %d", - static_cast(producer.raw), slot_index + static_cast(producer.raw), local_id ); failed = true; return nullptr; @@ -212,7 +211,6 @@ static bool wait_for_tensor_ready( }; auto wait_one_producer = [&](const ChipTaskSlotState &slot) { - uint8_t ring_id = 0; int32_t local_id = static_cast(slot.task->task_id.local()); uint64_t t0 = get_sys_cnt_aicpu(); int32_t spin_count = 0; @@ -228,8 +226,8 @@ static bool wait_for_tensor_ready( if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) { orch.report_fatal( SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller, - "Timeout (%llu cycles): producer (ring=%d, local=%d) not completed", - (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id + "Timeout (%llu cycles): producer (local=%d) not completed", + (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, local_id ); failed = true; return; @@ -239,7 +237,6 @@ static bool wait_for_tensor_ready( }; auto wait_one_consumers = [&](const ChipTaskSlotState &slot) { - uint8_t ring_id = 0; int32_t local_id = slot.task->task_id.local(); uint64_t t0 = get_sys_cnt_aicpu(); int32_t spin_count = 0; @@ -247,8 +244,8 @@ static bool wait_for_tensor_ready( // completed_watermark reaches the producer's highest consumer id (set at // submit in append_fanin_or_fail). Replaces the fanout_refcount == // fanout_count wiring check, which polling removes. - SharedMemoryRingHeader &cons_ring = orch.sm_header->ring; - while (cons_ring.completed_watermark.load(std::memory_order_acquire) < slot.last_consumer_local_id) { + SharedMemoryTaskHeader &cons_tasks = orch.sm_header->tasks; + while (cons_tasks.completed_watermark.load(std::memory_order_acquire) < slot.last_consumer_local_id) { SPIN_WAIT_HINT(); if ((++spin_count & 1023) == 0) { // A fatal latched elsewhere (e.g. the scheduler-side wiring @@ -260,8 +257,8 @@ static bool wait_for_tensor_ready( if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) { orch.report_fatal( SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller, - "Timeout (%llu cycles): consumers of producer (ring=%d, local=%d) not done", - (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id + "Timeout (%llu cycles): consumers of producer (local=%d) not done", + (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, local_id ); failed = true; return; diff --git a/src/a5/runtime/host_build_graph/runtime/runtime.h b/src/a5/runtime/host_build_graph/runtime/runtime.h index 50895c4442..ffe6b165d6 100644 --- a/src/a5/runtime/host_build_graph/runtime/runtime.h +++ b/src/a5/runtime/host_build_graph/runtime/runtime.h @@ -168,10 +168,10 @@ class Runtime { // NOTE: Made public for direct access from aicore code uint64_t func_id_to_addr_[RUNTIME_MAX_FUNC_ID]; - // Total tasks submitted by the host orchestrator — handed to the scheduler - // (SchedulerContext::on_orchestration_done) in place of latching the SM ring - // head on device. host_build_graph builds the whole graph on the host, so - // the boot thread reads this instead of counting SM ring heads. + // Total tasks the host orchestrator submitted, handed to the scheduler by + // SchedulerContext::on_orchestration_done. host_build_graph builds the whole + // graph on the host, so this scalar is the count's only carrier: the shared + // memory header holds no task counter for the boot thread to read. int32_t host_total_tasks; // Size of the shipped shared-memory image, argument pools included. Set by the diff --git a/src/a5/runtime/host_build_graph/runtime/runtime_core.h b/src/a5/runtime/host_build_graph/runtime/runtime_core.h index 840edd0aea..749b498083 100644 --- a/src/a5/runtime/host_build_graph/runtime/runtime_core.h +++ b/src/a5/runtime/host_build_graph/runtime/runtime_core.h @@ -15,7 +15,7 @@ * It provides a unified API for task graph construction and execution. * * Key Features: - * - Ring buffer based memory management (zero allocation overhead) + * - Bump-allocated task table and graph heap (zero allocation overhead) * - Lazy invalidation TensorMap for dependency discovery * - Manual scopes that bypass TensorMap discovery for explicit dependencies * - Per-task spinlocks for concurrent fanout updates @@ -41,7 +41,7 @@ #include "graph_cache.h" #include "submit_types.h" #include "shared_memory.h" -#include "ring_buffer.h" +#include "task_allocator.h" #include "tensormap.h" #include "scheduler/scheduler.h" #include "orchestrator.h" @@ -151,9 +151,10 @@ struct RuntimeArenaLayout { size_t off_copied_begin{0}; size_t off_copied_end{0}; - // Cached parameter (re-used by init_data + wire stages). hbg is single-ring, - // so this is the one ring's. - uint64_t task_window_size{0}; + // The task-table slot count this image was built for, resolved per bind from + // runtime_env.ring_task_window. The device needs it to bound the pitch it + // attaches with. + uint64_t task_capacity{0}; // Total arena byte size post-commit. Used by host to size the prebuilt // image buffer and as the rtMemcpy length, and requested of the device as @@ -234,7 +235,7 @@ static_assert( * device memory and may run on host. Returns the layout descriptor; caller * commits/attaches the arena before Phase 2/3. */ -RuntimeArenaLayout runtime_reserve_layout(DeviceArena &arena, uint64_t task_window_size); +RuntimeArenaLayout runtime_reserve_layout(DeviceArena &arena, uint64_t task_capacity); /** * Phase 2 — write the data half of the runtime arena: standalone fields, @@ -307,7 +308,7 @@ void rt_scope_begin(RuntimeContext *rt); * * Closes the innermost scope, and when that is the outermost MANUAL one, returns * later submits to TensorMap discovery. A scope bounds no task or buffer lifetime - * here: the ring is whole-graph-resident, so no task slot and no heap byte is + * here: the task table is whole-graph-resident, so no task slot and no heap byte is * reclaimed before the run ends. */ void rt_scope_end(RuntimeContext *rt); diff --git a/src/a5/runtime/host_build_graph/runtime/runtime_types.h b/src/a5/runtime/host_build_graph/runtime/runtime_types.h index 7096f27841..e7c374b114 100644 --- a/src/a5/runtime/host_build_graph/runtime/runtime_types.h +++ b/src/a5/runtime/host_build_graph/runtime/runtime_types.h @@ -70,21 +70,29 @@ // ============================================================================= // Task management -// Actual window size is passed at runtime to runtime_create_from_sm(). -// The slot is local_id masked by the window size, which is why the window is a power of two. -#define CHIP_TASK_WINDOW_SIZE 16384 // Default task window size (power of 2) - -// host_build_graph has one ring, and carries no per-ring dimension anywhere: -// host-orch builds the whole graph on the host, it fits that ring, and the -// device runs it once without reclaim (execution-time recycle removed). tmr's -// multi-ring design existed only to -// let inner scopes reclaim independently under small rings; with no reclaim and -// a whole-graph-resident ring, per-depth isolation is moot, so every scope depth -// uses the same ring. The RuntimeEnv ABI still carries RUNTIME_ENV_RING_COUNT -// slots because it is shared with tmr; this runtime reads slot 0 (see -// bind_callable_to_runtime_impl). - -// Memory pools (total = value, single ring) +// +// The task table is a flat array of slots, indexed directly by local task id: +// ids start at 0, are never recycled, and alloc() caps them at the table's size, +// so there is no wrap and no slot mask. The size need not be a power of two — +// nothing masks with it. +// +// This is the default; `CallConfig.runtime_env.ring_task_window` overrides it per +// task. The host mirror is allocated at whatever size is in effect and committed +// by first touch, so a run pays only for the slots and argument-pool bytes it +// actually writes. Raising it costs virtual address space, bounded by the int32 +// reach of a payload's self-relative region deltas (checked in +// SharedMemoryHandle::init). +#define CHIP_DEFAULT_GRAPH_TASKS 16384 + +// host_build_graph carries no per-scope-depth task partition: host-orch builds +// the whole graph on the host and the device runs it once without reclaim. tmr's +// multi-ring design existed only to let inner scopes reclaim independently under +// small rings; with no reclaim and a whole-graph-resident task table, per-depth +// isolation is moot. The RuntimeEnv ABI still carries RUNTIME_ENV_RING_COUNT +// slots because it is shared with tmr; this runtime reads none of them and warns +// when one is set (see bind_callable_to_runtime_impl). + +// Memory pools (total = value) #define CHIP_TENSORMAP_POOL_SIZE (65536) // TensorMap entry pool #define CHIP_TENSORMAP_NUM_BUCKETS 4096 // Power of 2 for fast hash (4096×8B=32KB fits L1) @@ -191,10 +199,11 @@ typedef enum { /** * Result of a unified task allocation. + * + * There is no separate slot: a task id indexes the task table directly. */ struct TaskAllocResult { - int32_t task_id; // Absolute task ID (not wrapped) - int32_t slot; // task_id & (window_size - 1) + int32_t task_id; // Task id, which is also its task-table index void *packed_base; // Heap allocation result (nullptr if failure) void *packed_end; // packed_base + aligned output_size @@ -463,7 +472,7 @@ struct TaskPayload { // The ring's payload storage is reused raw memory that no constructor runs // over, so an unset predicate reads back as whatever the slot last held — // and compact_live_image translates predicate.addr as a graph-heap address - // for every submitted slot. An ordinary ring task overwrites this right + // for every submitted slot. An ordinary task overwrites this right // after init(); a hidden-alloc task has nothing following it, so this is // the only value its predicate ever gets. predicate = DispatchPredicate{}; @@ -593,7 +602,7 @@ struct alignas(64) ChipTaskSlotState { std::atomic next_block_idx{0}; // Graph scheduling metadata occupies the slot's tail padding. Ordinary - // ring tasks keep the index invalid and the context null. + // Ordinary tasks keep the index invalid and the context null. int32_t graph_node_index{-1}; void *graph_context{nullptr}; diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h index 6f4e50fdb5..65cf34b583 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h @@ -18,7 +18,7 @@ * producer named in its inline fanin has set its completion_flags byte; * a producer publishes completion + drains its wake list on finish * 3. Publishing the host-visible task_state mirror (PENDING -> COMPLETED) and - * advancing the per-ring completed_watermark (consumer-retirement signal) + * advancing the completed_watermark (consumer-retirement signal) * 4. Two-stage mixed-task completion (subtask done bits -> mixed-task complete) * * The Scheduler runs on Device AI_CPU. host_build_graph is scheduler-only (the @@ -39,7 +39,7 @@ #include "aicpu/platform_regs.h" // get_reg_ptr / RegId for the early-dispatch doorbell #include "async_wait.h" #include "graph_execution.h" -#include "ring_buffer.h" +#include "task_allocator.h" #include "runtime_types.h" #include "shared_memory.h" @@ -474,21 +474,19 @@ struct SchedulerState { // Shared memory access SharedMemoryHeader *sm_header; - // Per-ring state - struct alignas(64) RingSchedState { - // --- Cache Line 0: ring pointer (read-only) + hot path (read-write) --- - SharedMemoryRingHeader *ring; - std::atomic advance_lock; // multi-thread CAS + // The task table's header, as a device address + struct alignas(64) TaskHeaderView { + SharedMemoryTaskHeader *tasks; - // Polling: no per-ring dep_pool. Readiness is derived from the SM ring's + // Polling: no dep_pool. Readiness is derived from the task table's // completion_flags; there is no arena-side wiring pool to reserve or wire. - // The `ring` field stores the device address of the SM ring header — + // The `tasks` field stores the device address of the SM task header — // computed via offset arithmetic, no SM dereference. bool init_data_from_layout(void *sm_dev_base); void destroy(); - } ring_sched_state; + } task_view; - // Ready queues remain global (scheduling is ring-agnostic) + // Ready queues are global: scheduling does not partition by task id ChipReadyQueue ready_queues[NUM_RESOURCE_SHAPES]; // Ready sync_start queues, one per shape. A ready sync_start cohort parks here @@ -570,7 +568,7 @@ struct SchedulerState { } } - // ---- Polling completion primitives (single-ring hbg) ---------------------- + // ---- Polling completion primitives --------------------------------------- // Readiness: a task is ready iff every producer named in its inline fanin has // set its completion_flags byte. Single-ring: all producers are ring 0, so // there is no per-edge ring indirection. @@ -586,10 +584,10 @@ struct SchedulerState { // completion re-scans its waiters via on_mixed_task_complete's wake drain. int classify_fanin_state(const ChipTaskSlotState *s) const { const TaskPayload &p = *s->payload; - const SharedMemoryRingHeader &ring = *ring_sched_state.ring; + const SharedMemoryTaskHeader &tasks = *task_view.tasks; const int32_t *fanin = p.fanin_data(); for (int32_t i = p.fanin_count - 1; i >= 0; i--) { - if (!ring.is_completion_flag_set(fanin[i])) return i; + if (!tasks.is_completion_flag_set(fanin[i])) return i; } return -1; } @@ -599,7 +597,7 @@ struct SchedulerState { // ready only when every fanin is met, else re-target the next unmet producer // and retry. Monotonic completion_flags guarantee termination. void register_wake(ChipTaskSlotState *producer, ChipTaskSlotState *consumer) { - SharedMemoryRingHeader &ring = *ring_sched_state.ring; + SharedMemoryTaskHeader &tasks = *task_view.tasks; while (true) { ChipTaskSlotState *expected = producer->wake_list_head.load(std::memory_order_relaxed); while (expected != WAKE_LIST_SENTINEL) { @@ -615,7 +613,7 @@ struct SchedulerState { push_ready_routed(consumer); return; } - producer = &ring.get_slot_state_by_task_id(consumer->payload->fanin_data()[state]); + producer = &tasks.get_slot_state_by_task_id(consumer->payload->fanin_data()[state]); } } @@ -624,13 +622,13 @@ struct SchedulerState { // (route/re-register each waiter), then CAS-advance the monotonic // completed_watermark (load-bearing: the host wait_for_consumers gates on // watermark >= producer.last_consumer_local_id). Whole-graph-resident hbg - // has no device slot reclaim, so no advance_ring_pointers here. + // has no device slot reclaim, so nothing advances a reclaim cursor here. void on_mixed_task_complete(ChipTaskSlotState &slot_state) { const int32_t task_id = static_cast(slot_state.task->task_id.local()); - SharedMemoryRingHeader &ring = *ring_sched_state.ring; + SharedMemoryTaskHeader &tasks = *task_view.tasks; slot_state.mark_completed(); // host-visible mirror (task_state = COMPLETED) - ring.set_completion_flag(task_id); + tasks.set_completion_flag(task_id); ChipTaskSlotState *waiter = slot_state.wake_list_head.exchange(WAKE_LIST_SENTINEL, std::memory_order_acq_rel); while (waiter != nullptr && waiter != WAKE_LIST_SENTINEL) { @@ -644,7 +642,7 @@ struct SchedulerState { if (state < 0) { push_ready_routed(waiter); } else { - register_wake(&ring.get_slot_state_by_task_id(waiter->payload->fanin_data()[state]), waiter); + register_wake(&tasks.get_slot_state_by_task_id(waiter->payload->fanin_data()[state]), waiter); } waiter = next; } @@ -656,7 +654,7 @@ struct SchedulerState { // makes the final value order-dependent: a low-id task completing after a // higher one would leave the watermark stuck below the true prefix, hanging // any wait_for_consumers whose last_consumer sits in the gap. - ring.update_completed_watermark(); + tasks.update_completed_watermark(); } // Polling: there is no ready-claim CAS (a producer routes each waiter exactly @@ -1091,7 +1089,7 @@ struct SchedulerState { if (!graph_completed) return outcome; // Internal nodes count as zero stream tasks. The final node publishes - // the outer ring task exactly once, waking external consumers and + // the outer task exactly once, waking external consumers and // contributing the one task the host actually submitted. if (execution->outer_slot != nullptr) { on_mixed_task_complete(*execution->outer_slot); @@ -1170,10 +1168,9 @@ struct SchedulerState { // Phase 3a: write everything *except* arena-internal pointer fields. // `sm_dev_base` is the device address of the SM (only stored, never // dereferenced here). Safe to call on a host arena that holds the - // prebuilt image buffer. (The orchestrator counterpart takes - // task_window_size for ring task_descriptors address arithmetic; the - // scheduler only needs the SM header / ring header base addresses, - // both window-size-independent.) + // prebuilt image buffer. (The orchestrator counterpart takes task_capacity + // for its task_descriptors address arithmetic; the scheduler only needs the + // SM header and task header base addresses, both capacity-independent.) bool init_data_from_layout(const SchedulerLayout &layout, DeviceArena &arena, void *sm_dev_base); // Phase 3b: write the arena-internal pointer fields diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp index 703b18b835..6fc25dd78a 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp @@ -226,25 +226,26 @@ void SchedulerContext::log_stall_diagnostics( ) { CoreTracker &tracker = core_trackers_[thread_idx]; - // T0 owns the shared-ring scan; printing it from other threads would + // T0 owns the task-table scan; printing it from other threads would // produce identical TASK lines once per scheduler thread. if (thread_idx == 0) { - int32_t cnt_ready = 0, cnt_waiting = 0, cnt_running = 0, submitted_in_ring = 0; - SharedMemoryRingHeader &ring = *sched_->ring_sched_state.ring; - int32_t ring_task_count = ring.fc.current_task_index.load(std::memory_order_relaxed); - submitted_in_ring += ring_task_count; - for (int32_t si = 0; si < ring_task_count; si++) { - ChipTaskSlotState &slot_state = ring.get_slot_state_by_task_id(si); + int32_t cnt_ready = 0, cnt_waiting = 0, cnt_running = 0; + SharedMemoryTaskHeader &tasks = *sched_->task_view.tasks; + // `task_count` is the run's task total, which both callers pass from + // total_tasks_. It bounds the scan as well as the SUMMARY line: every slot + // below it was claimed by the host orchestrator, and no slot above it was. + for (int32_t si = 0; si < task_count; si++) { + ChipTaskSlotState &slot_state = tasks.get_slot_state_by_task_id(si); ChipTaskState st = slot_state.task_state.load(std::memory_order_relaxed); // Polling: no fanin_refcount. Recompute met/total from the inline - // fanin ids vs the ring completion_flags (rc = satisfied producers, + // fanin ids vs the completion_flags (rc = satisfied producers, // fi = raw producer count) so the stall dump still shows readiness. int32_t fi = slot_state.payload != nullptr ? slot_state.payload->fanin_count : 0; int32_t rc = 0; if (slot_state.payload != nullptr) { const int32_t *fanin = slot_state.payload->fanin_data(); for (int32_t k = 0; k < fi; k++) { - if (ring.is_completion_flag_set(fanin[k], std::memory_order_relaxed)) rc++; + if (tasks.is_completion_flag_set(fanin[k], std::memory_order_relaxed)) rc++; } } int32_t kid_aic = slot_state.task->kernel_id[0]; @@ -302,12 +303,11 @@ void SchedulerContext::log_stall_diagnostics( thread_idx, idle_iterations, 0, task_id, rc, fi, kid_aic, kid_aiv0, kid_aiv1, fi - rc ); } - int32_t effective_total = task_count > 0 ? task_count : submitted_in_ring; int32_t c = completed_tasks_.load(std::memory_order_relaxed); LOG_INFO( "[STALL thread=%d idle_iterations=%d] SUMMARY completed=%d/%d last_progress_iteration=%d " "scan_ready=%d scan_waiting=%d scan_running=%d", - thread_idx, idle_iterations, c, effective_total, last_progress_count, cnt_ready, cnt_waiting, cnt_running + thread_idx, idle_iterations, c, task_count, last_progress_count, cnt_ready, cnt_waiting, cnt_running ); } @@ -905,21 +905,10 @@ int32_t SchedulerContext::post_handshake_init(Runtime *runtime) { #endif // Initialize task counters. Task count comes from shared memory. - if (runtime->get_gm_sm_ptr()) { - auto *header = static_cast(runtime->get_gm_sm_ptr()); - // Read at one-time boot init, before the SM is reset for the run, so a ring - // not yet written holds uninitialized memory (0xbe... under ASAN's - // malloc-fill). Only a plausible count is taken — (0, - // CHIP_TASK_WINDOW_SIZE], since the ring cannot hold more than its task - // window — so any garbage pattern, negative or positive, leaves the count - // at 0, which is the correct value at boot. - int32_t window_tasks = 0; - int32_t ring_tasks = header->ring.fc.current_task_index.load(std::memory_order_acquire); - if (ring_tasks > 0 && ring_tasks <= CHIP_TASK_WINDOW_SIZE) window_tasks = ring_tasks; - total_tasks_ = window_tasks; - } else { - total_tasks_ = 0; - } + // 0 is the correct count at boot: the graph is not attached yet, and + // on_orchestration_done latches the host-built total before releasing any + // scheduler thread. + total_tasks_ = 0; completed_tasks_.store(0, std::memory_order_release); // prepare_subtask_to_core fully writes a per-core payload / deferred-slab slot @@ -1041,10 +1030,9 @@ void SchedulerContext::bind_runtime(RuntimeContext *rt) { // ============================================================================= // Post-orchestration bookkeeping. Runs once on the boot leader after the // host-built image is attached; latches total_tasks_ and folds inline-completed -// tasks (or shuts down on a fatal orchestration error). The caller publishes -// runtime_init_ready_ (release) after this returns — that store is what makes -// total_tasks_ visible to the scheduler threads, which acquire it before -// dispatching. +// tasks (or shuts down on a fatal orchestration error). classify_ready_ is +// released after this call and is what publishes total_tasks_ to the peer threads, +// which acquire it before classify_partition reads the count. // ============================================================================= void SchedulerContext::on_orchestration_done( Runtime *runtime, RuntimeContext *rt, [[maybe_unused]] int32_t thread_idx, int32_t total_tasks @@ -1053,18 +1041,12 @@ void SchedulerContext::on_orchestration_done( // Allocate the per-S CompletedTaskQueues here on the boot leader, before it // releases runtime_init_ready_ — no scheduler thread can push until then. - // Completed-but-unresolved tasks in flight are bounded by BOTH the total task - // count and the ring's task window (a task must occupy a ring slot to run and - // complete), so size to the tighter of the two, rounded up to a power of two - // and floored at 256. The window already caps this, so there is no artificial - // ceiling and a producer never has to spin on a full queue. + // Completed-but-unresolved tasks in flight are bounded by the run's task count + // (a task must have been submitted to run and complete), so size to that, + // rounded up to a power of two and floored at 256. That bound is exact, so + // there is no artificial ceiling and a producer never has to spin on a full + // queue. uint64_t sp_bound = static_cast(total_tasks); - if (sched_->ring_sched_state.ring != nullptr) { - uint64_t window = static_cast(sched_->ring_sched_state.ring->task_window_mask) + 1; - if (window < sp_bound) { - sp_bound = window; - } - } uint64_t sp_cap = 256; while (sp_cap < sp_bound) { sp_cap <<= 1; @@ -1120,20 +1102,20 @@ void SchedulerContext::on_orchestration_done( // their external fanin follows the same ready/wake classification as any other // outer task. void SchedulerContext::classify_partition(int32_t thread_idx, int32_t nthreads) { - if (completed_.load(std::memory_order_acquire) || sched_->ring_sched_state.ring == nullptr) { + if (completed_.load(std::memory_order_acquire) || sched_->task_view.tasks == nullptr) { return; } - SharedMemoryRingHeader &ring = *sched_->ring_sched_state.ring; - const int32_t submitted = ring.fc.current_task_index.load(std::memory_order_acquire); + SharedMemoryTaskHeader &tasks = *sched_->task_view.tasks; + const int32_t submitted = total_tasks_; // Disjoint contiguous slices covering [0, submitted): thread t owns // [submitted*t/nthreads, submitted*(t+1)/nthreads). int64 math avoids overflow. const int32_t lo = static_cast((static_cast(submitted) * thread_idx) / nthreads); const int32_t hi = static_cast((static_cast(submitted) * (thread_idx + 1)) / nthreads); for (int32_t id = lo; id < hi; id++) { - if (ring.is_completion_flag_set(id)) { + if (tasks.is_completion_flag_set(id)) { continue; // completed on the host (hidden alloc); nothing to dispatch } - ChipTaskSlotState &slot = ring.get_slot_state_by_task_id(id); + ChipTaskSlotState &slot = tasks.get_slot_state_by_task_id(id); if (slot.task_kind == TaskKind::GRAPH) { if (graph_execution_localize(slot) == nullptr) slot.graph_context = nullptr; if (!sched_->push_graph_prepare(&slot, slot.task->task_id.raw, thread_idx)) return; @@ -1143,7 +1125,7 @@ void SchedulerContext::classify_partition(int32_t thread_idx, int32_t nthreads) sched_->push_ready_routed(&slot); } else { int32_t prod_local = slot.payload->fanin_data()[state]; - sched_->register_wake(&ring.get_slot_state_by_task_id(prod_local), &slot); + sched_->register_wake(&tasks.get_slot_state_by_task_id(prod_local), &slot); } } } diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_context.h b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_context.h index 6b69d1766d..7beec8d95d 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_context.h +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_context.h @@ -41,8 +41,9 @@ struct RuntimeContext; // reclaims a slot, so the queued ChipTaskSlotState* stays valid until P reads it. // Single producer (the owning S thread) / single consumer (P): plain // acquire/release on head/tail, no CAS. Capacity is a power of two sized to the -// in-flight bound (min of total tasks and the ring window), so a producer does -// not spin on a full queue in practice; the spin is a correctness backstop only +// run's task count, which bounds the completed-but-unresolved tasks in flight +// exactly, so a producer does not spin on a full queue in practice; the spin is a +// correctness backstop only // (P drains independently, so it always makes progress). The std::atomic cursors // make this type non-copyable / non-movable, so `buf` cannot be double-freed // through an accidental copy. diff --git a/src/a5/runtime/host_build_graph/runtime/shared/runtime_init.cpp b/src/a5/runtime/host_build_graph/runtime/shared/runtime_init.cpp index 88c84f1a99..75b7a1f3bb 100644 --- a/src/a5/runtime/host_build_graph/runtime/shared/runtime_init.cpp +++ b/src/a5/runtime/host_build_graph/runtime/shared/runtime_init.cpp @@ -24,7 +24,7 @@ #include "orchestrator.h" #include "runtime_core.h" -#include "ring_buffer.h" +#include "task_allocator.h" #include "shared_memory.h" #include "tensormap.h" #include "scheduler/scheduler.h" @@ -63,11 +63,10 @@ void ready_queue_destroy(ChipReadyQueue *queue) { // Scheduler // ============================================================================= -bool SchedulerState::RingSchedState::init_data_from_layout(void *sm_dev_base) { - // ring stores the device address of the SM ring header — pure offset +bool SchedulerState::TaskHeaderView::init_data_from_layout(void *sm_dev_base) { + // `tasks` is the device address of the SM task header — pure offset // arithmetic, no SM load. - ring = sm_layout::ring_header_addr(sm_dev_base); - advance_lock.store(0, std::memory_order_relaxed); + tasks = sm_layout::task_header_addr(sm_dev_base); // Per-slot SM-side initialization (reset_for_reuse + active_mask, and clearing // the completion flag) happens init-on-write in orch::prepare_task as each slot @@ -76,7 +75,7 @@ bool SchedulerState::RingSchedState::init_data_from_layout(void *sm_dev_base) { return true; } -void SchedulerState::RingSchedState::destroy() { ring = nullptr; } +void SchedulerState::TaskHeaderView::destroy() { tasks = nullptr; } SchedulerLayout SchedulerState::reserve_layout(DeviceArena &arena) { SchedulerLayout layout{}; @@ -111,7 +110,7 @@ bool SchedulerState::init_data_from_layout(const SchedulerLayout &layout, Device sched->tasks_consumed.store(0, std::memory_order_relaxed); #endif - if (!sched->ring_sched_state.init_data_from_layout(sm_dev_base)) { + if (!sched->task_view.init_data_from_layout(sm_dev_base)) { return false; } @@ -177,7 +176,7 @@ void SchedulerState::wire_arena_pointers(const SchedulerLayout &layout, DeviceAr void SchedulerState::destroy() { SchedulerState *sched = this; - sched->ring_sched_state.destroy(); + sched->task_view.destroy(); for (int i = 0; i < NUM_RESOURCE_SHAPES; i++) { ready_queue_destroy(&sched->ready_queues[i]); } @@ -198,42 +197,40 @@ void SchedulerState::destroy() { // ============================================================================= bool OrchestratorState::init( - void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t task_window_size, SchedulerState *scheduler_arg + void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t max_tasks, SchedulerState *scheduler_arg ) { auto *orch = this; *orch = OrchestratorState{}; - // A power-of-two window lets the slot index mask instead of dividing. - always_assert(task_window_size > 0 && (task_window_size & (task_window_size - 1)) == 0); + always_assert(max_tasks > 0); orch->sm_header = reinterpret_cast(sm_base); orch->fatal = false; orch->scheduler = scheduler_arg; auto *orch_err = sm_layout::orch_error_code_addr(sm_base); - auto *cur_idx_dev = sm_layout::ring_current_task_index_addr(sm_base); - orch->task_allocator.init(static_cast(task_window_size), cur_idx_dev, gm_heap, heap_size, orch_err); + orch->task_allocator.init(static_cast(max_tasks), gm_heap, heap_size, orch_err); // The mirror's argument pools. Offset arithmetic on the same base as sm_header, // so it holds for whichever SM this orchestrator was pointed at. The cursors // reset with the rest of the state above. auto *sm_bytes = static_cast(sm_base); - const auto pools = sm_layout::ring_segment_offsets(sm_layout::mirror_extents(task_window_size)); + const auto pools = sm_layout::segment_offsets(sm_layout::mirror_extents(max_tasks)); orch->fanin_pool = reinterpret_cast(sm_bytes + pools.fanin_pool); orch->tensor_pool = reinterpret_cast(sm_bytes + pools.tensor_pool); orch->scalar_pool = reinterpret_cast(sm_bytes + pools.scalar_pool); // Polling: no fanin-spill pool — producer ids are inline on the payload. - const auto window = static_cast(task_window_size); - orch->fanin_seen_epoch.reset(new (std::nothrow) uint32_t[window]); + const auto slots = static_cast(max_tasks); + orch->fanin_seen_epoch.reset(new (std::nothrow) uint32_t[slots]); if (orch->fanin_seen_epoch == nullptr) { - LOG_ERROR("Orchestrator scratch allocation failed (task_window=%" PRIu64 ")", task_window_size); + LOG_ERROR("Orchestrator scratch allocation failed (max_tasks=%" PRIu64 ")", max_tasks); return false; } - memset(orch->fanin_seen_epoch.get(), 0, window * sizeof(uint32_t)); + memset(orch->fanin_seen_epoch.get(), 0, slots * sizeof(uint32_t)); - if (!orch->tensor_map.init_default(static_cast(task_window_size))) { + if (!orch->tensor_map.init_default(static_cast(max_tasks))) { return false; } @@ -249,10 +246,10 @@ void OrchestratorState::set_scheduler(SchedulerState *scheduler) { this->schedul // Top-level runtime arena // ============================================================================= -RuntimeArenaLayout runtime_reserve_layout(DeviceArena &arena, uint64_t task_window_size) { +RuntimeArenaLayout runtime_reserve_layout(DeviceArena &arena, uint64_t task_capacity) { RuntimeArenaLayout layout{}; - layout.task_window_size = task_window_size; + layout.task_capacity = task_capacity; // Reservation order is the zone partition (see RuntimeArenaLayout): // everything the device initializes itself, then the one copied range. Each diff --git a/src/a5/runtime/host_build_graph/runtime/shared/shared_memory.cpp b/src/a5/runtime/host_build_graph/runtime/shared/shared_memory.cpp index 35e3e044fe..a497349736 100644 --- a/src/a5/runtime/host_build_graph/runtime/shared/shared_memory.cpp +++ b/src/a5/runtime/host_build_graph/runtime/shared/shared_memory.cpp @@ -27,10 +27,10 @@ // Size Calculation // ============================================================================= -uint64_t SharedMemoryHandle::calculate_size(uint64_t task_window_size) { - // Total SM size = offset just past the ring's slot_states, from the single - // source of truth for the layout (sm_layout::ring_segment_offsets). - return sm_layout::ring_segment_offsets(task_window_size).end; +uint64_t SharedMemoryHandle::calculate_size(uint64_t max_tasks) { + // Total SM size = offset just past the last segment, from the single + // source of truth for the layout (sm_layout::segment_offsets). + return sm_layout::segment_offsets(max_tasks).end; } // ============================================================================= @@ -41,60 +41,59 @@ void SharedMemoryHandle::setup_pointers(uint64_t pitch) { char *base = (char *)sm_base; header = (SharedMemoryHeader *)base; - // Per-ring descriptors / payloads / slot_states — offsets from the single source - // of truth (sm_layout::ring_segment_offsets), so this setup and the + // descriptors / payloads / slot_states — offsets from the single source + // of truth (sm_layout::segment_offsets), so this setup and the // device-address helpers cannot drift. // // Only the four slot-pitched segments, and no argument pool: the pool extents // differ between the mirror and the image, while these four depend on the pitch // alone. The orchestrator holds the mirror's pool bases; the device resolves an // argument region through the payload's delta and needs no base at all. - auto off = sm_layout::ring_segment_offsets(sm_layout::image_extents({pitch, 0, 0, 0})); - auto &ring = header->ring; - ring.task_descriptors = (TaskDescriptor *)(base + off.descriptors); - ring.task_payloads = (TaskPayload *)(base + off.payloads); - ring.slot_states = (ChipTaskSlotState *)(base + off.slot_states); - ring.completion_flags = (std::atomic *)(base + off.completion_flags); + auto off = sm_layout::segment_offsets(sm_layout::image_extents({pitch, 0, 0, 0})); + auto &tasks = header->tasks; + tasks.task_descriptors = (TaskDescriptor *)(base + off.descriptors); + tasks.task_payloads = (TaskPayload *)(base + off.payloads); + tasks.slot_states = (ChipTaskSlotState *)(base + off.slot_states); + tasks.completion_flags = (std::atomic *)(base + off.completion_flags); } -bool SharedMemoryHandle::init(void *sm_base_arg, uint64_t sm_size_arg, uint64_t task_window_size) { +bool SharedMemoryHandle::init(void *sm_base_arg, uint64_t sm_size_arg, uint64_t max_tasks) { if (!sm_base_arg || sm_size_arg == 0) return false; - const uint64_t mirror_bytes = calculate_size(task_window_size); + const uint64_t mirror_bytes = calculate_size(max_tasks); // The mirror has to stay inside an int32 delta's reach: a payload names its // argument regions that way, and set() leaves a field unbound rather than // truncating, so a pool out of reach would read back as null and be dereferenced // as one. attach_populated makes the same check on the shipped image. // - // Tested before the region's own size, because this rejects the ring capacity - // itself: no region, however large, makes such a window usable. + // Tested before the region's own size, because this rejects `max_tasks` itself: + // no region, however large, makes such a reservation usable. if (mirror_bytes > static_cast(INT32_MAX)) return false; if (sm_size_arg < mirror_bytes) return false; sm_base = sm_base_arg; sm_size = sm_size_arg; is_owner = false; - setup_pointers(task_window_size); - init_header(task_window_size); + setup_pointers(max_tasks); + init_header(); return true; } bool SharedMemoryHandle::attach_populated( - void *sm_base_arg, uint64_t sm_size_arg, uint64_t task_window_size, uint64_t live_slots, uint64_t image_bytes + void *sm_base_arg, uint64_t sm_size_arg, uint64_t max_tasks, uint64_t live_slots, uint64_t image_bytes ) { if (!sm_base_arg || sm_size_arg == 0) return false; - // A pitch above the capacity would name a slot the ring cannot address, and one + // A pitch above `max_tasks` names a slot the host could not have built, and one // below what the host used would place every segment past the descriptors // short. This is the whole contract between the two sides, checked once at // attach rather than trusted. - if (live_slots == 0 || live_slots > task_window_size) return false; + if (live_slots == 0 || live_slots > max_tasks) return false; // The image must at least hold the four slot-pitched segments; anything beyond // that is its argument pools, whose extents are the bind's cursors and are not // recomputable here. The first pool's offset is exactly where those four end. - const uint64_t slots_end = - sm_layout::ring_segment_offsets(sm_layout::image_extents({live_slots, 0, 0, 0})).fanin_pool; + const uint64_t slots_end = sm_layout::segment_offsets(sm_layout::image_extents({live_slots, 0, 0, 0})).fanin_pool; if (image_bytes < slots_end) return false; - // The region holds the compacted image, so that is what has to fit — the ring - // capacity is no longer its size. + // The region holds the compacted image, so that is what has to fit — the + // dimensioned size is no longer its size. if (sm_size_arg < image_bytes) return false; // A slot state names its payload and descriptor by an int32 delta from its own // address, and a payload names its argument regions the same way, so no two @@ -106,12 +105,12 @@ bool SharedMemoryHandle::attach_populated( is_owner = false; setup_pointers(live_slots); // Deliberately NO init_header: the SM already holds the host orchestrator's - // task graph (descriptors, slot states, ring counters). + // task graph (descriptors, slot states, completion flags). return true; } SharedMemoryHandle *SharedMemoryHandle::create_and_init_default(DeviceArena &arena) { - const uint64_t buffer_size = calculate_size(CHIP_TASK_WINDOW_SIZE); + const uint64_t buffer_size = calculate_size(CHIP_DEFAULT_GRAPH_TASKS); const size_t off_handle = arena.reserve(sizeof(SharedMemoryHandle), alignof(SharedMemoryHandle)); const size_t off_buffer = arena.reserve(static_cast(buffer_size), CHIP_ALIGN_SIZE); if (arena.commit() == nullptr) return nullptr; @@ -120,7 +119,7 @@ SharedMemoryHandle *SharedMemoryHandle::create_and_init_default(DeviceArena &are memset(handle, 0, sizeof(*handle)); void *buffer = arena.region_ptr(off_buffer); memset(buffer, 0, static_cast(buffer_size)); - if (!handle->init(buffer, buffer_size, CHIP_TASK_WINDOW_SIZE)) return nullptr; + if (!handle->init(buffer, buffer_size, CHIP_DEFAULT_GRAPH_TASKS)) return nullptr; return handle; } @@ -138,24 +137,21 @@ void SharedMemoryHandle::destroy() { // ============================================================================= // // no need init data in pool, init pool data when used -void SharedMemoryHandle::init_header(uint64_t task_window_size) { - // Flow control (starts at 0) - header->ring.fc.init(); - +void SharedMemoryHandle::init_header() { // Polling completion: -1 = "no task completed yet"; the first task to // complete (local_id 0) advances the watermark to 0. - header->ring.completed_watermark.store(-1, std::memory_order_relaxed); + header->tasks.completed_watermark.store(-1, std::memory_order_relaxed); + + // 0 until run_host_orchestration writes the run's count; init runs before + // orchestration, so the real value is not known here yet. + header->tasks.total_tasks = 0; header->orchestrator_done.store(0, std::memory_order_relaxed); - // Ring layout info - uint64_t offset = CHIP_ALIGN_UP(sizeof(SharedMemoryHeader), CHIP_ALIGN_SIZE); - header->ring.task_window_size = task_window_size; - header->ring.task_window_mask = static_cast(task_window_size - 1); - header->ring.task_descriptors_offset = offset; - offset += CHIP_ALIGN_UP(task_window_size * sizeof(TaskDescriptor), CHIP_ALIGN_SIZE); - offset += CHIP_ALIGN_UP(task_window_size * sizeof(TaskPayload), CHIP_ALIGN_SIZE); - offset += CHIP_ALIGN_UP(task_window_size * sizeof(ChipTaskSlotState), CHIP_ALIGN_SIZE); + // Layout info. The descriptors are the first segment, so their offset is where + // the header's own padded size ends — pitch-independent, unlike every segment + // after them. + header->tasks.task_descriptors_offset = CHIP_ALIGN_UP(sizeof(SharedMemoryHeader), CHIP_ALIGN_SIZE); header->total_size = sm_size; @@ -168,8 +164,8 @@ void SharedMemoryHandle::init_header(uint64_t task_window_size) { // Per-slot init (slot_states.reset_for_reuse() + active_mask, and clearing the // completion flag) happens init-on-write in orch::prepare_task as each slot // [0, total_tasks) is claimed, so the SM init/upload cost tracks the task - // count, not ring capacity. The device reads no slot past total_tasks, so the - // unclaimed tail is left uninitialized. + // count, not the size the table was dimensioned for. The device reads no slot + // past total_tasks, so the unclaimed tail is left uninitialized. } // ============================================================================= @@ -184,13 +180,12 @@ void SharedMemoryHandle::print_layout() { LOG_DEBUG("=== Shared Memory Layout ==="); LOG_DEBUG("Base address: %p", sm_base); LOG_DEBUG("Total size: %" PRIu64 " bytes", h->total_size); - LOG_DEBUG("Ring:"); - LOG_DEBUG(" task_window_size: %" PRIu64, h->ring.task_window_size); + LOG_DEBUG("Task table:"); LOG_DEBUG( - " descriptors_off: %" PRIu64 " (0x%" PRIx64 ")", h->ring.task_descriptors_offset, - h->ring.task_descriptors_offset + " descriptors_off: %" PRIu64 " (0x%" PRIx64 ")", h->tasks.task_descriptors_offset, + h->tasks.task_descriptors_offset ); - LOG_DEBUG(" current_task_idx: %d", h->ring.fc.current_task_index.load(std::memory_order_acquire)); + LOG_DEBUG(" completed_wm: %d", h->tasks.completed_watermark.load(std::memory_order_acquire)); LOG_DEBUG("orchestrator_done: %d", h->orchestrator_done.load(std::memory_order_acquire)); LOG_DEBUG("Error state:"); LOG_DEBUG(" orch_error_code: %d", h->orch_error_code.load(std::memory_order_relaxed)); @@ -204,28 +199,13 @@ bool SharedMemoryHandle::validate() { if (!sm_base) return false; if (!header) return false; - SharedMemoryHeader *h = header; - - if (!h->ring.fc.validate(this)) return false; - - return true; -} - -bool ChipRingFlowControl::validate(SharedMemoryHandle *handle) const { - if (!handle) return false; - if (!handle->header) return false; - - const SharedMemoryHeader *h = handle->header; + const SharedMemoryHeader *h = header; // Check that offsets are within bounds - if (h->ring.task_descriptors_offset >= h->total_size) return false; + if (h->tasks.task_descriptors_offset >= h->total_size) return false; // Check pointer alignment - if ((uintptr_t)h->ring.task_descriptors % CHIP_ALIGN_SIZE != 0) return false; - - // Check flow control pointer sanity - int32_t current = current_task_index.load(std::memory_order_acquire); - if (current < 0) return false; + if ((uintptr_t)h->tasks.task_descriptors % CHIP_ALIGN_SIZE != 0) return false; return true; } diff --git a/src/a5/runtime/host_build_graph/runtime/shared/tensormap.cpp b/src/a5/runtime/host_build_graph/runtime/shared/tensormap.cpp index 82799bff20..f75320d63d 100644 --- a/src/a5/runtime/host_build_graph/runtime/shared/tensormap.cpp +++ b/src/a5/runtime/host_build_graph/runtime/shared/tensormap.cpp @@ -54,40 +54,39 @@ uint64_t g_insert_count = 0; * Allocate the four arrays and reset the map to empty. Clears the bucket heads * and the per-task entry heads and resets the pool cursors (next_entry_idx / * free_num); the entry pool is left uninitialized and initialized on write by - * new_entry(), so this is O(num_buckets + task_window_size), not O(pool_size). + * new_entry(), so this is O(num_buckets + max_tasks), not O(pool_size). * * `new (std::nothrow) T[n]` rather than a vector: a vector would value-initialize * the whole entry pool, which is megabytes of zeroing the init-on-write path * exists to avoid, and nothrow keeps the failure reportable to a caller that * still has an alternative path. */ -bool ChipTensorMap::init(int32_t new_num_buckets, int32_t new_pool_size, int32_t new_task_window_size) { - // num_buckets must be a power of two for the hash truncation to work, and - // task_window_size for the task-chain slot mask. +bool ChipTensorMap::init(int32_t new_num_buckets, int32_t new_pool_size, int32_t new_max_tasks) { + // num_buckets must be a power of two for the hash truncation to work. The + // task-chain count needs no such property: a task id indexes its chain directly. always_assert(new_num_buckets > 0 && (new_num_buckets & (new_num_buckets - 1)) == 0); - always_assert(new_task_window_size > 0 && (new_task_window_size & (new_task_window_size - 1)) == 0); + always_assert(new_max_tasks > 0); always_assert(new_pool_size > 0); buckets.reset(new (std::nothrow) ChipTensorMapEntry *[new_num_buckets]); entry_pool.reset(new (std::nothrow) ChipTensorMapEntry[new_pool_size]); free_entry_list.reset(new (std::nothrow) ChipTensorMapEntry *[new_pool_size]); - task_entry_heads.reset(new (std::nothrow) ChipTensorMapEntry *[new_task_window_size]); + task_entry_heads.reset(new (std::nothrow) ChipTensorMapEntry *[new_max_tasks]); if (buckets == nullptr || entry_pool == nullptr || free_entry_list == nullptr || task_entry_heads == nullptr) { LOG_ERROR( - "TensorMap init failed (buckets=%d, pool=%d, window=%d)", new_num_buckets, new_pool_size, - new_task_window_size + "TensorMap init failed (buckets=%d, pool=%d, max_tasks=%d)", new_num_buckets, new_pool_size, new_max_tasks ); return false; } num_buckets = new_num_buckets; pool_size = new_pool_size; - task_window_size = new_task_window_size; + max_tasks = new_max_tasks; for (int32_t i = 0; i < num_buckets; i++) { buckets[i] = nullptr; } - for (int32_t i = 0; i < task_window_size; i++) { + for (int32_t i = 0; i < max_tasks; i++) { task_entry_heads[i] = nullptr; } @@ -111,7 +110,7 @@ void ChipTensorMap::reset() { for (int32_t i = 0; i < num_buckets; i++) { buckets[i] = nullptr; } - for (int32_t i = 0; i < task_window_size; i++) { + for (int32_t i = 0; i < max_tasks; i++) { task_entry_heads[i] = nullptr; } // Entries are reached only through a bucket chain, and every chain is now empty, so @@ -121,8 +120,8 @@ void ChipTensorMap::reset() { free_num = 0; } -bool ChipTensorMap::init_default(int32_t new_task_window_size) { - return init(CHIP_TENSORMAP_NUM_BUCKETS, CHIP_TENSORMAP_POOL_SIZE, new_task_window_size); +bool ChipTensorMap::init_default(int32_t new_max_tasks) { + return init(CHIP_TENSORMAP_NUM_BUCKETS, CHIP_TENSORMAP_POOL_SIZE, new_max_tasks); } // ============================================================================= diff --git a/src/a5/runtime/host_build_graph/runtime/shared_memory.h b/src/a5/runtime/host_build_graph/runtime/shared_memory.h index 4e441af4ce..13d5117930 100644 --- a/src/a5/runtime/host_build_graph/runtime/shared_memory.h +++ b/src/a5/runtime/host_build_graph/runtime/shared_memory.h @@ -9,13 +9,13 @@ * ----------------------------------------------------------------------------------------------------------- */ /** - * PTO Runtime2 - Shared Memory Layout + * Shared Memory Layout * * Defines the shared memory structure for Orchestrator-Scheduler communication. * - * Memory Layout (single ring): + * Memory Layout: * +---------------------------+ - * | SharedMemoryHeader | (flow control + sync) + * | SharedMemoryHeader | (completion watermark + sync + error state) * +---------------------------+ * | TaskDescriptor[] | * | TaskPayload[] | @@ -24,8 +24,8 @@ * * Design principles: * - Only data needed for Orchestrator<->Scheduler communication is here - * - TensorMap, scope_stack, ready_queues, dep_pool are in private memory - * - Flow control via atomic counters/flags (no locks needed for single-word R/W) + * - TensorMap, scope_stack and ready_queues are in private memory + * - Synchronization via atomic counters/flags (no locks needed for single-word R/W) * * Based on: docs/RUNTIME_LOGIC.md */ @@ -47,37 +47,19 @@ struct SharedMemoryHandle; /** - * Per-ring flow control state in shared memory. - * Written/read by Orchestrator and Scheduler for synchronization. - */ -struct alignas(64) ChipRingFlowControl { - // Written by Orchestrator, read by Scheduler. There is no reverse channel: - // the ring is whole-graph-resident, so the scheduler never reclaims task - // slots and has nothing to publish back. - alignas(64) std::atomic current_task_index; // Task ring head (next to allocate) - - // Per-boot SM reset. TaskAllocator::init() seeds its private - // local_task_id_ to 0 *without* dereferencing current_task_index — it - // relies on this reset running on every AICPU boot so 0 stays in sync. If - // you ever change the initial fc value or the boot ordering, update - // TaskAllocator::init (ring_buffer.h) in the same change, or - // submit IDs will be off by the divergence. - void init() { current_task_index.store(0, std::memory_order_relaxed); } - - bool validate(SharedMemoryHandle *handle) const; -}; - -static_assert(sizeof(ChipRingFlowControl) == 64, "ChipRingFlowControl must be exactly one cache line (64B)"); - -/** - * Per-ring shared memory header section. + * The task table's header in shared memory. * - * Groups flow-control, layout info, and per-ring data pointers for a single ring. - * Pointers are host-side only (set by setup_pointers, invalid on device). + * Groups the completion watermark, layout info, and the pointers to the four + * slot-pitched segments. Pointers are host-side only (set by setup_pointers, + * invalid on device). + * + * The run's task total sits here too, as a plain scalar. The graph is complete + * before the device starts, so the host writes it once into the mirror after + * orchestration and the restack ships it with the rest of the header; the device + * only ever reads it. Nothing publishes it incrementally, so it needs neither an + * atomic nor a cache line of its own. */ -struct alignas(64) SharedMemoryRingHeader { - ChipRingFlowControl fc; - +struct alignas(64) SharedMemoryTaskHeader { // Highest task_id such that every task with id in [0, completed_watermark] // has its completion_flags byte set. Advanced over the full contiguous // completed prefix at task-completion time (on_mixed_task_complete). The host @@ -87,11 +69,9 @@ struct alignas(64) SharedMemoryRingHeader { alignas(64) std::atomic completed_watermark; // Layout metadata (set once at init) - alignas(64) uint64_t task_window_size; - int32_t task_window_mask; - uint64_t task_descriptors_offset; // Offset from SM base, in bytes + alignas(64) uint64_t task_descriptors_offset; // Offset from SM base, in bytes - // Per-ring data pointers (host-side, set by setup_pointers) + // Segment pointers (host-side, set by setup_pointers) TaskDescriptor *task_descriptors; TaskPayload *task_payloads; ChipTaskSlotState *slot_states; @@ -100,25 +80,31 @@ struct alignas(64) SharedMemoryRingHeader { // 0 = pending, 1 = task fully COMPLETED. Writer = the task's completer at // on_mixed_task_complete; reader = consumer fanin polling (is_completion_flag_set). // Cleared per-slot in orch::prepare_task as each slot is claimed. Indexed by - // local_id & task_window_mask. + // local task id, like every other segment. std::atomic *completion_flags; + // Tasks this run submitted, i.e. the slot count the four segments above are + // pitched to. Written once by the host after orchestration (run_host_orchestration) + // and read-only from then on, so it packs into the padding rather than taking a + // line of its own. Bounds the completed_watermark walk: no slot at or above it was + // claimed, and the bytes past completion_flags[total_tasks - 1] are not flags. + int32_t total_tasks; + bool is_completion_flag_set(int32_t local_id, std::memory_order order = std::memory_order_acquire) const { - return completion_flags[local_id & task_window_mask].load(order) != 0; + return completion_flags[local_id].load(order) != 0; } void set_completion_flag(int32_t local_id, std::memory_order order = std::memory_order_release) const { - completion_flags[local_id & task_window_mask].store(1, order); + completion_flags[local_id].store(1, order); } // set completion flag first before updating the watermark (logic requirement) void update_completed_watermark() { int32_t curr_watermark = completed_watermark.load(std::memory_order_acquire); - const int32_t submitted = fc.current_task_index.load(std::memory_order_acquire); int32_t next = curr_watermark; while (true) { - while (next + 1 < submitted && is_completion_flag_set(next + 1)) { + while (next + 1 < total_tasks && is_completion_flag_set(next + 1)) { ++next; } if (next == curr_watermark) { @@ -137,37 +123,30 @@ struct alignas(64) SharedMemoryRingHeader { } } - int32_t get_slot_by_task_id(int32_t local_task_id) { return local_task_id & task_window_mask; } - - TaskDescriptor &get_task_by_slot(int32_t slot) { return task_descriptors[slot]; } + // A task id is its own slot index, so every segment is indexed directly. + TaskDescriptor &get_task_by_task_id(int32_t local_id) { return task_descriptors[local_id]; } - TaskDescriptor &get_task_by_task_id(int32_t local_id) { return task_descriptors[get_slot_by_task_id(local_id)]; } + // No get_payload_by_task_id here: a payload is reached through its slot + // state's `payload` delta, which the image's restack rebinds to the real + // address. - // No get_payload_by_slot / get_payload_by_task_id here: a payload is reached - // through its slot state's `payload` delta, which the image's restack rebinds to - // the real address. - - ChipTaskSlotState &get_slot_state_by_slot(int32_t slot) { return slot_states[slot]; } - - ChipTaskSlotState &get_slot_state_by_task_id(int32_t local_id) { - return slot_states[get_slot_by_task_id(local_id)]; - } + ChipTaskSlotState &get_slot_state_by_task_id(int32_t local_id) { return slot_states[local_id]; } }; -static_assert(sizeof(SharedMemoryRingHeader) == 192, "SharedMemoryRingHeader layout drift"); +static_assert(sizeof(SharedMemoryTaskHeader) == 128, "SharedMemoryTaskHeader layout drift"); static_assert( - offsetof(SharedMemoryRingHeader, task_descriptors_offset) == 144, - "SharedMemoryRingHeader task_descriptors_offset layout drift" + offsetof(SharedMemoryTaskHeader, task_descriptors_offset) == 64, + "SharedMemoryTaskHeader task_descriptors_offset layout drift" ); /** * Shared memory header structure * - * Contains per-ring flow control and global layout information. + * Contains the task table's header plus the run's global sync and error state. */ struct alignas(CHIP_ALIGN_SIZE) SharedMemoryHeader { - // === RING FLOW CONTROL + LAYOUT INFO (single ring, set once at init) === - SharedMemoryRingHeader ring; + // === TASK TABLE HEADER (set once at init) === + SharedMemoryTaskHeader tasks; // === GLOBAL FIELDS === std::atomic orchestrator_done; // Flag: orchestration complete @@ -188,9 +167,9 @@ struct alignas(CHIP_ALIGN_SIZE) SharedMemoryHeader { std::atomic sched_error_thread; // Thread index of last error writer }; -static_assert(sizeof(SharedMemoryHeader) == 256, "SharedMemoryHeader layout drift"); -static_assert(offsetof(SharedMemoryHeader, total_size) == 200, "SharedMemoryHeader total_size layout drift"); -static_assert(offsetof(SharedMemoryHeader, orch_error_code) == 208, "SharedMemoryHeader orch_error_code layout drift"); +static_assert(sizeof(SharedMemoryHeader) == 192, "SharedMemoryHeader layout drift"); +static_assert(offsetof(SharedMemoryHeader, total_size) == 136, "SharedMemoryHeader total_size layout drift"); +static_assert(offsetof(SharedMemoryHeader, orch_error_code) == 144, "SharedMemoryHeader orch_error_code layout drift"); // ============================================================================= // Shared Memory Handle @@ -211,10 +190,12 @@ struct SharedMemoryHandle { // === Static helpers === - static uint64_t calculate_size(uint64_t task_window_size); + // Bytes an SM image spans when dimensioned for `max_tasks` slots — the count + // the bind resolved from runtime_env.ring_task_window. + static uint64_t calculate_size(uint64_t max_tasks); // UT convenience: reserve wrapper + sm_base on `arena`, commit, and init using - // default CHIP_TASK_WINDOW_SIZE. Only valid when the arena is otherwise empty + // default CHIP_DEFAULT_GRAPH_TASKS. Only valid when the arena is otherwise empty // (the call performs the single commit). All memory is owned by the arena — // caller must not call destroy(). static SharedMemoryHandle *create_and_init_default(DeviceArena &arena); @@ -223,41 +204,39 @@ struct SharedMemoryHandle { // In-place init for caller-provided wrapper storage (e.g. a region carved // out of a DeviceArena). Sets is_owner = false, calls setup_pointers and - // init_header. Returns false when `sm_size` is too small for the requested - // `task_window_size`. - bool init(void *sm_base, uint64_t sm_size, uint64_t task_window_size); + // init_header. Returns false when `sm_size` is too small for `max_tasks`. + bool init(void *sm_base, uint64_t sm_size, uint64_t max_tasks); // Attach to an ALREADY-populated shared memory region: point the handle and - // every ring header's data pointers (descriptors / payloads / slot_states) - // at `sm_base`, but do NOT reset the flow-control counters / slot states. + // the task header's segment pointers (descriptors / payloads / slot_states) + // at `sm_base`, but do NOT reset the watermark / slot states. // Used by host_build_graph host-orch, where the host orchestrator populated // the SM and H2D'd it; the device must re-point at its own SM base without // wiping the contents (unlike init, which also resets the header). // // `live_slots` is the pitch the uploaded arrays were laid out with — the - // number of slots the host actually submitted, not the ring capacity. It must - // match what the host used or every segment past the descriptors resolves to - // the wrong address, so both sides derive it from the same submitted count. - // The capacity and mask in the header are unchanged, and `local_id & mask` - // yields `local_id`, which is below `live_slots` for every ring task. + // number of slots the host actually submitted, not the `max_tasks` the mirror + // was dimensioned for. It must match what the host used or every segment past + // the descriptors resolves to the wrong address, so both sides derive it from + // the same count. + // Every task id is below it, and indexes its slot directly. // // `image_bytes` is what the host shipped, pools included. The device cannot // recompute it — the pool extents are the bind's cursors, which only the host saw // — and it does not need to: a payload names its argument regions by delta, so no // pool base is resolved here. The value bounds the region and checks the int32 // delta reach. - bool attach_populated( - void *sm_base, uint64_t sm_size, uint64_t task_window_size, uint64_t live_slots, uint64_t image_bytes - ); + bool + attach_populated(void *sm_base, uint64_t sm_size, uint64_t max_tasks, uint64_t live_slots, uint64_t image_bytes); void destroy(); void print_layout(); bool validate(); private: - void init_header(uint64_t task_window_size); + void init_header(); // `pitch` is the slot count the arrays are dimensioned for. init passes the - // ring capacity (the mirror the orchestrator writes into); attach_populated + // mirror's reservation (what the orchestrator writes into); attach_populated // passes the submitted count (the compacted image that shipped). void setup_pointers(uint64_t pitch); }; @@ -267,7 +246,7 @@ struct SharedMemoryHandle { // ============================================================================= // // When the host pre-builds a runtime-arena image, it needs the device-side -// addresses of several SM sub-fields (ring flow-control counters, +// addresses of several SM sub-fields (the task header, // task_descriptors arrays, orch_error_code) so it can wire them into the // orchestrator / scheduler init_data path without dereferencing the SM — // the SM lives in device memory and cannot be touched from host. @@ -284,29 +263,22 @@ inline std::atomic *orch_error_code_addr(void *sm_dev_base) noexcept { ); } -inline SharedMemoryRingHeader *ring_header_addr(void *sm_dev_base) noexcept { - return reinterpret_cast( - static_cast(sm_dev_base) + offsetof(SharedMemoryHeader, ring) - ); -} - -inline std::atomic *ring_current_task_index_addr(void *sm_dev_base) noexcept { - return reinterpret_cast *>( - reinterpret_cast(ring_header_addr(sm_dev_base)) + offsetof(SharedMemoryRingHeader, fc) + - offsetof(ChipRingFlowControl, current_task_index) +inline SharedMemoryTaskHeader *task_header_addr(void *sm_dev_base) noexcept { + return reinterpret_cast( + static_cast(sm_dev_base) + offsetof(SharedMemoryHeader, tasks) ); } -// Byte offsets (from the SM base) of the ring's segments. The layout is: header, then +// Byte offsets (from the SM base) of the image's segments. The layout is: header, then // descriptors -> payloads -> slot_states -> completion_flags -> the three argument -// pools, every segment CHIP_ALIGN_UP-padded. RingImageExtents dimensions them: the +// pools, every segment CHIP_ALIGN_UP-padded. ImageExtents dimensions them: the // mirror for the worst case the API allows, the image for what this bind holds, which // is what makes the live prefixes contiguous and the upload one copy. // // The pools sit last because nothing on the device resolves a segment past // completion_flags: a payload names its argument regions by delta, so the four // slot-pitched offsets are all the attach path computes. -struct ChipRingSegmentOffsets { +struct SegmentOffsets { uint64_t descriptors; uint64_t payloads; uint64_t slot_states; @@ -318,22 +290,22 @@ struct ChipRingSegmentOffsets { }; // How many slots and how many pool elements a layout is dimensioned for. -struct RingImageExtents { +struct ImageExtents { uint64_t slots; uint64_t fanin_elems; uint64_t tensor_elems; uint64_t scalar_elems; }; -// The mirror the orchestrator writes into: the ring capacity, every pool sized so the -// worst case cannot overflow a bump — task_window tasks each at their full cap. That +// The mirror the orchestrator writes into: `max_tasks` slots, every pool sized so the +// worst case cannot overflow a bump — max_tasks tasks each at their full cap. That // bound is what lets prepare_task advance a cursor with no capacity check. -inline RingImageExtents mirror_extents(uint64_t task_window_size) noexcept { - return RingImageExtents{ - task_window_size, - task_window_size * CHIP_MAX_FANIN, - task_window_size * MAX_TENSOR_ARGS, - task_window_size * MAX_SCALAR_ARGS, +inline ImageExtents mirror_extents(uint64_t max_tasks) noexcept { + return ImageExtents{ + max_tasks, + max_tasks * CHIP_MAX_FANIN, + max_tasks * MAX_TENSOR_ARGS, + max_tasks * MAX_SCALAR_ARGS, }; } @@ -342,9 +314,9 @@ inline RingImageExtents mirror_extents(uint64_t task_window_size) noexcept { // `sm_base`) and the device-address helpers below (which add `sm_dev_base`). Adding // or reordering a segment is a one-line edit here; every consumer follows // automatically, so the layout walk can never silently disagree across call sites. -inline ChipRingSegmentOffsets ring_segment_offsets(const RingImageExtents &e) noexcept { +inline SegmentOffsets segment_offsets(const ImageExtents &e) noexcept { uint64_t off = CHIP_ALIGN_UP(sizeof(SharedMemoryHeader), CHIP_ALIGN_SIZE); - ChipRingSegmentOffsets o{}; + SegmentOffsets o{}; o.descriptors = off; off += CHIP_ALIGN_UP(e.slots * sizeof(TaskDescriptor), CHIP_ALIGN_SIZE); o.payloads = off; @@ -363,8 +335,8 @@ inline ChipRingSegmentOffsets ring_segment_offsets(const RingImageExtents &e) no return o; } -inline ChipRingSegmentOffsets ring_segment_offsets(uint64_t task_window_size) noexcept { - return ring_segment_offsets(mirror_extents(task_window_size)); +inline SegmentOffsets segment_offsets(uint64_t max_tasks) noexcept { + return segment_offsets(mirror_extents(max_tasks)); } // Every per-task region starts on a cache line, which TaskPayload::init's @@ -396,8 +368,8 @@ struct BindUsage { uint64_t scalar_elems; }; -inline RingImageExtents image_extents(const BindUsage &used) noexcept { - return RingImageExtents{ +inline ImageExtents image_extents(const BindUsage &used) noexcept { + return ImageExtents{ live_slot_pitch(used.submitted_tasks), used.fanin_elems, used.tensor_elems, @@ -432,16 +404,16 @@ inline uint64_t rebased_heap_addr(uint64_t addr, const HeapRebase &rebase) noexc return rebase.real_base + offset; } -// Restack the live prefix of every ring segment from the ring-pitched mirror the +// Restack the live prefix of every segment from the mirror the // orchestrator wrote into an image dimensioned for what this bind holds, where the // prefixes are contiguous and can travel as one copy. // // `out_base` must be CHIP_ALIGN_SIZE-aligned and hold -// `ring_segment_offsets(image_extents(used)).end` bytes. Returns that byte count. +// `segment_offsets(image_extents(used)).end` bytes. Returns that byte count. // // Three things the restack has to fix up, all because the image is not the mirror: // -// - the ring header's data pointers name the mirror's arrays, so they leave as +// - the task header's segment pointers name the mirror's arrays, so they leave as // null rather than carrying host addresses into device memory (the device // resolves them in attach_populated); // - a slot state names its payload and descriptor, and a payload names its three @@ -463,27 +435,27 @@ inline uint64_t rebased_heap_addr(uint64_t addr, const HeapRebase &rebase) noexc // falls inside the window. Orchestration must therefore pass a runtime-created // buffer as the tensor it got back, never as a scalar carrying its address. inline uint64_t compact_live_image( - const char *mirror_base, uint64_t task_window_size, const BindUsage &used, const HeapRebase &rebase, char *out_base + const char *mirror_base, uint64_t max_tasks, const BindUsage &used, const HeapRebase &rebase, char *out_base ) noexcept { // The mirror is dimensioned for the worst case, so a live count or a cursor past // it reads beyond the segment it is copying from and ships a corrupt image. // attach_populated tests the slot bound again on the device side. - const RingImageExtents mirror = mirror_extents(task_window_size); + const ImageExtents mirror = mirror_extents(max_tasks); always_assert(used.submitted_tasks <= mirror.slots); always_assert(used.fanin_elems <= mirror.fanin_elems); always_assert(used.tensor_elems <= mirror.tensor_elems); always_assert(used.scalar_elems <= mirror.scalar_elems); - const ChipRingSegmentOffsets from = ring_segment_offsets(mirror); - const ChipRingSegmentOffsets to = ring_segment_offsets(image_extents(used)); + const SegmentOffsets from = segment_offsets(mirror); + const SegmentOffsets to = segment_offsets(image_extents(used)); // The header and the descriptors offset are pitch-independent, so the header // lands where it already was. std::memcpy(out_base, mirror_base, to.descriptors); - auto &out_ring = reinterpret_cast(out_base)->ring; - out_ring.task_descriptors = nullptr; - out_ring.task_payloads = nullptr; - out_ring.slot_states = nullptr; - out_ring.completion_flags = nullptr; + auto &out_tasks = reinterpret_cast(out_base)->tasks; + out_tasks.task_descriptors = nullptr; + out_tasks.task_payloads = nullptr; + out_tasks.slot_states = nullptr; + out_tasks.completion_flags = nullptr; const uint64_t nt = used.submitted_tasks; std::memcpy(out_base + to.descriptors, mirror_base + from.descriptors, nt * sizeof(TaskDescriptor)); diff --git a/src/a5/runtime/host_build_graph/runtime/ring_buffer.h b/src/a5/runtime/host_build_graph/runtime/task_allocator.h similarity index 68% rename from src/a5/runtime/host_build_graph/runtime/ring_buffer.h rename to src/a5/runtime/host_build_graph/runtime/task_allocator.h index 6a2fe3f0ac..8a6c332f9d 100644 --- a/src/a5/runtime/host_build_graph/runtime/ring_buffer.h +++ b/src/a5/runtime/host_build_graph/runtime/task_allocator.h @@ -9,11 +9,9 @@ * ----------------------------------------------------------------------------------------------------------- */ /** - * PTO Runtime2 - Ring Buffer Data Structures - * * TaskAllocator - Unified task slot + output buffer allocation - * - Combines task ring (slot allocation) and heap ring (output buffer allocation) - * - O(1) forward bump allocation for both task slots and heap buffers + * - Combines task-table slot allocation and heap output-buffer allocation + * - O(1) forward bump allocation for both * - Neither resource is reclaimed during a run, so exhaustion of either is a * capacity error reported on the spot, never back-pressure to wait on * @@ -37,41 +35,41 @@ /** * Unified task slot + heap buffer allocator. * - * Since task and heap are always allocated together and the orchestrator is - * single-threaded, both pointers (task index, heap top) are tracked locally - * and published to shared memory via plain store — no fetch_add or CAS needed. + * Task ids and heap bytes are handed out by two local bump counters. The + * orchestrator is single-threaded and nothing outside it observes either + * counter, so neither needs an atomic or a published copy: the run's task total + * is read back through active_count() once orchestration ends. * * The alloc() method checks both resources BEFORE committing to either, * eliminating the need for rollback on partial failure. * * host_build_graph is whole-graph-resident: the device runs only after the host * has built the entire graph, so no task slot or heap byte is ever reclaimed - * while allocation is in progress. Both rings are therefore forward-only, and a - * request that does not fit can never become satisfiable by waiting — alloc() + * while allocation is in progress. Both counters are therefore forward-only, and + * a request that does not fit can never become satisfiable by waiting — alloc() * reports the exhausted resource and fails on the spot. + * + * A task id is also its slot index: ids are capped at `capacity` and never + * recycled, so the task table is a flat array indexed by id, with no wrap. */ class TaskAllocator { public: /** - * Initialize the allocator with task ring and heap ring resources. + * Initialize the allocator with its task capacity and heap resources. * * All pointer arguments are device addresses (live in SM / GM heap); this * function only stores them, no dereferences, so it is safe to invoke * from host code that constructs a prebuilt arena image. * - * The ring starts at task id 0, matching the SM flow-control counter that - * current_index_ptr points at (ChipRingFlowControl::init() runs on the AICPU - * during SM reset), so local_task_id_ stays in sync without reading the SM. - * Because ids are never reclaimed, alloc() caps them at window_size — they - * cannot run away toward INT32_MAX. + * `capacity` is the number of task slots the caller's task table holds — what + * the bind resolved from runtime_env.ring_task_window, defaulting to + * CHIP_DEFAULT_GRAPH_TASKS. It need not be a power of two: a task id indexes + * its slot directly, so nothing masks with it. Because ids are never + * reclaimed, alloc() caps them at `capacity` — they cannot run away toward + * INT32_MAX. */ - void init( - int32_t window_size, std::atomic *current_index_ptr, void *heap_base, uint64_t heap_size, - std::atomic *error_code_ptr - ) { - window_size_ = window_size; - window_mask_ = window_size - 1; - current_index_ptr_ = current_index_ptr; + void init(int32_t capacity, void *heap_base, uint64_t heap_size, std::atomic *error_code_ptr) { + capacity_ = capacity; heap_base_ = heap_base; heap_size_ = heap_size; error_code_ptr_ = error_code_ptr; @@ -90,9 +88,8 @@ class TaskAllocator { /** * Allocate a task slot and its associated output buffer in one call. * - * Both task index and heap top are maintained as local counters and - * published to shared memory only on success. Since the orchestrator is - * single-threaded, no CAS or fetch_add is needed — just check-then-commit. + * Both the task id and the heap top are local counters, so this is a plain + * check-then-commit with no atomic and no rollback. * * A fatal latched elsewhere short-circuits the allocation: the caller maps * the failed result to orch_mark_fatal without overwriting the first code. @@ -105,21 +102,21 @@ class TaskAllocator { output_size > 0 ? CHIP_ALIGN_UP(static_cast(output_size), CHIP_ALIGN_SIZE) : 0; if (error_code_ptr_ != nullptr && error_code_ptr_->load(std::memory_order_acquire) != SIMPLER_ERROR_NONE) { - return {-1, -1, nullptr, nullptr}; + return {-1, nullptr, nullptr}; } // Check both resources; commit only if both are available. - if (local_task_id_ >= window_size_) { + if (local_task_id_ >= capacity_) { report_capacity_exhausted(/*heap_blocked=*/false, aligned_size); - return {-1, -1, nullptr, nullptr}; + return {-1, nullptr, nullptr}; } void *heap_ptr = try_bump_heap(aligned_size); if (heap_ptr == nullptr) { report_capacity_exhausted(/*heap_blocked=*/true, aligned_size); - return {-1, -1, nullptr, nullptr}; + return {-1, nullptr, nullptr}; } - int32_t task_id = commit_task(); - return {task_id, task_id & window_mask_, heap_ptr, static_cast(heap_ptr) + aligned_size}; + int32_t task_id = local_task_id_++; + return {task_id, heap_ptr, static_cast(heap_ptr) + aligned_size}; } bool reserve_deferred_heap(int32_t output_size, void **packed_base, void **packed_end) { @@ -140,13 +137,13 @@ class TaskAllocator { // State queries // ========================================================================= - // Nothing retires during a run, so every task allocated so far is still - // live and the ring's head doubles as its occupancy. + // Nothing retires during a run, so every task allocated so far is still live + // and the next id doubles as the occupancy. Once orchestration has ended this + // is therefore also the run's task total, and the only place it is read from: + // no copy of the count is published anywhere else. int32_t active_count() const { return local_task_id_; } - int32_t task_head() const { return local_task_id_; } - - int32_t window_size() const { return window_size_; } + int32_t capacity() const { return capacity_; } uint64_t heap_available() const { return heap_size_ - heap_top_; } @@ -155,10 +152,8 @@ class TaskAllocator { uint64_t heap_used_bytes() const { return heap_top_; } private: - // --- Task Ring --- - int32_t window_size_ = 0; - int32_t window_mask_ = 0; - std::atomic *current_index_ptr_ = nullptr; + // --- Task table --- + int32_t capacity_ = 0; // --- Heap --- void *heap_base_ = nullptr; @@ -175,16 +170,6 @@ class TaskAllocator { // Internal helpers // ========================================================================= - /** - * Commit a task slot: bump local counter and publish to shared memory. - * Must only be called after space check has passed. - */ - int32_t commit_task() { - int32_t task_id = local_task_id_++; - current_index_ptr_->store(local_task_id_, std::memory_order_release); - return task_id; - } - /** * Bump the heap pointer for the given allocation size. * Returns the allocated pointer, or nullptr if insufficient space. @@ -212,34 +197,36 @@ class TaskAllocator { * Nothing is reclaimed during a run, so this is a sizing verdict with no wait * that could still succeed. * - * The task window is a configured capacity, so its branch is the one a real - * bind reaches. The heap is not configured: a bind hands this allocator the - * whole HEAP_VIRTUAL_CAPACITY span and commits the device region afterwards, so - * a graph that does not fit the device fails at that commit and not here. The - * heap branch stays because the allocator is also constructed directly, against - * a small heap, by the unit tests that cover this report. + * The task capacity is a configured size, so its branch is the one a real bind + * reaches. The heap is not configured: a bind hands + * this allocator the whole HEAP_VIRTUAL_CAPACITY span and commits the device + * region afterwards, so a graph that does not fit the device fails at that + * commit and not here. The heap branch stays because the allocator is also + * constructed directly, against a small heap, by the unit tests that cover + * this report. */ void report_capacity_exhausted(bool heap_blocked, uint64_t requested_bytes) { LOG_ERROR("========================================"); if (heap_blocked) { LOG_ERROR("FATAL: Graph Heap Exhausted!"); } else { - LOG_ERROR("FATAL: Task Window Exhausted!"); + LOG_ERROR("FATAL: Graph Too Large!"); } LOG_ERROR("========================================"); LOG_ERROR("The whole graph must fit at once; nothing is reclaimed mid-run."); - LOG_ERROR(" Task window: used=%d/%d", local_task_id_, window_size_); + LOG_ERROR(" Tasks: used=%d/%d", local_task_id_, capacity_); LOG_ERROR( - " Graph heap: used=%" PRIu64 "/%" PRIu64 ", available=%" PRIu64, heap_top_, heap_size_, heap_available() + " Graph heap: used=%" PRIu64 "/%" PRIu64 ", available=%" PRIu64, heap_top_, heap_size_, heap_available() ); - LOG_ERROR(" Requested: %" PRIu64 " bytes + 1 task slot", requested_bytes); + LOG_ERROR(" Requested: %" PRIu64 " bytes + 1 task slot", requested_bytes); LOG_ERROR("Solution:"); if (heap_blocked) { LOG_ERROR(" Shrink the graph's intermediate tensors; this heap has no configuration knob"); } else { LOG_ERROR( - " Increase task window (current: %d); CallConfig.runtime_env.ring_task_window= (e.g. %d)", - window_size_, window_size_ * 2 + " Raise the task capacity (current: %d) via CallConfig.runtime_env.ring_task_window, or shrink " + "the graph", + capacity_ ); } LOG_ERROR("========================================"); diff --git a/src/a5/runtime/host_build_graph/runtime/tensormap.h b/src/a5/runtime/host_build_graph/runtime/tensormap.h index 187194131e..0ad8d89a84 100644 --- a/src/a5/runtime/host_build_graph/runtime/tensormap.h +++ b/src/a5/runtime/host_build_graph/runtime/tensormap.h @@ -361,12 +361,10 @@ struct ChipTensorMap { int32_t next_entry_idx{0}; // id when next entry insert int32_t free_num{0}; // free entry number in entry pool - // Per-task entry tracking for O(1) unlinking of covered producers. - // Indexed by [local_id & (task_window_size - 1)] + // Per-task entry tracking for O(1) unlinking of covered producers. A task id + // is its own slot index, so this is indexed by local_id directly. std::unique_ptr task_entry_heads; - int32_t task_window_size{0}; // Task window size (for slot masking) - - uint32_t get_task_local_id_slot(uint32_t task_local_id) const { return task_local_id & (task_window_size - 1); } + int32_t max_tasks{0}; // Slots task_entry_heads is dimensioned for // Pool occupancy, read by the tensormap-exhaustion diagnostic and by the // Graph recording's capacity precheck. @@ -429,32 +427,32 @@ struct ChipTensorMap { /** * Allocate the four arrays and leave the map empty. num_buckets must be a - * power of two; task_window_size must be a power of two, since a producer's - * task chain is selected by `local_id & (task_window_size - 1)`. + * power of two; `new_max_tasks` is the number of task chains to reserve, one + * per task id the caller can place. * * Returns false when an allocation fails, so a caller that still has an * alternative path can take it rather than proceed without a hazard map. * - * Clearing is O(num_buckets + task_window_size), not O(pool_size): the entry + * Clearing is O(num_buckets + max_tasks), not O(pool_size): the entry * pool is left uninitialized and new_entry() puts each slot into the clean * unlinked state on first use, and free_entry_list is a stack meaningful only * below free_num. */ - bool init(int32_t new_num_buckets, int32_t new_pool_size, int32_t new_task_window_size); + bool init(int32_t new_num_buckets, int32_t new_pool_size, int32_t new_max_tasks); /** * Empty an already-initialized map without touching its allocations. * * Same post-state as init(): every bucket and task-chain head null, both pool * cursors at zero, the entry pool left to init-on-write. Costs - * O(num_buckets + task_window_size) stores against pages that are already + * O(num_buckets + max_tasks) stores against pages that are already * resident, where init() pays the allocation and the first touch of each. A * caller that records one body after another on the same map uses this. * - * Keeps the bucket count and task window init() reserved. Taking a new window - * here would have to be checked against the reserved length rather than the - * current one, and nothing tracks the reserved length once a smaller window has - * been set -- so the sizes stay init()'s and this takes no argument. + * Keeps the bucket count and task-chain count init() reserved. Taking a new + * count here would have to be checked against the reserved length rather than + * the current one, and nothing tracks the reserved length once a smaller count + * has been set -- so the sizes stay init()'s and this takes no argument. */ void reset(); @@ -462,7 +460,7 @@ struct ChipTensorMap { * Same as init() with default sizes (CHIP_TENSORMAP_NUM_BUCKETS, * CHIP_TENSORMAP_POOL_SIZE). */ - bool init_default(int32_t new_task_window_size); + bool init_default(int32_t new_max_tasks); /** * Lookup producer for a tensor region @@ -567,7 +565,7 @@ struct ChipTensorMap { #endif uint32_t bucket_index = hash(addr); auto local_id = producer_task_id.local(); - int32_t task_slot = local_id & (task_window_size - 1); + const int32_t task_slot = local_id; entry->producer_task_id = producer_task_id; @@ -605,7 +603,7 @@ struct ChipTensorMap { if (entry.prev_in_task == nullptr) { // Entry is the head of its task chain, update task_entry_heads int32_t local_id = static_cast(entry.producer_task_id.local()); - int32_t task_slot = local_id & (task_window_size - 1); + const int32_t task_slot = local_id; task_entry_heads[task_slot] = entry.next_in_task; } else { entry.prev_in_task->next_in_task = entry.next_in_task; diff --git a/src/a5/runtime/host_build_graph/runtime/types.h b/src/a5/runtime/host_build_graph/runtime/types.h index 89487dca0f..f110967f21 100644 --- a/src/a5/runtime/host_build_graph/runtime/types.h +++ b/src/a5/runtime/host_build_graph/runtime/types.h @@ -386,8 +386,10 @@ struct Arg : TaskArgsTpl { * * count == 0 is a valid "set empty" — it clears any previously stored deps * and returns. This lets callers that build the dep set conditionally pass - * the result through unguarded, including in the no-dep branch: - * TaskId deps[3]; + * the result through unguarded, including in the no-dep branch. Fill with + * TaskId::invalid() so an entry the branches never write is rejected by the + * orchestrator rather than read as a task id: + * TaskId deps[3] = {TaskId::invalid(), TaskId::invalid(), TaskId::invalid()}; * uint32_t n = 0; * if (have_prev) deps[n++] = prev; * if (is_last) deps[n++] = alloc; diff --git a/src/common/hierarchical/ring.h b/src/common/hierarchical/ring.h index 8105fe854e..2c02e0d671 100644 --- a/src/common/hierarchical/ring.h +++ b/src/common/hierarchical/ring.h @@ -15,10 +15,11 @@ * A single structure owns three correlated per-task resources: * * 1. A monotonic task id (`next_task_id_`), allocated by the Orchestrator. - * Unlike L2's `TaskAllocator` the id is NOT masked into a fixed-size - * window — slot state lives in parent-process heap (never crossed into - * child workers), so a ring index buys us nothing at L3 (see the plan's - * L2 Consistency Audit, allowed exception #6). + * The id is NOT masked into a fixed-size window — slot state lives in + * parent-process heap (never crossed into child workers), so a ring index + * buys us nothing at L3 (see the plan's L2 Consistency Audit, allowed + * exception #6). Of the two L2 `TaskAllocator`s only + * tensormap_and_ringbuffer's masks; host_build_graph's never wraps either. * 2. `MAX_RING_DEPTH` independent shared-memory heap slabs (Strict-1, * matches L2's `CHIP_MAX_RING_DEPTH = 4`). Each slab has its own * `mmap(MAP_SHARED)` region, bump cursor, FIFO reclamation pointer, diff --git a/src/common/runtime_status/error_names.h b/src/common/runtime_status/error_names.h index fb7b0e51b5..9810da9687 100644 --- a/src/common/runtime_status/error_names.h +++ b/src/common/runtime_status/error_names.h @@ -86,8 +86,8 @@ static inline const char *error_desc(int32_t code) { case SIMPLER_ERROR_HEAP_RING_DEADLOCK: return "the task allocator could not reserve the heap bytes required for another task"; case SIMPLER_ERROR_FLOW_CONTROL_DEADLOCK: - return "the task allocator could not admit another task because the configured task window " - "had no available slot"; + return "the task allocator could not admit another task because the task table had no " + "available slot"; case SIMPLER_ERROR_FANIN_CAPACITY_EXCEEDED: return "a task's producer dependencies exceeded the runtime's available fanin " "representation capacity"; @@ -137,8 +137,9 @@ static inline const char *error_hint(int32_t code) { return "raise runtime_env.ring_task_window or split the scope so slots are " "reclaimed sooner; enable CallConfig.enable_scope_stats to see which scope peaked"; case SIMPLER_ERROR_HEAP_RING_DEADLOCK: - return "raise runtime_env.ring_heap or shrink per-task args / intermediate tensors; the " - "'Ring buffer sizes:' line above reports the configured capacities"; + return "shrink per-task args / intermediate tensors; in a reclaiming runtime raise " + "runtime_env.ring_heap -- host_build_graph's graph heap has no knob, so a graph too " + "large for the device fails host-side at the region commit instead"; case SIMPLER_ERROR_FLOW_CONTROL_DEADLOCK: return "raise runtime_env.ring_task_window or shrink the graph; in a reclaiming " "runtime also check for nested submission or a stalled consumer"; diff --git a/tests/st/a2a3/host_build_graph/paged_attention/test_paged_attention.py b/tests/st/a2a3/host_build_graph/paged_attention/test_paged_attention.py index 3a527988a5..31c829565e 100644 --- a/tests/st/a2a3/host_build_graph/paged_attention/test_paged_attention.py +++ b/tests/st/a2a3/host_build_graph/paged_attention/test_paged_attention.py @@ -70,11 +70,11 @@ class TestPagedAttentionHostBuildGraph(SceneTestCase): { # Marked manual for host_build_graph: this batch=256 case submits # ~64K tasks, and host-orchestration populates the whole task graph - # before the device schedules — so the ring cannot reclaim - # mid-orchestration and must hold the entire graph at once. That - # exceeds the default ring window. Run it explicitly with a large - # runtime_env.ring_task_window if needed; the GM heap needs no knob, since - # it is committed to the size orchestration measured. + # before the device schedules — nothing is reclaimed mid-orchestration, + # so the table must hold the entire graph at once. That exceeds the + # default. Run it explicitly with a larger + # runtime_env.ring_task_window if needed; the GM heap needs no sizing, + # since it is committed to the size orchestration measured. "name": "Case1", "platforms": ["a2a3"], "manual": True, diff --git a/tests/st/a5/host_build_graph/paged_attention/test_paged_attention.py b/tests/st/a5/host_build_graph/paged_attention/test_paged_attention.py index 73c86bfc8f..6bb83b3efc 100644 --- a/tests/st/a5/host_build_graph/paged_attention/test_paged_attention.py +++ b/tests/st/a5/host_build_graph/paged_attention/test_paged_attention.py @@ -70,11 +70,11 @@ class TestPagedAttentionHostBuildGraphA5(SceneTestCase): { # Marked manual for host_build_graph: this batch=256 case submits # ~64K tasks, and host-orchestration populates the whole task graph - # before the device schedules — so the ring cannot reclaim - # mid-orchestration and must hold the entire graph at once. That - # exceeds the default ring window (16384 slots). Run it explicitly - # with a large runtime_env.ring_task_window if needed; the GM heap needs no - # knob, since it is committed to the size orchestration measured. + # before the device schedules — nothing is reclaimed mid-orchestration, + # so the table must hold the entire graph at once. That exceeds the + # default (16384 slots). Run it explicitly with a larger + # runtime_env.ring_task_window if needed; the GM heap needs no sizing, + # since it is committed to the size orchestration measured. "name": "Case1", "platforms": ["a5"], "manual": True, diff --git a/tests/ut/cpp/CMakeLists.txt b/tests/ut/cpp/CMakeLists.txt index bf7d0fc0e9..5d708c5326 100644 --- a/tests/ut/cpp/CMakeLists.txt +++ b/tests/ut/cpp/CMakeLists.txt @@ -885,7 +885,6 @@ target_sources(test_a5_hbg_scheduler_drain PRIVATE ) target_sources(test_hbg_ready_queue_seed PRIVATE ${HBG_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${HBG_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${HBG_RUNTIME_DIR}/shared/shared_memory.cpp ${HBG_RUNTIME_DIR}/shared/tensormap.cpp ${HBG_RUNTIME_DIR}/shared/runtime_init.cpp @@ -897,7 +896,6 @@ target_sources(test_hbg_ready_queue_seed PRIVATE add_a2a3_hbg_runtime_test(test_hbg_submit_poison a2a3/test_hbg_submit_poison.cpp) target_sources(test_hbg_submit_poison PRIVATE ${HBG_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${HBG_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${HBG_RUNTIME_DIR}/shared/shared_memory.cpp ${HBG_RUNTIME_DIR}/shared/tensormap.cpp ${HBG_RUNTIME_DIR}/shared/runtime_init.cpp @@ -907,7 +905,6 @@ target_sources(test_hbg_submit_poison PRIVATE add_a2a3_hbg_runtime_test(test_hbg_graph_submit_failure common/test_hbg_graph_submit_failure.cpp) target_sources(test_hbg_graph_submit_failure PRIVATE ${HBG_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${HBG_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${HBG_RUNTIME_DIR}/shared/shared_memory.cpp ${HBG_RUNTIME_DIR}/shared/tensormap.cpp ${HBG_RUNTIME_DIR}/shared/runtime_init.cpp @@ -918,7 +915,6 @@ add_a2a3_hbg_runtime_test(test_hbg_graph_async_submit common/test_hbg_graph_asyn add_a2a3_hbg_runtime_test(test_hbg_graph_definition_arena common/test_hbg_graph_definition_arena.cpp) target_sources(test_hbg_graph_definition_arena PRIVATE ${HBG_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${HBG_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${HBG_RUNTIME_DIR}/shared/shared_memory.cpp ${HBG_RUNTIME_DIR}/shared/tensormap.cpp ${HBG_RUNTIME_DIR}/shared/runtime_init.cpp @@ -930,7 +926,6 @@ target_sources(test_hbg_graph_definition_arena PRIVATE add_a2a3_hbg_runtime_test(test_hbg_slot_claim common/test_hbg_slot_claim.cpp) target_sources(test_hbg_slot_claim PRIVATE ${HBG_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${HBG_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${HBG_RUNTIME_DIR}/shared/shared_memory.cpp ${HBG_RUNTIME_DIR}/shared/tensormap.cpp ${HBG_RUNTIME_DIR}/shared/runtime_init.cpp @@ -940,7 +935,6 @@ target_sources(test_hbg_slot_claim PRIVATE add_a5_hbg_runtime_test(test_a5_hbg_graph_submit_failure common/test_hbg_graph_submit_failure.cpp) target_sources(test_a5_hbg_graph_submit_failure PRIVATE ${A5_HBG_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${A5_HBG_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${A5_HBG_RUNTIME_DIR}/shared/shared_memory.cpp ${A5_HBG_RUNTIME_DIR}/shared/tensormap.cpp ${A5_HBG_RUNTIME_DIR}/shared/runtime_init.cpp @@ -951,7 +945,6 @@ add_a5_hbg_runtime_test(test_a5_hbg_graph_async_submit common/test_hbg_graph_asy add_a5_hbg_runtime_test(test_a5_hbg_graph_definition_arena common/test_hbg_graph_definition_arena.cpp) target_sources(test_a5_hbg_graph_definition_arena PRIVATE ${A5_HBG_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${A5_HBG_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${A5_HBG_RUNTIME_DIR}/shared/shared_memory.cpp ${A5_HBG_RUNTIME_DIR}/shared/tensormap.cpp ${A5_HBG_RUNTIME_DIR}/shared/runtime_init.cpp @@ -961,7 +954,6 @@ target_sources(test_a5_hbg_graph_definition_arena PRIVATE add_a5_hbg_runtime_test(test_a5_hbg_slot_claim common/test_hbg_slot_claim.cpp) target_sources(test_a5_hbg_slot_claim PRIVATE ${A5_HBG_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${A5_HBG_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${A5_HBG_RUNTIME_DIR}/shared/shared_memory.cpp ${A5_HBG_RUNTIME_DIR}/shared/tensormap.cpp ${A5_HBG_RUNTIME_DIR}/shared/runtime_init.cpp @@ -975,7 +967,6 @@ add_a5_hbg_runtime_test(test_a5_hbg_self_relative_ptr common/test_hbg_self_relat add_a5_hbg_runtime_test(test_a5_hbg_sm_compaction common/test_hbg_sm_compaction.cpp) target_sources(test_a5_hbg_ready_queue_seed PRIVATE ${A5_HBG_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${A5_HBG_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${A5_HBG_RUNTIME_DIR}/shared/shared_memory.cpp ${A5_HBG_RUNTIME_DIR}/shared/tensormap.cpp ${A5_HBG_RUNTIME_DIR}/shared/runtime_init.cpp @@ -989,7 +980,6 @@ set(HBG_A5_RUNTIME_DIR ${CMAKE_SOURCE_DIR}/../../../src/a5/runtime/host_build_gr add_a5_hbg_runtime_test(test_a5_hbg_submit_poison a5/test_hbg_submit_poison.cpp) target_sources(test_a5_hbg_submit_poison PRIVATE ${HBG_A5_RUNTIME_DIR}/orchestrator_core/orchestrator.cpp - ${HBG_A5_RUNTIME_DIR}/orchestrator_core/ring_buffer.cpp ${HBG_A5_RUNTIME_DIR}/shared/shared_memory.cpp ${HBG_A5_RUNTIME_DIR}/shared/tensormap.cpp ${HBG_A5_RUNTIME_DIR}/shared/runtime_init.cpp diff --git a/tests/ut/cpp/a2a3/test_hbg_submit_poison.cpp b/tests/ut/cpp/a2a3/test_hbg_submit_poison.cpp index 165d600af8..5d47d8d683 100644 --- a/tests/ut/cpp/a2a3/test_hbg_submit_poison.cpp +++ b/tests/ut/cpp/a2a3/test_hbg_submit_poison.cpp @@ -11,14 +11,14 @@ /** * Poison test for the "every device-read SM field is written at submit" contract. * - * host_build_graph no longer zero-fills the shared-memory task window (init-on-write): - * init_header_per_ring writes only the header, and each slot's device-read fields are + * host_build_graph does not zero-fill the shared-memory task table (init-on-write): + * init_header writes only the header, and each slot's device-read fields are * written per task at submit (prepare_task + submit_task_common + TaskPayload::init). - * Nothing else clears the window, so a device-read field a submit forgets to write would + * Nothing else clears the table, so a device-read field a submit forgets to write would * read as 0 only by allocator accident — passing every zero-backed test and failing * non-deterministically on device. * - * This test fills the whole per-slot window with a 0xAA poison byte before submitting a + * This test fills the whole task table with a 0xAA poison byte before submitting a * representative mix, then asserts that for every claimed slot [0, total_tasks) the * device-read fields carry real values, not poison. Add a device-read field and forget * its submit-path write, and this fails in-tree. @@ -65,7 +65,7 @@ class HbgSubmitPoisonTest : public ::testing::Test { // Same order the AICPU boots in: the slot arrays are not part of the // uploaded image, so nothing can push until they carry their ramp. sched.seed_queue_slots(); - ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), 4096, CHIP_TASK_WINDOW_SIZE, &sched)); + ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), 4096, CHIP_DEFAULT_GRAPH_TASKS, &sched)); } void TearDown() override { @@ -74,21 +74,21 @@ class HbgSubmitPoisonTest : public ::testing::Test { sm_arena.release(); } - // Fill the per-slot window (descriptors / payloads / slot_states / completion_flags) - // with poison. init_header_per_ring wrote only the header, so this is the state the - // window is in before any submit writes it — modelling the never-zeroed device SM. - void poison_window() { - auto &ring = sm_handle->header->ring; // host_build_graph is single-ring - const size_t n = static_cast(ring.task_window_mask) + 1; - std::memset(ring.task_descriptors, POISON, n * sizeof(TaskDescriptor)); - std::memset(ring.task_payloads, POISON, n * sizeof(TaskPayload)); - std::memset(ring.slot_states, POISON, n * sizeof(ChipTaskSlotState)); - std::memset(ring.completion_flags, POISON, n * sizeof(std::atomic)); + // Fill the task table (descriptors / payloads / slot_states / completion_flags) + // with poison. init_header wrote only the header, so this is the state the table + // is in before any submit writes it — modelling the never-zeroed device SM. + void poison_task_table() { + auto &tasks = sm_handle->header->tasks; + const size_t n = static_cast(CHIP_DEFAULT_GRAPH_TASKS); + std::memset(tasks.task_descriptors, POISON, n * sizeof(TaskDescriptor)); + std::memset(tasks.task_payloads, POISON, n * sizeof(TaskPayload)); + std::memset(tasks.slot_states, POISON, n * sizeof(ChipTaskSlotState)); + std::memset(tasks.completion_flags, POISON, n * sizeof(std::atomic)); } }; TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { - poison_window(); + poison_task_table(); orch.begin_scope(); // 1. Zero-fanin root: a real mixed (AIV0) task with an output tensor and a scalar. @@ -127,17 +127,16 @@ TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { orch.end_scope(); - auto &ring = sm_handle->header->ring; - const int32_t total = ring.fc.current_task_index.load(std::memory_order_acquire); + auto &tasks = sm_handle->header->tasks; + const int32_t total = orch.task_allocator.active_count(); ASSERT_GE(total, 4); // Every claimed slot's device-read fields must carry real values, not poison. for (int32_t local = 0; local < total; local++) { SCOPED_TRACE(testing::Message() << "slot local_id=" << local); - const int32_t slot = ring.get_slot_by_task_id(local); - const TaskDescriptor &desc = ring.task_descriptors[slot]; - const TaskPayload &pl = ring.task_payloads[slot]; - const ChipTaskSlotState &st = ring.slot_states[slot]; + const TaskDescriptor &desc = tasks.task_descriptors[local]; + const TaskPayload &pl = tasks.task_payloads[local]; + const ChipTaskSlotState &st = tasks.slot_states[local]; // Descriptor: the task id is written to this exact local id. EXPECT_EQ(desc.task_id.local(), static_cast(local)); @@ -148,7 +147,7 @@ TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { EXPECT_TRUE(state == CHIP_TASK_PENDING || state == CHIP_TASK_COMPLETED); // Completion flag is written to a real 0/1 (pending vs pre-completed), not a // poison byte (0xAA). - const uint8_t cflag = ring.completion_flags[slot].load(std::memory_order_relaxed); + const uint8_t cflag = tasks.completion_flags[local].load(std::memory_order_relaxed); EXPECT_LE(cflag, uint8_t{1}); // Payload counts are real, not the poison bit pattern. EXPECT_GE(pl.fanin_count, 0); @@ -164,8 +163,8 @@ TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { } // Field-specific coverage on the real task: tensors, scalar, packed output buffer. - const TaskDescriptor &root_desc = ring.task_descriptors[ring.get_slot_by_task_id(root.task_id().local())]; - const TaskPayload &root_pl = ring.task_payloads[ring.get_slot_by_task_id(root.task_id().local())]; + const TaskDescriptor &root_desc = tasks.task_descriptors[root.task_id().local()]; + const TaskPayload &root_pl = tasks.task_payloads[root.task_id().local()]; EXPECT_EQ(root_pl.tensor_count, 1); EXPECT_EQ(root_pl.scalar_count, 1); EXPECT_EQ(root_pl.dump_metadata.dump_arg_mask, (uint64_t{1} << 0) | (uint64_t{1} << 1)); @@ -176,7 +175,7 @@ TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { EXPECT_EQ(root_desc.kernel_id[static_cast(SubtaskSlot::AIV0)], 0); // The consumer's fanin is written: two duplicate deps dedupe to one. - const TaskPayload &cons_pl = ring.task_payloads[ring.get_slot_by_task_id(consumer.task_id().local())]; + const TaskPayload &cons_pl = tasks.task_payloads[consumer.task_id().local()]; EXPECT_EQ(cons_pl.fanin_count, 1); EXPECT_EQ(cons_pl.fanin_data()[0], static_cast(root.task_id().local())); } diff --git a/tests/ut/cpp/a2a3/test_hbg_task_allocator.cpp b/tests/ut/cpp/a2a3/test_hbg_task_allocator.cpp index f7e2fa2daa..9ee460662e 100644 --- a/tests/ut/cpp/a2a3/test_hbg_task_allocator.cpp +++ b/tests/ut/cpp/a2a3/test_hbg_task_allocator.cpp @@ -25,8 +25,8 @@ * cannot arrive, and there is no wall-clock deadlock backstop. * - Zero-size allocation is a no-op returning the current top. Two consecutive * zero-size allocs return the SAME pointer. - * - Task ids start at 0 and are capped by the window, so they cannot approach - * INT32_MAX. + * - Task ids start at 0 and are capped by the capacity, so they cannot approach + * INT32_MAX, and each id is its own task-table slot. */ #include @@ -35,24 +35,22 @@ #include #include -#include "ring_buffer.h" +#include "task_allocator.h" #include "task_interface/assert_compat.h" class HbgTaskAllocatorTest : public ::testing::Test { protected: - static constexpr int32_t WINDOW_SIZE = 16; + static constexpr int32_t MAX_TASKS = 16; static constexpr uint64_t HEAP_SIZE = 4096; alignas(64) uint8_t heap_buf[HEAP_SIZE]{}; - std::atomic current_index{0}; std::atomic error_code{SIMPLER_ERROR_NONE}; TaskAllocator allocator{}; void SetUp() override { std::memset(heap_buf, 0, sizeof(heap_buf)); - current_index.store(0); error_code.store(SIMPLER_ERROR_NONE); - allocator.init(WINDOW_SIZE, ¤t_index, heap_buf, HEAP_SIZE, &error_code); + allocator.init(MAX_TASKS, heap_buf, HEAP_SIZE, &error_code); } }; @@ -61,7 +59,7 @@ class HbgTaskAllocatorTest : public ::testing::Test { // ============================================================================= TEST_F(HbgTaskAllocatorTest, InitialState) { - EXPECT_EQ(allocator.window_size(), WINDOW_SIZE); + EXPECT_EQ(allocator.capacity(), MAX_TASKS); EXPECT_EQ(allocator.active_count(), 0); EXPECT_EQ(allocator.heap_top(), 0u); EXPECT_EQ(allocator.heap_capacity(), HEAP_SIZE); @@ -73,7 +71,6 @@ TEST_F(HbgTaskAllocatorTest, AllocNonZeroSize) { auto result = allocator.alloc(100); ASSERT_FALSE(result.failed()); EXPECT_EQ(result.task_id, 0); - EXPECT_EQ(result.slot, 0); EXPECT_NE(result.packed_base, nullptr); uint64_t expected_aligned = CHIP_ALIGN_UP(100u, CHIP_ALIGN_SIZE); EXPECT_EQ(expected_aligned, 128u); @@ -90,11 +87,9 @@ TEST_F(HbgTaskAllocatorTest, SequentialTaskIds) { auto result = allocator.alloc(0); ASSERT_FALSE(result.failed()) << "Alloc failed at i=" << i; EXPECT_EQ(result.task_id, prev_id + 1) << "Task IDs must be monotonically increasing"; - EXPECT_EQ(result.slot, result.task_id & (WINDOW_SIZE - 1)); prev_id = result.task_id; } - EXPECT_EQ(allocator.active_count(), 5); - EXPECT_EQ(current_index.load(), 5) << "Head is published to shared memory"; + EXPECT_EQ(allocator.active_count(), 5) << "the next id is both the occupancy and the run's total"; } TEST_F(HbgTaskAllocatorTest, OutputSizeAlignment) { @@ -108,15 +103,39 @@ TEST_F(HbgTaskAllocatorTest, OutputSizeAlignment) { EXPECT_EQ(allocator.heap_top(), 192u); } -TEST_F(HbgTaskAllocatorTest, SlotMappingPowerOfTwoWindow) { - std::set slots; - for (int i = 0; i < WINDOW_SIZE; i++) { +// A task id IS its task-table index: ids run 0..capacity-1 with no wrap, so the +// whole table is addressed exactly once and no two tasks share a slot. +TEST_F(HbgTaskAllocatorTest, TaskIdIsItsOwnSlot) { + std::set ids; + for (int i = 0; i < MAX_TASKS; i++) { auto r = allocator.alloc(0); ASSERT_FALSE(r.failed()); - EXPECT_EQ(r.slot, r.task_id & (WINDOW_SIZE - 1)); - slots.insert(r.slot); + EXPECT_EQ(r.task_id, i) << "ids are handed out in order, with no wrap"; + ids.insert(r.task_id); } - EXPECT_EQ(slots.size(), static_cast(WINDOW_SIZE)) << "Every configured slot is usable exactly once"; + EXPECT_EQ(ids.size(), static_cast(MAX_TASKS)) << "Every slot is usable exactly once"; + EXPECT_TRUE(allocator.alloc(0).failed()) << "the capacity is terminal, not a wrap point"; +} + +// Nothing masks with the capacity, so it need not be a power of two: an odd count +// hands out exactly that many ids and then reports exhaustion. This is what lets a +// bind pass an arbitrary runtime_env.ring_task_window straight through. +TEST_F(HbgTaskAllocatorTest, NonPowerOfTwoCapacitySaturatesExactly) { + constexpr int32_t ODD_CAPACITY = 10; + TaskAllocator odd{}; + odd.init(ODD_CAPACITY, heap_buf, HEAP_SIZE, &error_code); + EXPECT_EQ(odd.capacity(), ODD_CAPACITY); + + for (int32_t i = 0; i < ODD_CAPACITY; i++) { + auto r = odd.alloc(0); + ASSERT_FALSE(r.failed()) << "Alloc failed at i=" << i; + EXPECT_EQ(r.task_id, i) << "ids run 0..capacity-1 for an odd capacity too"; + } + EXPECT_EQ(odd.active_count(), ODD_CAPACITY); + + auto overflow = odd.alloc(0); + EXPECT_TRUE(overflow.failed()) << "the odd capacity is the cap, not rounded up to a power of two"; + EXPECT_EQ(error_code.load(), SIMPLER_ERROR_FLOW_CONTROL_DEADLOCK); } // Zero-size allocs return the same address and don't advance the top. @@ -182,20 +201,18 @@ TEST_F(HbgTaskAllocatorTest, AllocLargerThanHeap) { EXPECT_EQ(error_code.load(), SIMPLER_ERROR_HEAP_RING_DEADLOCK); } -TEST_F(HbgTaskAllocatorTest, TaskWindowSaturates) { - for (int i = 0; i < WINDOW_SIZE; i++) { +TEST_F(HbgTaskAllocatorTest, TaskCapacitySaturates) { + for (int i = 0; i < MAX_TASKS; i++) { auto r = allocator.alloc(0); ASSERT_FALSE(r.failed()) << "Alloc failed at i=" << i; EXPECT_EQ(r.task_id, i); } - EXPECT_EQ(allocator.active_count(), WINDOW_SIZE); - EXPECT_EQ(current_index.load(), WINDOW_SIZE); + EXPECT_EQ(allocator.active_count(), MAX_TASKS); auto overflow = allocator.alloc(0); EXPECT_TRUE(overflow.failed()); EXPECT_EQ(error_code.load(), SIMPLER_ERROR_FLOW_CONTROL_DEADLOCK); - EXPECT_EQ(allocator.active_count(), WINDOW_SIZE); - EXPECT_EQ(current_index.load(), WINDOW_SIZE) << "A rejected allocation must not publish a new task"; + EXPECT_EQ(allocator.active_count(), MAX_TASKS) << "A rejected allocation must not consume a slot"; } // A failing alloc leaves the heap pointer untouched, so the reported figures @@ -204,12 +221,10 @@ TEST_F(HbgTaskAllocatorTest, FailedHeapAllocLeavesStateUnchanged) { ASSERT_FALSE(allocator.alloc(1024).failed()); uint64_t top_before = allocator.heap_top(); int32_t count_before = allocator.active_count(); - int32_t current_index_before = current_index.load(); EXPECT_TRUE(allocator.alloc(HEAP_SIZE).failed()); EXPECT_EQ(allocator.heap_top(), top_before) << "Heap pointer must not move on failure"; - EXPECT_EQ(allocator.active_count(), count_before) << "No task slot is consumed on failure"; - EXPECT_EQ(current_index.load(), current_index_before) << "No task index is published on failure"; + EXPECT_EQ(allocator.active_count(), count_before) << "No task slot is consumed and no task id handed out"; } // Once a fatal is latched, alloc() short-circuits without overwriting the first @@ -284,11 +299,11 @@ TEST_F(HbgTaskAllocatorTest, LatchedFatalShortCircuitsReserveDeferredHeap) { TEST_F(HbgTaskAllocatorTest, InitRejectsAHeapOverlappingTheRecordingVirtualRange) { TaskAllocator overlapping{}; auto *base = reinterpret_cast(GRAPH_RECORD_VIRTUAL_BASE); - EXPECT_THROW(overlapping.init(WINDOW_SIZE, ¤t_index, base, HEAP_SIZE, &error_code), AssertionError); + EXPECT_THROW(overlapping.init(MAX_TASKS, base, HEAP_SIZE, &error_code), AssertionError); TaskAllocator straddling{}; auto *just_below = reinterpret_cast(GRAPH_RECORD_VIRTUAL_BASE - 64); - EXPECT_THROW(straddling.init(WINDOW_SIZE, ¤t_index, just_below, HEAP_SIZE, &error_code), AssertionError); + EXPECT_THROW(straddling.init(MAX_TASKS, just_below, HEAP_SIZE, &error_code), AssertionError); } // What the production path passes: the graph heap is allocated out of the virtual @@ -301,7 +316,7 @@ TEST_F(HbgTaskAllocatorTest, AcceptsTheVirtualHeapWindow) { TaskAllocator virtual_heap{}; auto *base = reinterpret_cast(HEAP_VIRTUAL_BASE); - virtual_heap.init(WINDOW_SIZE, ¤t_index, base, HEAP_VIRTUAL_CAPACITY, &error_code); + virtual_heap.init(MAX_TASKS, base, HEAP_VIRTUAL_CAPACITY, &error_code); EXPECT_EQ(virtual_heap.heap_capacity(), HEAP_VIRTUAL_CAPACITY); auto r = virtual_heap.alloc(64); diff --git a/tests/ut/cpp/a5/test_hbg_submit_poison.cpp b/tests/ut/cpp/a5/test_hbg_submit_poison.cpp index 165d600af8..5d47d8d683 100644 --- a/tests/ut/cpp/a5/test_hbg_submit_poison.cpp +++ b/tests/ut/cpp/a5/test_hbg_submit_poison.cpp @@ -11,14 +11,14 @@ /** * Poison test for the "every device-read SM field is written at submit" contract. * - * host_build_graph no longer zero-fills the shared-memory task window (init-on-write): - * init_header_per_ring writes only the header, and each slot's device-read fields are + * host_build_graph does not zero-fill the shared-memory task table (init-on-write): + * init_header writes only the header, and each slot's device-read fields are * written per task at submit (prepare_task + submit_task_common + TaskPayload::init). - * Nothing else clears the window, so a device-read field a submit forgets to write would + * Nothing else clears the table, so a device-read field a submit forgets to write would * read as 0 only by allocator accident — passing every zero-backed test and failing * non-deterministically on device. * - * This test fills the whole per-slot window with a 0xAA poison byte before submitting a + * This test fills the whole task table with a 0xAA poison byte before submitting a * representative mix, then asserts that for every claimed slot [0, total_tasks) the * device-read fields carry real values, not poison. Add a device-read field and forget * its submit-path write, and this fails in-tree. @@ -65,7 +65,7 @@ class HbgSubmitPoisonTest : public ::testing::Test { // Same order the AICPU boots in: the slot arrays are not part of the // uploaded image, so nothing can push until they carry their ramp. sched.seed_queue_slots(); - ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), 4096, CHIP_TASK_WINDOW_SIZE, &sched)); + ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), 4096, CHIP_DEFAULT_GRAPH_TASKS, &sched)); } void TearDown() override { @@ -74,21 +74,21 @@ class HbgSubmitPoisonTest : public ::testing::Test { sm_arena.release(); } - // Fill the per-slot window (descriptors / payloads / slot_states / completion_flags) - // with poison. init_header_per_ring wrote only the header, so this is the state the - // window is in before any submit writes it — modelling the never-zeroed device SM. - void poison_window() { - auto &ring = sm_handle->header->ring; // host_build_graph is single-ring - const size_t n = static_cast(ring.task_window_mask) + 1; - std::memset(ring.task_descriptors, POISON, n * sizeof(TaskDescriptor)); - std::memset(ring.task_payloads, POISON, n * sizeof(TaskPayload)); - std::memset(ring.slot_states, POISON, n * sizeof(ChipTaskSlotState)); - std::memset(ring.completion_flags, POISON, n * sizeof(std::atomic)); + // Fill the task table (descriptors / payloads / slot_states / completion_flags) + // with poison. init_header wrote only the header, so this is the state the table + // is in before any submit writes it — modelling the never-zeroed device SM. + void poison_task_table() { + auto &tasks = sm_handle->header->tasks; + const size_t n = static_cast(CHIP_DEFAULT_GRAPH_TASKS); + std::memset(tasks.task_descriptors, POISON, n * sizeof(TaskDescriptor)); + std::memset(tasks.task_payloads, POISON, n * sizeof(TaskPayload)); + std::memset(tasks.slot_states, POISON, n * sizeof(ChipTaskSlotState)); + std::memset(tasks.completion_flags, POISON, n * sizeof(std::atomic)); } }; TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { - poison_window(); + poison_task_table(); orch.begin_scope(); // 1. Zero-fanin root: a real mixed (AIV0) task with an output tensor and a scalar. @@ -127,17 +127,16 @@ TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { orch.end_scope(); - auto &ring = sm_handle->header->ring; - const int32_t total = ring.fc.current_task_index.load(std::memory_order_acquire); + auto &tasks = sm_handle->header->tasks; + const int32_t total = orch.task_allocator.active_count(); ASSERT_GE(total, 4); // Every claimed slot's device-read fields must carry real values, not poison. for (int32_t local = 0; local < total; local++) { SCOPED_TRACE(testing::Message() << "slot local_id=" << local); - const int32_t slot = ring.get_slot_by_task_id(local); - const TaskDescriptor &desc = ring.task_descriptors[slot]; - const TaskPayload &pl = ring.task_payloads[slot]; - const ChipTaskSlotState &st = ring.slot_states[slot]; + const TaskDescriptor &desc = tasks.task_descriptors[local]; + const TaskPayload &pl = tasks.task_payloads[local]; + const ChipTaskSlotState &st = tasks.slot_states[local]; // Descriptor: the task id is written to this exact local id. EXPECT_EQ(desc.task_id.local(), static_cast(local)); @@ -148,7 +147,7 @@ TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { EXPECT_TRUE(state == CHIP_TASK_PENDING || state == CHIP_TASK_COMPLETED); // Completion flag is written to a real 0/1 (pending vs pre-completed), not a // poison byte (0xAA). - const uint8_t cflag = ring.completion_flags[slot].load(std::memory_order_relaxed); + const uint8_t cflag = tasks.completion_flags[local].load(std::memory_order_relaxed); EXPECT_LE(cflag, uint8_t{1}); // Payload counts are real, not the poison bit pattern. EXPECT_GE(pl.fanin_count, 0); @@ -164,8 +163,8 @@ TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { } // Field-specific coverage on the real task: tensors, scalar, packed output buffer. - const TaskDescriptor &root_desc = ring.task_descriptors[ring.get_slot_by_task_id(root.task_id().local())]; - const TaskPayload &root_pl = ring.task_payloads[ring.get_slot_by_task_id(root.task_id().local())]; + const TaskDescriptor &root_desc = tasks.task_descriptors[root.task_id().local()]; + const TaskPayload &root_pl = tasks.task_payloads[root.task_id().local()]; EXPECT_EQ(root_pl.tensor_count, 1); EXPECT_EQ(root_pl.scalar_count, 1); EXPECT_EQ(root_pl.dump_metadata.dump_arg_mask, (uint64_t{1} << 0) | (uint64_t{1} << 1)); @@ -176,7 +175,7 @@ TEST_F(HbgSubmitPoisonTest, EveryDeviceReadFieldIsWrittenOverPoison) { EXPECT_EQ(root_desc.kernel_id[static_cast(SubtaskSlot::AIV0)], 0); // The consumer's fanin is written: two duplicate deps dedupe to one. - const TaskPayload &cons_pl = ring.task_payloads[ring.get_slot_by_task_id(consumer.task_id().local())]; + const TaskPayload &cons_pl = tasks.task_payloads[consumer.task_id().local()]; EXPECT_EQ(cons_pl.fanin_count, 1); EXPECT_EQ(cons_pl.fanin_data()[0], static_cast(root.task_id().local())); } diff --git a/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp b/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp index 8bafa0545f..8e73fe17bc 100644 --- a/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp +++ b/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp @@ -62,7 +62,7 @@ class HbgGraphDefinitionArenaTest : public ::testing::Test { ASSERT_TRUE(sched.init_data_from_layout(sched_layout, runtime_arena, sm_handle->sm_base)); sched.wire_arena_pointers(sched_layout, runtime_arena); - ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), HEAP_BYTES, CHIP_TASK_WINDOW_SIZE, &sched)); + ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), HEAP_BYTES, CHIP_DEFAULT_GRAPH_TASKS, &sched)); } void TearDown() override { diff --git a/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp b/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp index b4b6311408..652420d522 100644 --- a/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp +++ b/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp @@ -71,7 +71,7 @@ class HbgGraphSubmitFailureTest : public ::testing::Test { ASSERT_TRUE(sched.init_data_from_layout(sched_layout, runtime_arena, sm_handle->sm_base)); sched.wire_arena_pointers(sched_layout, runtime_arena); - ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), HEAP_BYTES, CHIP_TASK_WINDOW_SIZE, &sched)); + ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), HEAP_BYTES, CHIP_DEFAULT_GRAPH_TASKS, &sched)); definition_staging.assign(STAGING_BYTES, std::byte{0}); arena.base = definition_staging.data(); @@ -427,7 +427,7 @@ TEST_F(HbgGraphSubmitFailureTest, CachedGraphUsesFinalTaskWindowSlot) { ASSERT_EQ(orch.task_allocator.active_count(), 1); TaskAllocator &allocator = orch.task_allocator; - while (allocator.active_count() < allocator.window_size() - 1) { + while (allocator.active_count() < allocator.capacity() - 1) { ASSERT_FALSE(allocator.alloc(0).failed()); } @@ -435,9 +435,9 @@ TEST_F(HbgGraphSubmitFailureTest, CachedGraphUsesFinalTaskWindowSlot) { EXPECT_FALSE(replay.execute_block); ASSERT_TRUE(replay.task_id.is_valid()); - EXPECT_EQ(replay.task_id.local(), static_cast(allocator.window_size() - 1)); - EXPECT_EQ(allocator.active_count(), allocator.window_size()); - EXPECT_EQ(sm_handle->header->ring.fc.current_task_index.load(std::memory_order_acquire), allocator.window_size()); + EXPECT_EQ(replay.task_id.local(), static_cast(allocator.capacity() - 1)); + EXPECT_EQ(allocator.active_count(), allocator.capacity()); + EXPECT_EQ(allocator.active_count(), allocator.capacity()); EXPECT_EQ(sm_handle->header->orch_error_code.load(std::memory_order_acquire), SIMPLER_ERROR_NONE); } @@ -742,9 +742,9 @@ TEST_F(HbgGraphSubmitFailureTest, AnOrdinaryAllocationInterleavesWithADeferredSh EXPECT_FALSE(orch.fatal); EXPECT_GT(orch.task_allocator.heap_top(), heap_after_ordinary) << "the shell's block sits above the ordinary task's, not before it"; - SharedMemoryRingHeader &ring = sm_handle->header->ring; - const int32_t shell_slot = ring.get_slot_by_task_id(static_cast(graph.task_id.local())); - const TaskDescriptor *shell = ring.slot_states[shell_slot].task.get(); + SharedMemoryTaskHeader &tasks = sm_handle->header->tasks; + const int32_t shell_slot = static_cast(graph.task_id.local()); + const TaskDescriptor *shell = tasks.slot_states[shell_slot].task.get(); ASSERT_NE(shell, nullptr); ASSERT_NE(shell->packed_buffer_base, nullptr); EXPECT_GE( @@ -865,13 +865,13 @@ TEST_F(HbgGraphSubmitFailureTest, RecordsAGraphWhoseBoundaryLivesInTheHeapWindow } // A hidden-alloc task's payload passes through TaskPayload::init() and nothing -// else — unlike an ordinary ring task, no dispatch-predicate assignment follows it, +// else — unlike an ordinary task, no dispatch-predicate assignment follows it, // and unlike an outer GRAPH task, no graph_reset_outer_payload precedes it. So // init() is where its predicate has to acquire a defined value: the ring's payload // storage is reused raw memory that no constructor runs over, and compact_live_image // translates every submitted slot's predicate.addr as a graph-heap address. TEST_F(HbgGraphSubmitFailureTest, AHiddenAllocTaskLeavesItsDispatchPredicateDefined) { - TaskPayload *payloads = sm_handle->header->ring.task_payloads; + TaskPayload *payloads = sm_handle->header->tasks.task_payloads; ASSERT_NE(payloads, nullptr); // 0x4A repeated has 01 as its top two bits, so read as an address it lands // inside [HEAP_VIRTUAL_BASE, GRAPH_RECORD_VIRTUAL_BASE) — the quarter of the @@ -894,7 +894,7 @@ TEST_F(HbgGraphSubmitFailureTest, AHiddenAllocTaskLeavesItsDispatchPredicateDefi ASSERT_TRUE(outputs.task_id().is_valid()); ASSERT_FALSE(orch.fatal); - const uint64_t slot = outputs.task_id().local() & (CHIP_TASK_WINDOW_SIZE - 1); + const uint64_t slot = outputs.task_id().local(); ASSERT_LT(slot, static_cast(kPoisonedSlots)) << "the submitted slot must be one this test poisoned"; EXPECT_EQ(payloads[slot].predicate.op, PredicateOp::NONE); EXPECT_EQ(payloads[slot].predicate.addr, 0u); diff --git a/tests/ut/cpp/common/test_hbg_slot_claim.cpp b/tests/ut/cpp/common/test_hbg_slot_claim.cpp index 20167d81c0..a90deec3fc 100644 --- a/tests/ut/cpp/common/test_hbg_slot_claim.cpp +++ b/tests/ut/cpp/common/test_hbg_slot_claim.cpp @@ -55,7 +55,7 @@ class HbgSlotClaimTest : public ::testing::Test { ASSERT_TRUE(sched.init_data_from_layout(sched_layout, runtime_arena, sm_handle->sm_base)); sched.wire_arena_pointers(sched_layout, runtime_arena); - ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), HEAP_BYTES, CHIP_TASK_WINDOW_SIZE, &sched)); + ASSERT_TRUE(orch.init(sm_handle->sm_base, gm_heap.data(), HEAP_BYTES, CHIP_DEFAULT_GRAPH_TASKS, &sched)); definition_staging.assign(STAGING_BYTES, std::byte{0}); GraphDefinitionArena arena{}; @@ -81,25 +81,25 @@ class HbgSlotClaimTest : public ::testing::Test { // pass left is what the claim has to overwrite. Every value here is one a // completed task really leaves behind. void poison_slot(int32_t slot) { - ChipTaskSlotState &state = sm_handle->header->ring.get_slot_state_by_slot(slot); + ChipTaskSlotState &state = sm_handle->header->tasks.get_slot_state_by_task_id(slot); state.wake_list_head.store(WAKE_LIST_SENTINEL, std::memory_order_relaxed); - state.next_in_wake_list = &sm_handle->header->ring.get_slot_state_by_slot(slot + 1); + state.next_in_wake_list = &sm_handle->header->tasks.get_slot_state_by_task_id(slot + 1); state.any_subtask_deferred.store(true, std::memory_order_relaxed); state.completed_subtasks.store(7, std::memory_order_relaxed); state.next_block_idx.store(3, std::memory_order_relaxed); state.graph_node_index = 11; - sm_handle->header->ring.completion_flags[slot].store(1, std::memory_order_relaxed); + sm_handle->header->tasks.completion_flags[slot].store(1, std::memory_order_relaxed); } void expect_slot_pristine(int32_t slot) { - ChipTaskSlotState &state = sm_handle->header->ring.get_slot_state_by_slot(slot); + ChipTaskSlotState &state = sm_handle->header->tasks.get_slot_state_by_task_id(slot); EXPECT_EQ(state.wake_list_head.load(std::memory_order_relaxed), nullptr) << "a stale SENTINEL closes the wake list, so no consumer can register on this producer"; EXPECT_EQ(state.next_in_wake_list, nullptr); EXPECT_FALSE(state.has_any_subtask_deferred()); EXPECT_EQ(state.completed_subtasks.load(std::memory_order_relaxed), 0); EXPECT_EQ(state.next_block_idx.load(std::memory_order_relaxed), 0); - EXPECT_EQ(sm_handle->header->ring.completion_flags[slot].load(std::memory_order_relaxed), 0) + EXPECT_EQ(sm_handle->header->tasks.completion_flags[slot].load(std::memory_order_relaxed), 0) << "a stale completion flag reports this task done before it has run"; } }; @@ -119,7 +119,7 @@ TEST_F(HbgSlotClaimTest, OrdinarySubmitClaimsAPoisonedSlot) { expect_slot_pristine(0); } -// The outer task of a recorded Graph takes a ring slot like any other task, and +// The outer task of a recorded Graph takes a table slot like any other task, and // takes it through its own placement rather than through prepare_task — so it // needs the same claim-time reset. It occupies one heap block and one slot for a // whole sub-DAG, which is exactly why it is easy to overlook. @@ -149,7 +149,7 @@ TEST_F(HbgSlotClaimTest, GraphOuterTaskClaimsAPoisonedSlot) { ASSERT_TRUE(orch.graph_end()); ASSERT_EQ(orch.task_allocator.active_count(), 1); - ChipTaskSlotState &outer = sm_handle->header->ring.get_slot_state_by_slot(0); + ChipTaskSlotState &outer = sm_handle->header->tasks.get_slot_state_by_task_id(0); ASSERT_EQ(outer.task_kind, TaskKind::GRAPH); expect_slot_pristine(0); } diff --git a/tests/ut/cpp/common/test_hbg_sm_compaction.cpp b/tests/ut/cpp/common/test_hbg_sm_compaction.cpp index ce11f50bf6..b47794ecae 100644 --- a/tests/ut/cpp/common/test_hbg_sm_compaction.cpp +++ b/tests/ut/cpp/common/test_hbg_sm_compaction.cpp @@ -9,7 +9,7 @@ * ----------------------------------------------------------------------------------------------------------- */ /** - * The orchestrator writes a ring-pitched shared-memory mirror; the image that + * The orchestrator writes a reservation-pitched shared-memory mirror; the image that * ships is pitched to the submitted count so its four live prefixes are * contiguous and travel as one copy. compact_live_image is the restack, and it * owes the device three things: the live payloads, a header carrying no host @@ -29,7 +29,7 @@ namespace { -constexpr uint64_t WINDOW = 64; // stands in for the ring capacity +constexpr uint64_t WINDOW = 64; // stands in for the mirror's task capacity constexpr uint64_t SUBMITTED = 5; // The per-task argument shape these tests write. Deliberately far below every cap, so @@ -69,17 +69,17 @@ class AlignedImage { class Mirror { public: Mirror() : - image_(sm_layout::ring_segment_offsets(WINDOW).end) { - const auto off = sm_layout::ring_segment_offsets(WINDOW); + image_(sm_layout::segment_offsets(WINDOW).end) { + const auto off = sm_layout::segment_offsets(WINDOW); auto *header = reinterpret_cast(image_.base()); - auto &ring = header->ring; - ring.task_window_size = WINDOW; - ring.task_window_mask = static_cast(WINDOW - 1); - ring.fc.current_task_index.store(static_cast(SUBMITTED), std::memory_order_relaxed); - ring.task_descriptors = descriptors(); - ring.task_payloads = payloads(); - ring.slot_states = slot_states(); - ring.completion_flags = completion_flags(); + auto &tasks = header->tasks; + tasks.task_descriptors_offset = off.descriptors; + tasks.completed_watermark.store(-1, std::memory_order_relaxed); + tasks.total_tasks = static_cast(SUBMITTED); + tasks.task_descriptors = descriptors(); + tasks.task_payloads = payloads(); + tasks.slot_states = slot_states(); + tasks.completion_flags = completion_flags(); (void)off; // Each live slot takes a packed region in each pool, exactly as the @@ -134,30 +134,26 @@ class Mirror { const char *base() const { return image_.base(); } TaskDescriptor *descriptors() { - return reinterpret_cast(image_.base() + sm_layout::ring_segment_offsets(WINDOW).descriptors); + return reinterpret_cast(image_.base() + sm_layout::segment_offsets(WINDOW).descriptors); } TaskPayload *payloads() { - return reinterpret_cast(image_.base() + sm_layout::ring_segment_offsets(WINDOW).payloads); + return reinterpret_cast(image_.base() + sm_layout::segment_offsets(WINDOW).payloads); } simpler::hbg::Tensor *tensor_pool() { - return reinterpret_cast( - image_.base() + sm_layout::ring_segment_offsets(WINDOW).tensor_pool - ); + return reinterpret_cast(image_.base() + sm_layout::segment_offsets(WINDOW).tensor_pool); } uint64_t *scalar_pool() { - return reinterpret_cast(image_.base() + sm_layout::ring_segment_offsets(WINDOW).scalar_pool); + return reinterpret_cast(image_.base() + sm_layout::segment_offsets(WINDOW).scalar_pool); } int32_t *fanin_pool() { - return reinterpret_cast(image_.base() + sm_layout::ring_segment_offsets(WINDOW).fanin_pool); + return reinterpret_cast(image_.base() + sm_layout::segment_offsets(WINDOW).fanin_pool); } ChipTaskSlotState *slot_states() { - return reinterpret_cast( - image_.base() + sm_layout::ring_segment_offsets(WINDOW).slot_states - ); + return reinterpret_cast(image_.base() + sm_layout::segment_offsets(WINDOW).slot_states); } std::atomic *completion_flags() { return reinterpret_cast *>( - image_.base() + sm_layout::ring_segment_offsets(WINDOW).completion_flags + image_.base() + sm_layout::segment_offsets(WINDOW).completion_flags ); } @@ -188,14 +184,14 @@ struct Compacted { explicit Compacted( Mirror &mirror, uint64_t submitted = SUBMITTED, const sm_layout::HeapRebase &rebase = kNoHeapAddresses ) : - image(sm_layout::ring_segment_offsets(sm_layout::image_extents(usage_for(submitted))).end, 0xAA), + image(sm_layout::segment_offsets(sm_layout::image_extents(usage_for(submitted))).end, 0xAA), bytes(0), used(usage_for(submitted)) { bytes = sm_layout::compact_live_image(mirror.base(), WINDOW, used, rebase, image.base()); } - sm_layout::ChipRingSegmentOffsets off(uint64_t submitted = SUBMITTED) const { - return sm_layout::ring_segment_offsets(sm_layout::image_extents(usage_for(submitted))); + sm_layout::SegmentOffsets off(uint64_t submitted = SUBMITTED) const { + return sm_layout::segment_offsets(sm_layout::image_extents(usage_for(submitted))); } TaskPayload *payload_at(uint64_t i) { return reinterpret_cast(image.base() + off().payloads) + i; } @@ -219,7 +215,7 @@ TEST(HbgSmCompaction, ShipsOnlyTheLivePrefix) { Compacted compacted(mirror); EXPECT_EQ(compacted.bytes, compacted.off().end); - EXPECT_LT(compacted.bytes, sm_layout::ring_segment_offsets(WINDOW).end); + EXPECT_LT(compacted.bytes, sm_layout::segment_offsets(WINDOW).end); } TEST(HbgSmCompaction, CarriesEveryLiveSlotsContent) { @@ -237,9 +233,12 @@ TEST(HbgSmCompaction, CarriesEveryLiveSlotsContent) { } // The header's pitch-independent fields come across; the mirror slot past the // prefix does not. - auto &ring = reinterpret_cast(compacted.image.base())->ring; - EXPECT_EQ(ring.task_window_size, WINDOW); - EXPECT_EQ(ring.fc.current_task_index.load(std::memory_order_relaxed), static_cast(SUBMITTED)); + auto &tasks = reinterpret_cast(compacted.image.base())->tasks; + EXPECT_EQ(tasks.task_descriptors_offset, sm_layout::segment_offsets(WINDOW).descriptors); + EXPECT_EQ(tasks.completed_watermark.load(std::memory_order_relaxed), -1); + // The device bounds its completed_watermark walk with this, and the restack is + // the only thing that carries it there. + EXPECT_EQ(tasks.total_tasks, static_cast(SUBMITTED)); } // The load-bearing one. Restacking changes the distance between a slot state and @@ -284,11 +283,11 @@ TEST(HbgSmCompaction, LeavesNoHostPointerInTheHeader) { Mirror mirror; Compacted compacted(mirror); - auto &ring = reinterpret_cast(compacted.image.base())->ring; - EXPECT_EQ(ring.task_descriptors, nullptr); - EXPECT_EQ(ring.task_payloads, nullptr); - EXPECT_EQ(ring.slot_states, nullptr); - EXPECT_EQ(ring.completion_flags, nullptr); + auto &tasks = reinterpret_cast(compacted.image.base())->tasks; + EXPECT_EQ(tasks.task_descriptors, nullptr); + EXPECT_EQ(tasks.task_payloads, nullptr); + EXPECT_EQ(tasks.slot_states, nullptr); + EXPECT_EQ(tasks.completion_flags, nullptr); } // A bind that submits nothing still ships its header and still attaches. Its pools @@ -299,10 +298,10 @@ TEST(HbgSmCompaction, ZeroSubmittedShipsTheHeaderAlone) { Compacted compacted(mirror, /*submitted=*/0); EXPECT_EQ(compacted.bytes, compacted.off(0).end); - EXPECT_LT(compacted.bytes, sm_layout::ring_segment_offsets(1).end); - auto &ring = reinterpret_cast(compacted.image.base())->ring; - EXPECT_EQ(ring.task_window_size, WINDOW); - EXPECT_EQ(ring.task_descriptors, nullptr); + EXPECT_LT(compacted.bytes, sm_layout::segment_offsets(1).end); + auto &tasks = reinterpret_cast(compacted.image.base())->tasks; + EXPECT_EQ(tasks.task_descriptors_offset, sm_layout::segment_offsets(WINDOW).descriptors); + EXPECT_EQ(tasks.task_descriptors, nullptr); } // The pools ship what the bind used, not what the mirror is dimensioned for. That @@ -314,7 +313,7 @@ TEST(HbgSmCompaction, PackedPoolsShipFewerBytesAndStillResolve) { // The mirror's pools cover WINDOW tasks at their full caps; the image's cover // SUBMITTED tasks at the shape they actually used. - EXPECT_LT(compacted.bytes, sm_layout::ring_segment_offsets(WINDOW).end); + EXPECT_LT(compacted.bytes, sm_layout::segment_offsets(WINDOW).end); for (uint64_t i = 0; i < SUBMITTED; ++i) { TaskPayload *shipped = compacted.payload_at(i); @@ -408,16 +407,45 @@ TEST(HbgSmCompaction, LayoutStaysWithinDeltaReach) { EXPECT_LT(compacted.bytes, static_cast(INT32_MAX)); constexpr uint64_t REACH = static_cast(INT32_MAX); - EXPECT_LE(sm_layout::ring_segment_offsets(CHIP_TASK_WINDOW_SIZE).end, REACH); + EXPECT_LE(sm_layout::segment_offsets(CHIP_DEFAULT_GRAPH_TASKS).end, REACH); // The bound is a property of the layout, not of the capacity alone: there is a // capacity past which the mirror no longer fits, and it is above the default. - uint64_t window = CHIP_TASK_WINDOW_SIZE; - while (window <= (UINT64_MAX / 2) && sm_layout::ring_segment_offsets(window).end <= REACH) { + uint64_t window = CHIP_DEFAULT_GRAPH_TASKS; + while (window <= (UINT64_MAX / 2) && sm_layout::segment_offsets(window).end <= REACH) { window *= 2; } - EXPECT_GT(window, static_cast(CHIP_TASK_WINDOW_SIZE)); - EXPECT_GT(sm_layout::ring_segment_offsets(window).end, REACH); + EXPECT_GT(window, static_cast(CHIP_DEFAULT_GRAPH_TASKS)); + EXPECT_GT(sm_layout::segment_offsets(window).end, REACH); +} + +// A task id indexes its slot directly, so no capacity is masked and a bind may pass +// any positive runtime_env.ring_task_window through. The layout walk therefore has to +// hold for a slot count that is not a power of two: every segment stays +// CHIP_ALIGN_SIZE-aligned, and each one leaves room for its own array at that pitch. +TEST(HbgSmCompaction, SegmentLayoutHoldsForANonPowerOfTwoCapacity) { + for (uint64_t capacity : {uint64_t{1}, uint64_t{3}, uint64_t{10}, uint64_t{1000}, uint64_t{16383}}) { + const sm_layout::ImageExtents e = sm_layout::mirror_extents(capacity); + const sm_layout::SegmentOffsets off = sm_layout::segment_offsets(e); + + // The descriptors sit right after the padded header, whatever the pitch. + EXPECT_EQ(off.descriptors, CHIP_ALIGN_UP(sizeof(SharedMemoryHeader), CHIP_ALIGN_SIZE)) + << "capacity " << capacity; + + const uint64_t starts[] = {off.descriptors, off.payloads, off.slot_states, off.completion_flags, + off.fanin_pool, off.tensor_pool, off.scalar_pool, off.end}; + const uint64_t spans[] = { + capacity * sizeof(TaskDescriptor), capacity * sizeof(TaskPayload), + capacity * sizeof(ChipTaskSlotState), capacity * sizeof(std::atomic), + e.fanin_elems * sizeof(int32_t), e.tensor_elems * sizeof(simpler::hbg::Tensor), + e.scalar_elems * sizeof(uint64_t), + }; + for (size_t i = 0; i < std::size(spans); ++i) { + EXPECT_EQ(starts[i] % CHIP_ALIGN_SIZE, 0u) << "capacity " << capacity << " segment " << i; + EXPECT_GE(starts[i + 1] - starts[i], spans[i]) << "capacity " << capacity << " segment " << i; + } + EXPECT_EQ(off.end % CHIP_ALIGN_SIZE, 0u) << "capacity " << capacity; + } } // The graph heap's device region is committed after orchestration, so the