Refactor: hbg records and ships Definitions in one retained block - #1988
Merged
ChaoWao merged 1 commit intoAug 24, 2026
Merged
Conversation
Each distinct Graph Definition was uploaded as its own device object: for
every one, bind acquired a retained device block, allocated a host staging
vector, assembled [GraphDefinitionHeader][Definition image] into it, issued
its own rtMemcpy, and freed the vector -- and the image it copied from was
itself a vector the recording had allocated and would return. A dsv4 bind
has eight Definitions, so that is eight allocate-copy-free rounds for
1.0 MB, on a path whose cost is per-call rather than per-byte.
The objects now share one device block and one host staging block, both
retained per pipeline slot and grown to a high-water mark, and the
recordings build their images directly into that staging:
- bind reads the retained staging before orchestration and hands it to the
Graph host state as a GraphDefinitionArena. The capacity on offer is
whatever the previous bind left, because a run's total is not known until
every recording has ended.
- graph_build_definition splits at the point the size becomes known.
graph_layout_definition settles the counts, section offsets and
total_bytes without writing anything; the recording thread claims an
object of that size; graph_fill_definition writes the image at the
claimed offset.
- the upload writes the object headers into the room the arena left in
front of each image and ships the used prefix with one copy_to_device.
A steady-state bind therefore copies no image at all, and acquires no
host memory for its Definitions.
Three properties make the arena safe to write into from several recording
threads while the submitting thread is still submitting shells:
- the claim is a compare-exchange that fails rather than advancing the
cursor past the capacity, so a Definition that does not fit costs the run
its own slot and no one else's. It is built in a buffer of its own
instead and copied in by the upload, which then grows the block -- so
outgrowing the retained capacity costs a copy, not the run, and the next
bind is sized for it. graph_upload's new `spilled=` counts those, and is
0 on every bind after the first.
- the arena never moves during orchestration, since a recorder holds
addresses inside it. Only the upload may grow it, which preserves the
content and is why a published record names an offset rather than an
address; by then nothing is recording.
- an object is aligned and padded to GRAPH_DEFINITION_OBJECT_ALIGN, a named
constant beside the header whose size is a multiple of it, so every image
base carries the alignment its section offsets assume. The fill asserts
that rather than trusting the arena's provider.
The image is still zero-filled before it is written, and now for a second
reason: the alignment slack between sections is inside the hashed range, so a
slot a previous bind wrote would give two structurally identical Definitions
unequal content hashes.
HostApi grows get_graph_definition_staging and replaces
acquire_graph_definition_buffer with acquire_graph_definition_block, which
takes no Definition key and returns both blocks, since the layout inside them
belongs to the caller. The device block is still zeroed when it is
(re)allocated, so a region no upload has covered reads as zero and fails the
magic gate instead of being read as a stale object.
Measured on dsv4, interleaved base/measure/base/measure at six rounds each
(8 Definitions, 86 Graph submissions, 129 host tasks per bind on both arms):
graph_upload loses the ~40 minor faults per bind it took allocating those
staging vectors (median 38 and 43 -> 1 and 1) and its minimum drops 0.225 ->
0.141 ms and 0.282 -> 0.106 ms. host_orch does not move -- the two
repetitions disagree in sign on both its fault count and its duration -- and
the investigation entry records why: a freed 126 KB block is reused from the
heap without re-faulting, so an image vector was never part of that segment's
fault tail. The control-plane total is not resolvable from this A/B either
(+0.355 ms then -0.166 ms), which is what a ~0.1 ms change looks like on a
box whose load average sat between 40 and 66.
test_hbg_graph_definition_arena covers what the scene tests cannot see, since
they pass either way: objects land at aligned, disjoint offsets that the
claimed prefix exactly covers, and a run with no arena or with one too small
still publishes a valid image.
|
Warning Review limit reachedNext included review available in 5 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (23)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
5 tasks
6 tasks
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Aug 25, 2026
Every bind allocated the host mirror of the runtime shared memory from the heap and freed it on the way out. The mirror is dimensioned for the run's configured task count, which is 82 MB at the dsv4 case's 16384 — far above every glibc reuse threshold, so each bind was one mmap and one guaranteed munmap of that size. A bind's own faults then cost 14-33 us each instead of ~1.7 us, because the unmap holds mmap_lock for write and excludes every faulting thread in the address space (docs/investigations/2026-08-host-orch-phase-tail-is-page-faults.md). The mirror is now the platform runner's, one buffer per pipeline slot, grown to the high-water mark and mapped for the life of the process — the same shape hw-native-sys#1988 gave the Graph Definition staging block, reached through a new HostApi::acquire_sm_mirror. The buffer is an uninitialized owning block rather than a container: the layout is init-on-write, so a value-initializing resize would fault in every page of a capacity whose live prefix is a fraction of it, and would copy bytes that mean nothing between binds. First touch still commits the mirror, so a run pays only for the bytes it writes. Reuse needs no new invariant. A bind already cleared only the fixed-size header and relied on init-on-write for the per-slot segments, which is why compact_live_image can ship each segment's used prefix: prepare_task writes the slot state and clears the completion flag as it claims a slot, and TaskPayload::init is the single payload-init point. The one shipped range no device-side reader reaches is the alignment padding the fanin cursor rounds past, which lies outside every payload's fanin_count. The buffer is released at Worker finalization on both the ordinary and the force-reset path, since it is host memory that no device failure invalidates. Measured on dsv4, interleaved twice: mallinfo2's hblkhd stops returning to its pre-bind value on 6 of 6 binds, which is the map/unmap going away, and the cold-bind fault count matches the per-bind allocation it replaces. host_orch's warm-bind fault count and the control-plane duration resolve in neither direction — the mirror is ~6 THP faults of a ~1200-fault bind. This closes the investigation entry's "Where a fix would go" list, so the entry is amended to say so: items 1-3 shipped as hw-native-sys#1981 and item 4 as hw-native-sys#1988 plus this change, and none of them reached the ~1100 faults the submitting thread takes per bind. hw-native-sys#1981 already reported those unmoved when the recording's allocations went away; this is the second measurement of the same thing. Attributing them needs mincore() on a buffer's pages before the write, not another guess at which allocation it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Aug 25, 2026
Every bind allocated the host mirror of the runtime shared memory from the heap and freed it on the way out. The mirror is dimensioned for the run's configured task count, which is 82 MB at the dsv4 case's 16384 — far above every glibc reuse threshold, so each bind was one mmap and one guaranteed munmap of that size. A bind's own faults then cost 14-33 us each instead of ~1.7 us, because the unmap holds mmap_lock for write and excludes every faulting thread in the address space (docs/investigations/2026-08-host-orch-phase-tail-is-page-faults.md). The mirror is now the platform runner's, one buffer per pipeline slot, grown to the high-water mark and held across binds until Worker finalization — the same shape hw-native-sys#1988 gave the Graph Definition staging block, reached through a new HostApi::acquire_sm_mirror. The buffer is an uninitialized owning block rather than a container: the layout is init-on-write, so a value-initializing resize would fault in every page of a capacity whose live prefix is a fraction of it, and would copy bytes that mean nothing between binds. First touch still commits the mirror, so a run pays only for the bytes it writes. Reuse needs no new invariant. A bind already cleared only the fixed-size header and relied on init-on-write for the per-slot segments, which is why compact_live_image can ship each segment's used prefix: prepare_task writes the slot state and clears the completion flag as it claims a slot, and TaskPayload::init is the single payload-init point. The one shipped range no device-side reader reaches is the alignment padding the fanin cursor rounds past, which lies outside every payload's fanin_count. The buffer is released at Worker finalization on both the ordinary and the force-reset path, since it is host memory that no device failure invalidates. Measured on dsv4, interleaved twice: mallinfo2's hblkhd stops returning to its pre-bind value on 6 of 6 binds, which is the map/unmap going away, and the cold-bind fault count matches the per-bind allocation it replaces. host_orch's warm-bind fault count and the control-plane duration resolve in neither direction — the mirror is ~6 THP faults of a ~1200-fault bind. This closes the investigation entry's "Where a fix would go" list, so the entry is amended to say so: items 1-3 shipped as hw-native-sys#1981 and item 4 as hw-native-sys#1988 plus this change, and none of them reached the ~1100 faults the submitting thread takes per bind. hw-native-sys#1981 already reported those unmoved when the recording's allocations went away; this is the second measurement of the same thing. Attributing them needs mincore() on a buffer's pages before the write, not another guess at which allocation it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ChaoWao
added a commit
that referenced
this pull request
Aug 25, 2026
Every bind allocated the host mirror of the runtime shared memory from the heap and freed it on the way out. The mirror is dimensioned for the run's configured task count, which is 82 MB at the dsv4 case's 16384 — far above every glibc reuse threshold, so each bind was one mmap and one guaranteed munmap of that size. A bind's own faults then cost 14-33 us each instead of ~1.7 us, because the unmap holds mmap_lock for write and excludes every faulting thread in the address space (docs/investigations/2026-08-host-orch-phase-tail-is-page-faults.md). The mirror is now the platform runner's, one buffer per pipeline slot, grown to the high-water mark and held across binds until Worker finalization — the same shape #1988 gave the Graph Definition staging block, reached through a new HostApi::acquire_sm_mirror. The buffer is an uninitialized owning block rather than a container: the layout is init-on-write, so a value-initializing resize would fault in every page of a capacity whose live prefix is a fraction of it, and would copy bytes that mean nothing between binds. First touch still commits the mirror, so a run pays only for the bytes it writes. Reuse needs no new invariant. A bind already cleared only the fixed-size header and relied on init-on-write for the per-slot segments, which is why compact_live_image can ship each segment's used prefix: prepare_task writes the slot state and clears the completion flag as it claims a slot, and TaskPayload::init is the single payload-init point. The one shipped range no device-side reader reaches is the alignment padding the fanin cursor rounds past, which lies outside every payload's fanin_count. The buffer is released at Worker finalization on both the ordinary and the force-reset path, since it is host memory that no device failure invalidates. Measured on dsv4, interleaved twice: mallinfo2's hblkhd stops returning to its pre-bind value on 6 of 6 binds, which is the map/unmap going away, and the cold-bind fault count matches the per-bind allocation it replaces. host_orch's warm-bind fault count and the control-plane duration resolve in neither direction — the mirror is ~6 THP faults of a ~1200-fault bind. This closes the investigation entry's "Where a fix would go" list, so the entry is amended to say so: items 1-3 shipped as #1981 and item 4 as #1988 plus this change, and none of them reached the ~1100 faults the submitting thread takes per bind. #1981 already reported those unmoved when the recording's allocations went away; this is the second measurement of the same thing. Attributing them needs mincore() on a buffer's pages before the write, not another guess at which allocation it is.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Each distinct Graph Definition was uploaded as its own device object: for every one, bind acquired a retained device block, allocated a host staging vector, assembled
[GraphDefinitionHeader][Definition image]into it, issued its ownrtMemcpy, and freed the vector — and the image it copied from was itself a vector the recording had allocated and would return. A dsv4 bind has eight Definitions, so that is eight allocate-copy-free rounds for 1.0 MB, on a path whose cost is per-call rather than per-byte.The objects now share one device block and one host staging block, both retained per pipeline slot and grown to a high-water mark, and the recordings build their images directly into that staging:
GraphDefinitionArena. The capacity on offer is whatever the previous bind left, because a run's total is not known until every recording has ended.graph_build_definitionsplits at the point the size becomes known.graph_layout_definitionsettles the counts, section offsets andtotal_byteswithout writing anything; the recording thread claims an object of that size;graph_fill_definitionwrites the image at the claimed offset.copy_to_device. A steady-state bind copies no image at all, and acquires no host memory for its Definitions.Three properties make the arena safe to write into from several recording threads while the submitting thread is still submitting shells:
graph_upload's newspilled=counts those, and is 0 on every bind after the first.GRAPH_DEFINITION_OBJECT_ALIGN, a named constant beside the header whose size is a multiple of it, so every image base carries the alignment its section offsets assume. The fill asserts that rather than trusting the arena's provider.The image is still zero-filled before it is written, and now for a second reason: the alignment slack between sections is inside the hashed range, so a slot a previous bind wrote would give two structurally identical Definitions unequal content hashes.
HostApigrowsget_graph_definition_stagingand replacesacquire_graph_definition_bufferwithacquire_graph_definition_block, which takes no Definition key and returns both blocks, since the layout inside them belongs to the caller. The device block is still zeroed when it is (re)allocated, so a region no upload has covered reads as zero and fails the magic gate instead of being read as a stale object.Measurement
dsv4, interleaved
base, measure, base, measureat six rounds each per.claude/skills/hbg-bind-phases; both arms ran the same command string (the logs'[stamp]lines differ only in the commit) and did the same work — 8 Definitions, 86 Graph submissions, 129 host tasks per bind. Two figures per cell, one per repetition:graph_uploadminflt, min / median / maxgraph_uploaddur min (ms)host_orchminflt, min / medianOnly
graph_uploadmoved, and it moved consistently in both repetitions: the ~40 minor faults per bind it took allocating one staging vector per Definition are gone, and so is the copy.host_orchdid not move, and the investigation entry now records why: the two repetitions disagree in sign on both its fault count and its duration, which is that file's own criterion for "not resolvable". A freed 126 KB block is reused from the heap without re-faulting, so an image vector was never part of that segment's fault tail — size against glibc's mmap and trim thresholds, not byte count, decides what shows up there. The control-plane total is not resolvable either (+0.355 ms then −0.166 ms), which is what a ~0.1 ms change looks like on a box whose load average sat between 40 and 66.Testing
examples tests/stona2a3sim(21/21 cases) anda5sim(17/17), both after the rebase onto66ba5c4a1examples tests/st -m 'not sdma' --platform a2a3 --exclude-level 4undertask-submit, 150 selected / 57 resource-phase cases, twicectest -LE requires_hardware— 119/119, including the newtest_hbg_graph_definition_arenafor a2a3 and a5pytest tests/ut -m "not requires_hardware"— 1886 passed, 7 skippedspilled=verified across rounds: 2 of 12 binds (one per rank, cold) spill; the other 10 reportspilled=0test_hbg_graph_definition_arenacovers what the scene tests cannot see, since they pass either way: objects land at aligned, disjoint offsets that the claimed prefix exactly covers, and a run with no arena or with one too small still publishes a valid image.One caveat from the hardware runs, recorded rather than swept under: the first corpus run on this branch hit a device-side AIV
ub address out of boundsinworker_async_endpoint, which cascaded to507018/SCHEDULER_TIMEOUT. It did not reproduce — that case passed 3/3 alone on66ba5c4a1, the same corpus passed 57/57 on66ba5c4a1, and the second corpus run on this branch passed 57/57 — and the case records no Graph at all, so this diff does not execute in it. Worth an eye on CI: a UB fault is not a flaky timeout.