Mid-generation self-healing: recovery decision logic#2
Open
dddabtc wants to merge 10 commits into
Open
Conversation
…in-region coordinator) Pipelined async-draft coordinator (specpipe.coordinate_pipe) takes gpt-oss-120B from a latency-bound ~18 to ~40 tok/s (peak ~42), greedy/exact, over WAN on 3 scattered RTX 4090s + a coordinator. Two structural wins beyond pipelining: a 3-stage 12-layer ring (4 WAN hops not 5, fits a 24GB card) and placing the layer-less coordinator in-region (cut the ring 174->102ms). Also lands a fixed-topology tree-verify path (fastverify.tree_decode, validated bit-exact) kept as groundwork. launch_oss.py = the swarm launcher. Verifiable receipt: docs/receipts/gpt-oss-120b-wan-20260619.json (4 distinct GPU UUIDs / IPs / US states, WAN edge RTTs, output token hash, sync==pipelined token match). This is the build target for the permissionless Phases 3-6.
…JOIN/step 1) Replace the shared-SHARD_PSK TCP wire with a libp2p sidecar: a Go go-libp2p daemon that runs next to the Python engine as a transparent TCP-over-libp2p tunnel. Each node holds its own ed25519 key (PeerId) — no shared secret anywhere. The engine swaps transport by one import (wire -> shard.transport, the PSK-free message codec); its logic is untouched. Proven: gpt-oss-120B split across 4 scattered boxes (UT/CA/NV/WA) served over libp2p produces bit-identical greedy tokens to the trusted-wire receipt — same 4 physical GPUs, sha f646e0db...3f70. Receipt: docs/receipts/gpt-oss-120b-libp2p-20260619.json. - sidecar/: go-libp2p tunnel + per-node identity (tunnel + self-test modes) - shard/transport.py: PSK-free send_msg/recv_msg, drop-in for wire - phase0/specpipe.py: --dump on the sync coordinator path - docs/INTEGRATION.md: full done-right architecture (5-verb build); STATE.md: status - docs/NETWORK.md: the network narrative (c0mpute = network, Shard = engine) Next (step 2): NAT traversal (DCUtR + relay; today used mapped public ports), identity<->cwt_ account binding, and re-enabling the pipelined perf path (~40 tok/s) over the sidecar.
…better Re-enable the pipelined perf path over the sidecar. The earlier failure was a latent race in serve_tail_fast: it identified the coordinator's direct-return channel by arrival order (first-readable), but over the sidecar the fire-forwarded reset can reach the predecessor connection before hello_return reaches the return connection, so the tail mis-read a reset as the hello and tore both down. Now identify the return channel by content (hello_return), tolerant of either order — transport-agnostic, no regression on TCP. Result on the 4-box 120B ring over libp2p (plain TCP, no QUIC): PIPE K=4 depth=2 warm 44.79 tok/s, bit-identical greedy output (tokens_match_sync=True, same sha as the wire receipt). vs trusted-wire 39.8 — parity-or-better this window. Receipt updated to the perf path.
…d (step 2.1/2.2) Add the NAT-traversal stack to the sidecar so home GPUs behind a router can join: QUIC, DCUtR hole-punching, circuit-relay-v2 (service + client), AutoNAT, -announce (advertise the real public addr behind a container's port mapping), explicit client.Reserve (deterministic + observable, vs autorelay's black-box finder), and a connection monitor that logs RELAY vs DIRECT. Proven on real boxes (UT relay, CA public, a genuinely NAT-blocked node): the NAT'd node reserves a relay slot and data crosses the relay both ways — i.e. a bedroom GPU can join. The direct hole-punched line (DCUtR) is wired but can't be demonstrated in a 3-node Docker test: DCUtR waits for the node's confirmed public address, which needs several observer peers / AutoNAT, and a Docker box is likely symmetric-NAT. Not a code bug — it activates on the real multi-node network with a real (cone) home NAT. Next: 2.3 identity <-> cwt_ binding (unblocked); validate the direct line on the real network or by standing up AutoNAT/relay helper nodes.
…T lab (step 2.2) Bump go-libp2p v0.33.2 -> v0.48.0 (Go 1.25.11). The old version isn't the issue, but latest is the right place for production (hardened DCUtR/AutoNAT). Code compiled clean against v0.48 (no API breaks). Proved the direct line end-to-end in a controlled lab: two Linux network namespaces, each behind its own NAT (full-cone, UPnP-style), a relay in the middle. AutoNAT confirmed each node's public address, DCUtR upgraded the relay rendezvous to a DIRECT QUIC connection between the two NAT'd nodes, and 100 KB transferred byte-identical over it (circuit-relay caps ~2 KB, so 100 KB proves it went direct, not relayed). Gotchas worth recording: host ufw silently drops the lab's UDP (bypass for the bridge); rp_filter drops NAT'd returns (disable); and libp2p refuses to dial TEST-NET ranges (203.0.113.x) as "blocked observed address" -> use a routable range (11.0.0.x). So: every node joins; full-cone home routers get a direct line via DCUtR, restricted/ symmetric fall back to the proven relay. Same model IPFS runs at scale.
A node proves it controls its libp2p PeerId by signing a challenge with its node key:
sidecar -prove <nonce> -> PEERID + base64 SIG
sidecar -verify peerid,nonce,b64sig -> OK/FAIL (reference for the c0mpute verifier)
The node-agent sends {PeerId, sig} to c0mpute alongside its cwt_ token; c0mpute verifies
and records PeerId <-> account (that half lives in the c0mpute repo, per the boundary law).
Tested: correct proof verifies, tampered nonce + impersonated PeerId both fail; and
cross-language (Go signs, c0mpute's TS verifier confirms). Step 2 (JOIN) complete.
Context was silently capped at 2048. The fast verify static KV cache was a fixed 2048 with no bounds check and corrupted past it, and the vLLM draft was started with max_model_len 2048. Both are configurable now (--max-ctx on the stages, --max-len on the draft) and an overflow raises a clean ContextOverflow instead of a CUDA out of bounds, so the node logs it and stays up. Long prompts also need memory efficient prefill, which eager attention can't give us: gpt-oss attention sinks block sdpa and the flash sink kernel is Hopper only, so eager was the only option and it OOMs around 8k. Moved the stages onto flex_attention (Ada native, handles sinks and the sliding window, validated at cosine 0.9996 vs eager) and added chunked prefill so per chunk activations stay bounded while the KV cache grows. A 95,690 token prompt now runs through an N=4 split on four 4090s with no OOM, prefill around 207 tok/s, and the model answers it correctly. Decode at long context is still slow on the simple path; graphed flex decode is the next step. Separately, settled the spec decode question: output is deterministic at a fixed K and matches the trusted wire receipt. The differences across different K are floating point non associativity in the batched verify, not a quality regression. Numbers and repro details are in docs/receipts.
…ndowed-draft spec-decode Long-context decode was unusably slow, about 0.5 tok/s at 95k, because flex recompiled every decode step as the KV grew. Two changes fix it. 1. Split prefill and decode. Prefill stays on flex (memory efficient, the only thing that survives a big query on Ada). Decode flips the layers back to eager and runs the existing CUDA graphed path: eager attention with q=K+1 is cheap even over 100k keys and the graph is a fixed-shape replay, so nothing recompiles. That alone took plain decode to 3.5 tok/s at 95k. 2. Spec-decode with a windowed draft. A full-context draft is ~800ms/round at 95k and kills spec-decode. The draft only needs recent context to predict the next tokens, so the draft query is windowed to the last few thousand tokens (--draft-ctx) while the full swarm still verifies with the complete context. That drops the draft to ~100ms and hides it behind the verify. With draft_ctx=2048, depth 4, spec-decode runs at 7.6 tok/s at 95k with correct output, about 15x the naive path. Mechanics: FastVerify does a chunked flex prefill into its static cache (mask via _compile=True so it doesn't materialise the dense mask and OOM); the fast serve loop routes prefill chunks vs decode by a flag and forwards it down the ring (else a middle stage treats a prefill chunk as a decode and tries to allocate a 49GB eager score matrix); coordinate_pipe chunks the prefill and windows the draft. The verify round-trip (about 250ms over four WAN-separated stages) is the floor now. Next lever for 10+ is sliding window KV: half the layers only need a 128 key window but currently read the whole buffer. Numbers in docs/receipts.
When a stage node dies during a request the swarm currently fails the whole request. This adds the decision logic to recover instead: localize the failed stage, swap in a warm spare holding the same layer block, replay the committed prefix, and resume. Output stays token-identical because greedy decoding resumes from the exact committed tokens, and recovery aborts loudly if the replay does not reproduce the known driver token. shard/heal.py is pure decision logic (no torch, no sockets), so it is unit tested with the standard library (tests/test_heal.py, 13 cases). Engine wiring (phase0/specpipe.py control channel + reconnectable downstream socket) and the libp2p sidecar runtime forward-swap are follow-ups that need a multi-node GPU rig; docs/research/self-healing.md records the design, scope, and those steps.
Author
|
Quick heads-up: I should have a real multi-node GPU swarm available in a couple of days. Once it's up I can take the deferred pieces end to end on actual hardware rather than in isolation:
That also lets me do the more realistic latency/throughput tuning the recovery path needs (replay cost, localization timing) instead of guessing. So this PR is the testable-in-isolation core; the hardware-dependent follow-ups can land properly after that. If you have a preference for how the engine should call into this (the |
leyten
force-pushed
the
master
branch
2 times, most recently
from
fd32138 to
df9792d
Compare
leyten
added a commit
that referenced
this pull request
Jul 15, 2026
Both broke the first residential join (LAUNCH.md P0-#2): - node_kv's flat `import transport` (the pushed-flat box layout) now falls back to shard.transport, so a repo checkout needs no hand-set PYTHONPATH. - m25_pull_range read /root/.hf_token unconditionally and died on any non-vast box: now env HF_TOKEN wins, else ~/.hf_token, else huggingface_hub's own login chain (public/cached repos need none). Pinned by tests/test_shard_stage.py (lands with the shard.stage entrypoint): the clean-env no-PYTHONPATH subprocess gate + the token-resolution unit.
leyten
added a commit
that referenced
this pull request
Jul 15, 2026
…emon Promotes the operator SSH launch string (m25_scatter_pipe.stage_cmd) into the CLI the Leg-7 node daemon execs (c0mpute NODE_DAEMON.md §4): the ring assignment arrives as flags, the engine env is derived in-process, and the stage speaks a machine-readable stdout contract a supervisor can wait on (SHARD_STAGE_OK preflight / SHARD_STAGE_READY from serve() / SHARD_STAGE_FATAL + nonzero exit). Secrets stay env-only, never argv. --check gives the daemon a no-network preflight (engine import + model-dir sanity). Also: the receipt node-key default is ~-relative (identical on root boxes, works elsewhere). tests/test_shard_stage.py: clean-env subprocess --check (PYTHONPATH stripped — the P0-#2 regression gate), FATAL contract on bad dir/missing assignment, the READY line + a frame flowing through the real serve() loop on CPU, HF-token resolution order. Full suite 554 passed / 1 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
First increment toward mid-generation fault tolerance. Today, if one stage node dies during a request, the whole request fails (
coordinate*raisesTransportError,node_kvheados._exits). Detection is already fast; this adds the recovery decision logic so a single node death becomes recoverable instead of fatal — important because the swarm targets churning consumer GPUs.Recovery is stop, swap, replay, continue: localize the dead stage → swap in a warm spare holding the same layer block → reset live KV and replay the committed prefix → resume. Output stays token-identical because greedy decoding resumes from the exact committed tokens.
Scope of this PR
Only the pure decision module
shard/heal.py— no torch, no sockets, no model — so it runs and is unit-tested anywhere:locate_failure(reports)— the coordinator only sits on the stage-0 and tail edges, so a middle death surfaces as a result that never returns. Each stage reports reachability + highest verify-chunk received/forwarded; the failed node is the first unreachable stage or the first gap (predecessor forwarded a chunk the successor never received). This also catches a frozen, not-dead node.plan_replay(prompt_ids, committed_out)— the replay prefix is the prompt plus all committed output except the current driver token, since re-feeding it must greedily reproduce that token.run_recovery(ops, ...)— the ordered state machine over an injectedSwarmOpsinterface (the engine implements the real node ops later), and it aborts loudly if the replay does not reproduce the known driver token rather than silently diverging. That check is the greedy-identity guarantee.tests/test_heal.py— 13 stdlibunittestcases (localization incl. gap/frozen, replay planning edge cases, and recovery call-order / no-spare / replay-mismatch / head-death paths). All pass.docs/research/self-healing.mdrecords the design, the scope, and the deferred items.Deliberately deferred (named, not hidden)
Follow-ups (need a multi-node GPU rig to validate end to end)
phase0/specpipe.py: a per-stage control channel beside the data socket; re-dialing the downstream socket after startup (stages today only re-accept upstream); and turning the coordinator's "timeout → fail" into "timeout →run_recovery".sidecar/main.go: a small local admin API to swap a forward route to the spare at runtime (no weight reload, no restart).Acceptance for the engine PR: fixed prompt/
K/depth, recordoutput_ids; rerun killing the protected middle stage after a committed chunk; assert the finaloutput_idsmatch exactly (ids, not text) with no weight reload and no no-failure slowdown.I don't have a GPU swarm to run the end-to-end path, so this PR is scoped to the part that's fully testable in isolation. Happy to adjust the API or scope if it doesn't fit how you'd like the engine side to call in.