Summary
The criu-v2 engine no longer needs the LD_PRELOAD interception stack, but the
webhook still injects it into workloads. internal/agent/checkpoint_v2.go:75
states that criu-v2 is the only CRIU engine and that the legacy injection is
obsolete, while internal/webhook/auto_inject.go:143 still sets
LD_PRELOAD=libnvsnap_intercept.so and adds patched uvloop, libuv and libzmq
init containers to matching pods.
Correction after a closer look: nothing is injected today. The injection is
opt-in per pod, autoInjectPatches returns nil unless the pod carries
nvsnap.io/auto-inject: "true", and no manifest in this repository sets that
annotation. So the cost is not a runtime tax on workloads. It is dead code, four
builder images still built and published, four external forks still referenced,
and a trap: a BYOC pod that does set the annotation receives a stack the engine
no longer needs.
Remove the stack from main and keep the implementation reachable through an
archive tag.
Inventory
Measured against main at c3e9ba9.
Delete outright, about 15500 lines:
| Item |
Size |
lib/nvsnap_intercept/ |
59 files, 578 KB, 13732 lines of C and headers |
internal/webhook/auto_inject.go |
200 lines |
docker/libuv/, docker/libzmq/, docker/pyzmq/, docker/uvloop/ |
4 Dockerfiles |
docker/init/ |
2 files, confirm the L2 path does not use it |
| 10 dedicated scripts |
1520 lines of shell |
Scripts: build-deps.sh 264, test-vllm-zmq.sh 294, local-uvloop-test.sh 235,
validate-libzmq-fork.sh 230, build-pyzmq-wheel.sh 161,
build-libzmq-image.sh 116, build-uvloop-wheel.sh 102,
auto-inject-init.sh 53, _deps.sh 37, build-intercept-lib-local.sh 28.
Edit to drop references:
- Go, 11 files:
cmd/agent/main.go, cmd/restore-entrypoint/main.go,
internal/agent/{blob_uploader,checkpoint,checkpoint_plan_a,checkpoint_v2,restore,wake_threads}.go,
internal/agent/quiesce_test.go,
internal/webhook/{mutate,restore_entrypoint}.go.
- Chart:
deploy/helm/nvsnap/values.yaml,
deploy/helm/nvsnap/templates/_helpers.tpl,
deploy/helm/nvsnap/templates/agent-daemonset.yaml.
- Manifests:
deploy/k8s/agent-daemonset.yaml plus 7 workload and benchmark
files (vllm-small, sglang-8b, sglang-small, three whisper-large-v3
variants, vllm-small-rootfs-restore).
scripts/versions.sh loses 4 image version variables and 3 fork repo and ref
pairs.
Notes
The legacy multi-GPU D2H path cannot be kept. It requires the interposer
preloaded into the workload, as recorded at internal/agent/checkpoint.go:2006,
so NVSNAP_LEGACY_MULTI_GPU_D2H and its gated blocks go with the stack.
The patched forks look superseded rather than merely unused. The libuv fork
exists for the lost SQ-array identity map, which was fixed on the CRIU side, and
scripts/versions.sh:67 already records that rings survive checkpoint and
restore with UV_USE_IO_URING=1. Removing the references orphans
balajinvda/libzmq, balajinvda/libuv and balajinvda/uvloop, which needs a
separate decision about those repositories.
Risk
Lower than it first looked. The workload manifests were already migrated: their
matches are comments recording that criu-v2 needs no injection, not live
LD_PRELOAD or init containers. Only 7 live references remain, all of them the
three builder image arguments in the agent DaemonSet and its Helm template.
nvsnap.io/auto-inject is not a documented public surface. It does not appear
in docs/ or fern/, and nothing outside the nvsnap subtree references it. It
is still worth confirming with whoever owns BYOC onboarding that no external
consumer sets the annotation, since removal makes it silently inert rather than
an error.
End to end on vllm-small, one sglang workload, and whisper remains worthwhile
as a plain regression check on the rebuilt agent, rather than as proof that the
engine can live without the injected libraries.
Plan
- Tag the pre-removal state so the implementation stays findable.
- Strip the injection references from the manifests and run end to end on
vllm-small, one sglang workload, and whisper to prove the engine does not
need them.
- Delete the code once step 2 passes.
References
Relates to #730
Relates to #1023
Summary
The criu-v2 engine no longer needs the LD_PRELOAD interception stack, but the
webhook still injects it into workloads.
internal/agent/checkpoint_v2.go:75states that criu-v2 is the only CRIU engine and that the legacy injection is
obsolete, while
internal/webhook/auto_inject.go:143still setsLD_PRELOAD=libnvsnap_intercept.soand adds patched uvloop, libuv and libzmqinit containers to matching pods.
Correction after a closer look: nothing is injected today. The injection is
opt-in per pod,
autoInjectPatchesreturns nil unless the pod carriesnvsnap.io/auto-inject: "true", and no manifest in this repository sets thatannotation. So the cost is not a runtime tax on workloads. It is dead code, four
builder images still built and published, four external forks still referenced,
and a trap: a BYOC pod that does set the annotation receives a stack the engine
no longer needs.
Remove the stack from main and keep the implementation reachable through an
archive tag.
Inventory
Measured against main at c3e9ba9.
Delete outright, about 15500 lines:
lib/nvsnap_intercept/internal/webhook/auto_inject.godocker/libuv/,docker/libzmq/,docker/pyzmq/,docker/uvloop/docker/init/Scripts:
build-deps.sh264,test-vllm-zmq.sh294,local-uvloop-test.sh235,validate-libzmq-fork.sh230,build-pyzmq-wheel.sh161,build-libzmq-image.sh116,build-uvloop-wheel.sh102,auto-inject-init.sh53,_deps.sh37,build-intercept-lib-local.sh28.Edit to drop references:
cmd/agent/main.go,cmd/restore-entrypoint/main.go,internal/agent/{blob_uploader,checkpoint,checkpoint_plan_a,checkpoint_v2,restore,wake_threads}.go,internal/agent/quiesce_test.go,internal/webhook/{mutate,restore_entrypoint}.go.deploy/helm/nvsnap/values.yaml,deploy/helm/nvsnap/templates/_helpers.tpl,deploy/helm/nvsnap/templates/agent-daemonset.yaml.deploy/k8s/agent-daemonset.yamlplus 7 workload and benchmarkfiles (
vllm-small,sglang-8b,sglang-small, threewhisper-large-v3variants,
vllm-small-rootfs-restore).scripts/versions.shloses 4 image version variables and 3 fork repo and refpairs.
Notes
The legacy multi-GPU D2H path cannot be kept. It requires the interposer
preloaded into the workload, as recorded at
internal/agent/checkpoint.go:2006,so
NVSNAP_LEGACY_MULTI_GPU_D2Hand its gated blocks go with the stack.The patched forks look superseded rather than merely unused. The libuv fork
exists for the lost SQ-array identity map, which was fixed on the CRIU side, and
scripts/versions.sh:67already records that rings survive checkpoint andrestore with
UV_USE_IO_URING=1. Removing the references orphansbalajinvda/libzmq,balajinvda/libuvandbalajinvda/uvloop, which needs aseparate decision about those repositories.
Risk
Lower than it first looked. The workload manifests were already migrated: their
matches are comments recording that criu-v2 needs no injection, not live
LD_PRELOADor init containers. Only 7 live references remain, all of them thethree builder image arguments in the agent DaemonSet and its Helm template.
nvsnap.io/auto-injectis not a documented public surface. It does not appearin
docs/orfern/, and nothing outside the nvsnap subtree references it. Itis still worth confirming with whoever owns BYOC onboarding that no external
consumer sets the annotation, since removal makes it silently inert rather than
an error.
End to end on
vllm-small, one sglang workload, and whisper remains worthwhileas a plain regression check on the rebuilt agent, rather than as proof that the
engine can live without the injected libraries.
Plan
vllm-small, one sglang workload, and whisper to prove the engine does notneed them.
References
Relates to #730
Relates to #1023