Skip to content

nvsnap: remove the LD_PRELOAD interception stack that criu-v2 no longer uses #2094

Description

@balajinvda

Summary

The criu-v2 engine no longer needs the LD_PRELOAD interception stack, but the
webhook still injects it into workloads. internal/agent/checkpoint_v2.go:75
states that criu-v2 is the only CRIU engine and that the legacy injection is
obsolete, while internal/webhook/auto_inject.go:143 still sets
LD_PRELOAD=libnvsnap_intercept.so and adds patched uvloop, libuv and libzmq
init containers to matching pods.

Correction after a closer look: nothing is injected today. The injection is
opt-in per pod, autoInjectPatches returns nil unless the pod carries
nvsnap.io/auto-inject: "true", and no manifest in this repository sets that
annotation. So the cost is not a runtime tax on workloads. It is dead code, four
builder images still built and published, four external forks still referenced,
and a trap: a BYOC pod that does set the annotation receives a stack the engine
no longer needs.

Remove the stack from main and keep the implementation reachable through an
archive tag.

Inventory

Measured against main at c3e9ba9.

Delete outright, about 15500 lines:

Item Size
lib/nvsnap_intercept/ 59 files, 578 KB, 13732 lines of C and headers
internal/webhook/auto_inject.go 200 lines
docker/libuv/, docker/libzmq/, docker/pyzmq/, docker/uvloop/ 4 Dockerfiles
docker/init/ 2 files, confirm the L2 path does not use it
10 dedicated scripts 1520 lines of shell

Scripts: build-deps.sh 264, test-vllm-zmq.sh 294, local-uvloop-test.sh 235,
validate-libzmq-fork.sh 230, build-pyzmq-wheel.sh 161,
build-libzmq-image.sh 116, build-uvloop-wheel.sh 102,
auto-inject-init.sh 53, _deps.sh 37, build-intercept-lib-local.sh 28.

Edit to drop references:

  • Go, 11 files: cmd/agent/main.go, cmd/restore-entrypoint/main.go,
    internal/agent/{blob_uploader,checkpoint,checkpoint_plan_a,checkpoint_v2,restore,wake_threads}.go,
    internal/agent/quiesce_test.go,
    internal/webhook/{mutate,restore_entrypoint}.go.
  • Chart: deploy/helm/nvsnap/values.yaml,
    deploy/helm/nvsnap/templates/_helpers.tpl,
    deploy/helm/nvsnap/templates/agent-daemonset.yaml.
  • Manifests: deploy/k8s/agent-daemonset.yaml plus 7 workload and benchmark
    files (vllm-small, sglang-8b, sglang-small, three whisper-large-v3
    variants, vllm-small-rootfs-restore).
  • scripts/versions.sh loses 4 image version variables and 3 fork repo and ref
    pairs.

Notes

The legacy multi-GPU D2H path cannot be kept. It requires the interposer
preloaded into the workload, as recorded at internal/agent/checkpoint.go:2006,
so NVSNAP_LEGACY_MULTI_GPU_D2H and its gated blocks go with the stack.

The patched forks look superseded rather than merely unused. The libuv fork
exists for the lost SQ-array identity map, which was fixed on the CRIU side, and
scripts/versions.sh:67 already records that rings survive checkpoint and
restore with UV_USE_IO_URING=1. Removing the references orphans
balajinvda/libzmq, balajinvda/libuv and balajinvda/uvloop, which needs a
separate decision about those repositories.

Risk

Lower than it first looked. The workload manifests were already migrated: their
matches are comments recording that criu-v2 needs no injection, not live
LD_PRELOAD or init containers. Only 7 live references remain, all of them the
three builder image arguments in the agent DaemonSet and its Helm template.

nvsnap.io/auto-inject is not a documented public surface. It does not appear
in docs/ or fern/, and nothing outside the nvsnap subtree references it. It
is still worth confirming with whoever owns BYOC onboarding that no external
consumer sets the annotation, since removal makes it silently inert rather than
an error.

End to end on vllm-small, one sglang workload, and whisper remains worthwhile
as a plain regression check on the rebuilt agent, rather than as proof that the
engine can live without the injected libraries.

Plan

  1. Tag the pre-removal state so the implementation stays findable.
  2. Strip the injection references from the manifests and run end to end on
    vllm-small, one sglang workload, and whisper to prove the engine does not
    need them.
  3. Delete the code once step 2 passes.

References

Relates to #730
Relates to #1023

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions