mgr: MIG GPU passthrough via DRA/CDI (deviceMgr + MIG cap-node mknod) - #84
Open
m-amin-sanati wants to merge 3 commits into
Open
mgr: MIG GPU passthrough via DRA/CDI (deviceMgr + MIG cap-node mknod)#84m-amin-sanati wants to merge 3 commits into
m-amin-sanati wants to merge 3 commits into
Conversation
Adds the device manager (deviceMgr) package ported from the upstream
rodny-gpu branch: devicer radix-tree with root/net/amd/intel/nvidia
devicers, per-container device node creation under sysboxLibDir/devices,
and DeviceMounts for bind-mounting into the container.
The nvidia devicer reconciles container device paths against host roots
{"/", "/run/nvidia/driver"}, which covers both the direct /dev nodes
and MIG caps under /dev/nvidia-caps for GPU-Operator DRA/CDI setups.
Wires it into mgr.go: constructs the DeviceMgr, registers the
SetupDevices gRPC callback, removes devices on container teardown, and
appends device bind-mounts during reqMounts. Adds go-immutable-radix dep.
Signed-off-by: Amin San'ati <amin.sanati32@gmail.com>
Adds tests for the ported deviceMgr: - root devicer passthrough (unsupported devices must not be altered) - nvidia devicer DRA/CDI reconciliation across host driver roots (/dev and /run/nvidia/driver), covering MIG caps under /dev/nvidia-caps - full SetupDevices/DeviceMounts/RemoveDevices round-trip (skipped when unprivileged since device-node creation needs CAP_MKNOD) Signed-off-by: Amin San'ati <amin.sanati32@gmail.com>
When the nvidia driver is containerized (GPU-Operator), per-slice MIG capability nodes under /dev/nvidia-caps never materialize on the host; they only exist inside the driver container at /run/nvidia/driver/dev. sysbox runs containers in a user namespace and bind-mounts device nodes from a host path, so Discover() returning nil makes sysbox-runc fail with ENOENT and the workspace CrashLoopBackOff. Add isMigCapNode() to detect char nodes under /dev/nvidia-caps/nvidia-cap* and mknodMigCapNode() to create the missing node on the host from its (major, minor) pair (idempotent, skips if present). Discover() invokes the fallback when both /proc capability reads fail and the request is a MIG cap node, so the existing bind path succeeds. sysbox-mgr runs as root on the host, so there is no user-namespace mknod restriction. Signed-off-by: Amin San'ati <amin.sanati32@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the sysbox-mgr side of the MIG/DRA/CDI GPU passthrough feature.
Changes:
deviceMgr): a radix-tree of devicers (root/net/amd/intel/nvidia), per-container device-node creation under the sysbox lib dir, andDeviceMountsfor bind-mounting devices into the container. The nvidia devicer reconciles container device paths against host roots{"/", "/run/nvidia/driver"}, covering both direct/devnodes and MIG caps under/dev/nvidia-caps.isMigCapNode()detects char nodes under/dev/nvidia-caps/nvidia-cap*.mknodMigCapNode()idempotently creates a missing MIG cap node on the host from its (major, minor) pair, invoked fromDiscover()when capability reads fail for a MIG cap node (sysbox-mgr runs as root, so hostmknodis allowed). This fixes the case where per-slice MIG cap nodes never materialize on the host when the NVIDIA driver is containerized (GPU-Operator).SetupDevicesgRPC callback, removes devices on container teardown, and appends device bind-mounts during mount-request handling. Addsgo-immutable-radixdependency.Closes #83
This is part of a 3-repo feature: