Skip to content

MIG GPU passthrough via DRA/CDI (deviceMgr + MIG cap-node mknod) #83

Description

@m-amin-sanati

Need
System containers need access to NVIDIA MIG GPUs allocated through Kubernetes DRA/CDI. When the NVIDIA driver is containerized (GPU-Operator), the per-slice MIG capability nodes under /dev/nvidia-caps never materialize on the host — they only exist inside the driver container at /run/nvidia/driver/dev. sysbox runs containers in a user namespace and bind-mounts device nodes from a host path, so without these nodes the container setup fails and the workload crashes.

Proposal

  • Add a device manager (deviceMgr) that reconciles device discovery: a radix-tree of devicers (root/net/amd/intel/nvidia), per-container device-node creation under the sysbox lib dir, and DeviceMounts for bind-mounting into the container. The nvidia devicer reconciles container device paths against host roots {"/", "/run/nvidia/driver"} covering both direct /dev nodes and MIG caps under /dev/nvidia-caps.
  • Add isMigCapNode() to detect char nodes under /dev/nvidia-caps/nvidia-cap* and mknodMigCapNode() to create a missing node on the host from its (major, minor) pair — idempotent, invoked from Discover() when capability reads fail for a MIG cap node. sysbox-mgr runs as root, so host mknod is allowed.
  • Register the SetupDevices gRPC callback, remove devices on container teardown, and append device bind-mounts during mount request handling. Add go-immutable-radix dependency.

Benefits
Lets sysbox secure containers pass through NVIDIA MIG GPUs allocated via DRA/CDI, unblocking GPU-Operator-based MIG workspaces.

Part of a 3-repo feature (see sysbox-runc and sysbox-ipc PRs).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions