[Bug] `offload_train` train worker dies on CUDA 13 with `libcudart.so.12: cannot open shared object file` — the torch_memory_saver `LD_PRELOAD` `.so` is chosen by filename, not by CUDA runtime

[littlemex]

Bug Description

With --offload-train on the Megatron backend, slime/ray/actor_group.py sets LD_PRELOAD to a torch_memory_saver preload .so. It picks the file from a hard-coded list — ..._preload_cu12.abi3.so, then the unsuffixed ..._preload.abi3.so — and takes the first that exists on disk (os.path.exists).

On a CUDA 13 image this is wrong twice over:

1. The cu13 variant is never enumerated.
2. The test is file existence, not loadability. The cu12 .so exists as a file, so it is selected — but it links libcudart.so.12, which is absent on CUDA 13. Once it is LD_PRELOADed, every child process (including the train worker) fails with libcudart.so.12: cannot open shared object file: No such file or directory.

The wheel actually ships a correct cu13 build right next to the selected one; the selection logic just never considers it.

Location — actor_group.py L64-L84:

if self.args.offload_train and self.args.train_backend == "megatron":
    import torch_memory_saver

    for path in [
        "torch_memory_saver_hook_mode_preload_cu12.abi3.so",
        "torch_memory_saver_hook_mode_preload.abi3.so",
    ]:
        dynlib_path = os.path.join(
            os.path.dirname(os.path.dirname(torch_memory_saver.__file__)),
            path,
        )
        if os.path.exists(dynlib_path):
            break
    else:
        raise FileNotFoundError(...)

    env_vars["LD_PRELOAD"] = dynlib_path

Steps to Reproduce

1. Environment: a CUDA 13 build of PyTorch (torch 2.11.0+cu130) with a torch_memory_saver that ships CUDA-suffixed preload builds (0.0.9.post1: _cu12 / _cu13 / unsuffixed).
2. Run any Megatron GRPO recipe with --offload-train (the default offload path), e.g. the Qwen3-4B colocated recipe.
3. Shortly after the SGLang rollout servers come up, the train worker crashes:

   bash: error while loading shared libraries: libcudart.so.12: cannot open shared object file: No such file or directory
   

Deterministic minimal reproduction of the selection defect (no full job, no multi-GPU) — load each candidate the way LD_PRELOAD would:

import os, ctypes, torch, torch_memory_saver
base = os.path.dirname(os.path.dirname(torch_memory_saver.__file__))
print(torch.__version__, "| torch.version.cuda =", torch.version.cuda)
for name in [
    "torch_memory_saver_hook_mode_preload_cu12.abi3.so",  # current candidate[0]
    "torch_memory_saver_hook_mode_preload.abi3.so",        # current candidate[1]
    "torch_memory_saver_hook_mode_preload_cu13.abi3.so",   # correct, but not in the list
]:
    p = os.path.join(base, name)
    try:
        ctypes.CDLL(p); load = "CDLL OK"
    except OSError as e:
        load = f"CDLL FAIL: {e}"
    print(f"  exists={os.path.exists(p)!s:5} {load}  <- {name}")

Output on a CUDA 13 image:

2.11.0+cu130 | torch.version.cuda = 13.0
  exists=True  CDLL FAIL: libcudart.so.12: cannot open shared object file: No such file or directory  <- ..._preload_cu12.abi3.so
  exists=True  CDLL FAIL: libcudart.so.12: cannot open shared object file: No such file or directory  <- ..._preload.abi3.so
  exists=True  CDLL OK                                                                                <- ..._preload_cu13.abi3.so

So os.path.exists passes for all three, slime picks the first (cu12), and it cannot load. The unsuffixed build is byte-identical to the cu12 build (same missing dependency), so the fallback cannot rescue a CUDA 13 host:

$ md5sum ..._preload.abi3.so ..._preload_cu12.abi3.so ..._preload_cu13.abi3.so
9e724677af9e636c1b3c3e910faef96d  ..._preload.abi3.so
9e724677af9e636c1b3c3e910faef96d  ..._preload_cu12.abi3.so   # identical to the unsuffixed one
1bd3044ce135ab84e633f3db24b35965  ..._preload_cu13.abi3.so
$ ldd ..._preload_cu12.abi3.so | grep libcudart  ->  libcudart.so.12 => not found
$ ldd ..._preload_cu13.abi3.so | grep libcudart  ->  libcudart.so.13 => /usr/local/cuda/.../libcudart.so.13

Expected Behavior

On CUDA 13, LD_PRELOAD should point at the cu13 preload build (or fail with a clear, actionable error if none matches the runtime), and the train worker should start.

Actual Behavior

LD_PRELOAD points at a cu12 build that cannot load; every child process dies with libcudart.so.12: cannot open shared object file, and the train worker never starts. There is no message pointing at the CUDA-version mismatch.

Environment

• slime version: current main (the defect is at slime/ray/actor_group.py L64-L84); the same bug is also present on the v0.2.4 tag.
• Python version: 3.12
• PyTorch version: 2.11.0+cu130
• CUDA/ROCm version: CUDA 13.0
• GPU type and count: NVIDIA H200, 2 nodes × 8 GPUs
• OS: Linux
• SGLang version (if relevant): 0.5.12.post1
• Megatron-LM version (if relevant): 3714d81 (train backend; the crash is on the --offload-train megatron path)
• torch_memory_saver: 0.0.9.post1 (ships _cu12 / _cu13 / unsuffixed preload builds and the get_binary_path_from_package resolver)

Logs

(SGLangEngine ...) The server is fired up and ready to roll!   # rollout side comes up
(raylet, ip=...) bash: error while loading shared libraries: libcudart.so.12: cannot open shared object file: No such file or directory

Additional Context

Root cause. The selection decides on filename presence rather than CUDA-runtime match, and never enumerates the cu13 variant. torch_memory_saver already solves exactly this: torch_memory_saver/utils.py exposes get_binary_path_from_package(stem), which detects the CUDA major (torch.version.cuda first, then probing libcudart.so.<major> over (13, 12)) and returns the matching <stem>_cu<major>.*.so; the library uses it internally for this very stem in hooks/mode_preload.py. slime hard-codes a filename list instead of calling this resolver.

Suggested fix (PR attached). Delegate .so selection to torch_memory_saver's own CUDA-aware resolver when present, and otherwise fall back to a candidate list keyed on the detected CUDA major (torch.version.cuda, which needs no GPU/driver in the process — important because slime resolves this on the Ray driver, which may be a CPU-only head node). Note: slime's docker/Dockerfile pins torch_memory_saver@a193d9dd, which already includes the resolver, so the resolver path is what ships; the fallback keeps the fix correct against older/other pins.

Not a duplicate. I searched issues/PRs and found nothing covering this CUDA-major selection defect. Three nearby tickets touch the same area but are distinct root causes: #1936 / #1937 (an "invalid LD_PRELOAD" assertion under --disable-weights-backuper; #1937 edits the same actor_group.py block but changes whether the preload setup runs, not which .so is chosen — orthogonal), #1690 (cudaErrorInvalidValue after checkpoint save; a runtime-interception bug), and #1895 (CUresult error 1 in actor.sleep(); a double-sleep regression).

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and my issue is not addressed there.
• I have searched for existing issues and this is not a duplicate.
• I have provided a minimal, reproducible example.

[littlemex]

PR: #2187