Repository navigation
fsdp2_scripts: one multi-GPU FSDP2 launcher per python_scripts file, generated by update_all_notebooks.py - #416
Conversation
update_all_notebooks.py gains generate_fsdp2_scripts and render_fsdp2_launcher. After each python_scripts conversion (the full pass and the AMD-only pass) it writes fsdp2_scripts/<name>.py for every python_scripts/<name>.py, copies scripts/fsdp2_launch.py to fsdp2_scripts/unsloth_fsdp2_launch.py and renders fsdp2_scripts/README.md. Files are staged before anything is replaced, only changed files are written, and launchers with no script are removed (scoped to AMD-* in the AMD-only pass). _assert_fsdp2_mirror checks both directions. tests/test_fsdp2_scripts_mirror.py gates the committed tree against the generator and exercises idempotence, stale cleanup and the AMD-only scope. fsdp2_scripts is added to the exec-literal lint targets, the CI paths filter and the CODEOWNERS notes.
tests/test_fsdp2_launch.py needs CPU torch, so it runs in the api-drift job; tests/test_fsdp2_adapter_covers_scripts.py runs with the no-torch static checks.
scripts/fsdp2_launch.py runs a python_scripts/<name>.py mirror on every visible GPU with Unsloth native FSDP2: it counts GPUs in a child process, re-execs itself under torch.distributed.run --standalone for 2+ ranks and runs the target in-process for 0 or 1. Inside torchrun the target runs via runpy with a narrow get_ipython stand-in: shell steps and file writes run once per node on local rank 0 while the other local ranks wait on a marker carrying the return code, %env and %cd apply on every rank, and package installs are skipped unless UNSLOTH_FSDP2_RUN_INSTALLS=1. tests/test_fsdp2_launch.py covers the argv, count precedence, refusal, worker mode and a real two-rank CPU torchrun. tests/test_fsdp2_adapter_covers_scripts.py checks every live get_ipython() call in python_scripts/ is supported.
…two dynamic-execution call sites
…ction - Marker folders are unique per launch: the parent sets UNSLOTH_FSDP2_RUN_ID before starting torchrun, other launchers fall back to the parent pid. Local rank 0 removes the folder at exit once the other ranks have read the last marker or exited. - A LOCAL_RANK left in the shell without WORLD_SIZE / LOCAL_WORLD_SIZE is ignored, shell steps only synchronise with 2+ local ranks, and a waiting rank gives up after UNSLOTH_FSDP2_WAIT_TIMEOUT (default 6 hours) with a clear error. - Installs: by default a pip / uv pip / apt-get / ollama install is skipped only when everything it asks for is already present (versions checked), otherwise it runs once per node. Installs into another environment always run. UNSLOTH_FSDP2_RUN_INSTALLS=1 / 0 force either way, and the single process path follows the same rules. - Refuse to start torchrun from each of several SLURM / MPI tasks. - Under IPython / Jupyter (%run) torchrun runs as a child process. - multiprocessing spawn / forkserver children re-import the script with get_ipython defined and without repeating its shell steps. - %cd prints the error and continues on a missing folder, like IPython. - The single-process note names what was requested, and a failing GPU count prints the child's error.
… job - Replacing or removing a read-only launcher clears the read-only bit and retries, as _rmtree_robust does. - Unchanged launchers still get their permissions reset. - The generate_fsdp2_scripts docstring says what a failure leaves behind. - Ignore the .fsdp2_scripts.staging-* folder. - The git diff summary counts only the root README.md as a README change. - tests/test_fsdp2_launch.py runs once in its own CPU torch job instead of in every api-drift matrix cell. - fsdp2_scripts/README.md describes the new install rules and variables.
…rors - The marker folder is created 0700 and refused when it is a symlink, owned by another uid, or group / world writable. Markers are written to an exclusive, no-follow temp file and then os.replace'd, and their JSON content is validated before it is believed. - Every shell / file step is a handshake: the other local ranks record their arrival with a hash of the unexpanded step, local rank 0 waits for all of them, then records its return code, output, install decision and any error. Ranks that reach different steps fail with a divergence error. - x = !cmd runs once per node under the same install rules and every rank gets local rank 0's output. Only local rank 0 decides whether an install runs; the others follow its record. - %%file errors on local rank 0 are raised on every rank. - Workers always use the adapter, even where an IPython get_ipython exists. - Log lines name a step by its unexpanded text with URL user info, index URLs, header values and token-like strings masked. - Shell steps refuse to run without bash, and on Windows refuse WSL's bash. - main() takes single_process / reason: such a script runs single process with a note, and more processes are refused unless UNSLOTH_FSDP2_FORCE=1. - --dry-run names the get_ipython adapter in use.
…k safe - FSDP2_SINGLE_PROCESS in update_all_notebooks.py names, by base name across the AMD- / Kaggle- / HuggingFace Course- variants, the scripts that start servers on fixed ports, clone or install through Python subprocess calls, or share temporary files between ranks, each with its reason. Their launchers pass single_process=True and say so in their header. - A static scan of python_scripts/ flags subprocess.Popen, os.system, vllm serve and --port outside get_ipython() calls in any script missing from the list, and stale entries in the list. - Launchers import os and sys on separate lines. - _assert_fsdp2_mirror also requires README.md. - The README documents shell step ordering, single-process scripts and UNSLOTH_FSDP2_FORCE.
|
@codex review |
|
@codex security review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b1e96d0059
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) | ||
| from unsloth_fsdp2_launch import main | ||
| if __name__ == "__main__": | ||
| main(__file__, 'Oute_TTS_(1B).py') |
There was a problem hiding this comment.
With two or more visible GPUs, this launcher runs the installation capture cell in python_scripts/Oute_TTS_(1B).py:38 on every rank. Although the adapter coordinates the git clone shell step, the subsequent three os.remove(...) calls execute independently on every rank against the same checkout. Once one rank deletes a file, another raises FileNotFoundError, aborting the distributed run before model loading. Add this script to the generator's single-process list, or coordinate those Python-side deletions as well.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Guarded in adb30ae. Oute_TTS_(1B) and its AMD- and Kaggle- copies are now in FSDP2_SINGLE_PROCESS. The reason given is that the install cell clones OuteTTS and then deletes files from that single checkout with os.remove. Their launchers run the script in one process and print that reason. More than one process is refused unless UNSLOTH_FSDP2_FORCE=1.
The static scan in tests/test_fsdp2_scripts_mirror.py now reads live %%capture bodies as well as top-level code. Besides servers and Python-side process launches, it flags os.remove, os.unlink, shutil.rmtree, shutil.move and os.rename. It found no other script that is not already on the list.
In the committed python_scripts/ the install cell of all three Oute_TTS mirrors is commented out, so the removes are not live there today. The list entry covers the cell if it becomes live again.
| path = re.split(r"[?#]", spec, 1)[0].rstrip("/") | ||
| last = path.rsplit("/", 1)[-1] |
There was a problem hiding this comment.
When the other dependencies are already installed, the smart install check can incorrectly skip a required triton_kernels installation. The live installation cells in scripts such as gpt_oss_(20B)_Reinforcement_Learning_2048_Game.py:27 request git+https://github.com/triton-lang/triton.git@...#subdirectory=python/triton_kernels, but this parser discards the fragment and identifies the distribution as triton. Consequently, an installed triton satisfies the check even when triton_kernels is absent, and the entire install command is skipped. Treat subdirectory URLs with unknown distribution metadata as uncheckable rather than assuming the repository basename is the package name.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in adb30ae. The install check no longer guesses a distribution from a repository URL.
- A requirement with
#subdirectory=cannot be checked, so the install runs. That covers the tritonpython/triton_kernelsline ingpt_oss_(20B)_Reinforcement_Learning_2048_Game.py. - Any other URL without an explicit name (
#egg=orname @ url) cannot be checked either, so the install runs. - Only the
unslothai/unslothandunslothai/unsloth-zoorepositories map to their names, and only without a subdirectory.
tests/test_fsdp2_launch.py::test_url_requirements_need_an_explicit_name covers these cases. It checks the exact triton URL, which a second test confirms is still in the script.
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
- Oute_TTS_(1B) and its AMD- / Kaggle- copies join FSDP2_SINGLE_PROCESS: its install cell clones OuteTTS and then deletes files from that one checkout with os.remove, which fails on the second rank. - The single-process scan now also reads live %%capture bodies and flags os.remove / os.unlink / shutil.rmtree / shutil.move / os.rename, besides servers and Python-side process launches. - The install check no longer guesses a distribution from a repository URL. A #subdirectory= requirement (for example triton's python/triton_kernels) or a URL without an explicit name (#egg= or name @ url) cannot be checked, so the install runs. Only the unslothai/unsloth and unsloth-zoo repos map to their names, and only without a subdirectory.
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
# Conflicts: # .github/workflows/notebooks-tests-ci.yml
|
I tested this PR on a machine with two GPUs (RTX 6000 Ada 48 GB and RTX 3090 24 GB). The training path works. I found a few problems in how the launcher handles one script and package installs, and fixed them in four commits. They are in a mirror PR on my fork, and they apply cleanly on top of this PR's current head ( What I ran on two GPUsEach run used the generated launcher in
The four commits
All tests pass on the branch, and the new tests fail on the previous launcher. For each case where the old code wrongly skipped an install, I reproduced it with real installs into throwaway environments and confirmed the package now installs in the right place. I did not test ROCm, Windows, multi-node, or a full GRPO run with its final save. |
|
FSDP2 results on two GPUs (RTX 6000 Ada 48 GB + RTX 3090 24 GB, connected over PCIe through the host bridge, no NVLink). Torch 2.12.1, unsloth 2026.10.10. LoRA r=16 on alpaca-cleaned, batch 2, sequence length 1024, global batch 8 in every run (1 GPU uses gradient accumulation 4, 2 GPUs use 2). All runs went through the launcher. The default layout copies the frozen base model to every GPU and shards only the LoRA weights. Unsloth picks it whenever the model fits on one GPU. The split layout ( Llama-3.2-3B-Instruct, 30 steps
In every run, both GPUs generated the same text after training and the LoRA adapter was saved. Qwen3-14B 16-bit (about 28 GB of weights), 10 steps
Notebook launcher
Summary
|
(cherry picked from commit 0faf739)
(cherry picked from commit 588e516)
…ON redirects (cherry picked from commit de9feb7)
(cherry picked from commit 1cee870)
…reter; document sharded load and generate memory The exported python / python3 / pip / pip3 / unsloth functions call $UNSLOTH_FSDP2_PYTHON. When that is a bash wrapper (sys.executable is the wrapper under `exec -a "$0"`, as in Nix-style wrappers) which runs `python3` or `pip` by name, the wrapper imports the exported functions and calls itself again without end. The pinned call now sets _UNSLOTH_FSDP2_PINNED, and below it the names resolve to PATH. _shell_env() clears the variable so a launcher started from another launcher's shell step pins again. README: document UNSLOTH_FSDP2_SHARDED_LOAD (read by Unsloth, 16-bit only) and that model.generate under native FSDP2 gathers the full model on every process, so it can run out of memory on a small GPU after training.
|
@oobabooga Thanks for testing this on two GPUs and for the four commits. I cherry-picked all of them onto the PR branch with your authorship kept ( What I checked before taking them:
One problem I found and fixed on top in 237e076: if A side effect worth knowing: The launcher and |
|
@oobabooga Thanks, these numbers are very useful, especially the Qwen3-14B run on the mixed 48 GB + 24 GB pair. I added both points to the README in 237e076. The template is in
I left the launchers unchanged: they still do not set |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 237e076bad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) | ||
| from unsloth_fsdp2_launch import main | ||
| if __name__ == "__main__": | ||
| main(__file__, 'Llasa_TTS_(1B).py') |
There was a problem hiding this comment.
With multiple visible GPUs, every rank runs preprocess_and_save in python_scripts/Llasa_TTS_(1B).py, opening the same processed_data_memmap/input_ids.memmap with mode w+ and overwriting input_ids_shape.npy after every example. These Python-side writes bypass the shell adapter's coordination. A rank finishing preprocessing can therefore load a shape file while another rank truncates and rewrites it, causing np.load to fail or giving ranks different dataset lengths before distributed training. The 3B script and AMD/Kaggle variants use the same implementation. Add both Llasa base names to the generator's single-process list until their preprocessing writes and subsequent reads are coordinated.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in ab83e62. Llasa_TTS_(1B) and Llasa_TTS_(3B) (and their AMD- and Kaggle- copies) are now in FSDP2_SINGLE_PROCESS, so their launchers run one process and refuse --nproc-per-node above 1 unless UNSLOTH_FSDP2_FORCE=1. The mirror test now flags np.memmap(..., mode = 'w+'), so a new script that writes a shared memmap from every rank has to be listed too (4 failures on the previous head, 210 / 210 passing now across the fsdp2 test files). --dry-run on the 1B launcher prints the single-process note with the reason.
The notebook builds its outer model with OmniVoice.from_pretrained and then calls model.merge_and_unload() and omnivoice.save_pretrained() directly, so the export never goes through Unsloth's collective FSDP2 path. On several ranks the merge fails with "aten.add_.Tensor got mixed torch.Tensor and DTensor" when the model fits one GPU, and the save fails with "invalid python storage" when it is sharded. Run it (and its Kaggle variant) in one process, and teach the mirror test to flag OmniVoice.from_pretrained so a new script with the same pattern must be listed.
|
Pushed 898a06c: Why: the notebook builds its outer model with
Training was numerically fine on every arm (loss 2.788172960281372 on all three). Only the merge and save fail. A collective export would mean changing the notebook's save cell and depending on an Unsloth export path that OmniVoice-built models do not use today. Single process is the same treatment Meta_Synthetic_Data_Llama3_2_(3B) got. Change:
Evidence:
|
preprocess_and_save opens processed_data_memmap/input_ids.memmap with mode w+ and rewrites input_ids_shape.npy after every example. On several ranks each process truncates and rewrites the same files while another may already be reading the shape back, so np.load can fail or ranks can see different dataset lengths. Run both scripts (and their AMD- and Kaggle- copies) in one process, and teach the mirror test to flag np.memmap opened for writing so a new script with the same pattern must be listed.
|
@codex review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 similar comment
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
On Windows os.execv starts a detached process, joins the arguments without quoting (a path with a space splits) and exits with code 0 at once, so the shell got its prompt back while torchrun ran on and never saw its exit code. Use the same subprocess path as the IPython case there.
The recursion guard read $_UNSLOTH_FSDP2_PINNED bare, so a !cmd or
%%bash step that turns on nounset (set -u, set -euo pipefail) and then
calls python, pip or unsloth stopped with 'unbound variable' (exit 127),
where the notebook itself runs fine. Read it as ${...:-}.
…ocess EmbeddingGemma2_(300M)-Audio, -Image_Text and -Multimodal_Search write their media files with open(path, "wb").write(...) and read them back for training or encoding. On several ranks each process truncates and rewrites the same paths, and Multimodal_Search's MSR-VTT loop skips a video that already exists, so a rank can decode a file another rank has created but not yet filled (zero bytes, "moov atom not found"). Run the three scripts and their Kaggle copies in one process, and teach the mirror test to flag open(..., "wb").write(...) so a new script with the same pattern must be listed.
_launcher_env_problem() ran only when two or more processes were about to be launched. In a SLURM step with one visible GPU per task (or with --nproc-per-node 1, no GPU, or a single-process script) every task fell through to a plain single-process run, so SLURM_STEP_NUM_TASKS=2 without torchrun rank variables quietly started independent training runs that write the same output paths. Check the scheduler environment once the process is known not to be a worker, before the single-process branch and the GPU-count choice. Workers with LOCAL_RANK and WORLD_SIZE / LOCAL_WORLD_SIZE still run.
A sweep of every python_scripts base for plain-Python writes that all ranks make to one path found two more groups: - Deepseek_OCR_(3B), Deepseek_OCR_(3B)-Eval and Deepseek_OCR_2_(3B) save a dataset image to your_image.jpg and pass that path to model.infer. One process can open the JPEG while another is truncating it, and the model's loader then fails with AttributeError. - OpenEnv_gpt_oss_(20B)_Reinforcement_Learning_2048_Game and its _BF16 copy start a uvicorn OpenEnv server through launch_openenv on a random port. The RNG is seeded the same on every rank, so all ranks pick the same port; the later ones fail to bind, pass the health check against the first rank's server and play its single game. Run them and their AMD- / Kaggle- copies in one process. The mirror test now flags launch_openenv( and PIL saves to a fixed image path, and a new AST check flags any literal path written from Python (open w/a/x, .save, sf.write, to_json, copy, ...) that the same run names again, leaving out Unsloth's collective saves and rewrites of the same path, so a new script that writes a file and reads it back must be listed.
install_decision() ignored every segment that was not an install, so a step such as "pip install x" followed by printf, git clone, wget, mkdir or python -c was skipped whole once x was installed, and its other commands never ran. Now a step is skipped only when the rest is read-only (apt's index refresh, `command -v`, an output filter like tail / grep, or ollama's own curl | sh); pip subcommands other than install / uninstall / list / show / freeze / check also keep it running.
Every process generates its own dataset through the OpenAI API and then writes it to raft_train.jsonl with Dataset.to_json. The exports can differ in length, so interleaved writes can leave bytes of a longer export after a shorter one and an invalid JSONL file. Run it (and its AMD- and Kaggle- copies) in one process, and teach the mirror test to flag to_json / to_csv to a fixed path.
…aunchers datasets' Dataset.map(num_proc = N) starts its workers through the multiprocess package, whose spawn module was not patched, so a spawned child re-imported the script without get_ipython and died with NameError. Patch multiprocess.spawn the same way as multiprocessing.spawn when it is installed. The node-local process count fell back to the global WORLD_SIZE when LOCAL_WORLD_SIZE was missing, so under DeepSpeed's launcher (LOCAL_SIZE) on two nodes with one GPU each the first shell step waited for a local rank that does not exist. Read LOCAL_SIZE and the Open MPI / MPICH local counts first.
… are here The install check parsed `pkg[extra]` down to `pkg`, so `pip install "requests[socks]"` was skipped whenever requests was installed, even without PySocks. Read each requested extra's dependencies from the installed distribution's metadata and skip only when every one of them is installed at a matching version; an extra the distribution does not declare, or metadata that cannot be read, keeps the step running.
…nd pip The pinned python / python3 / pip / pip3 / unsloth functions always ran the launcher's interpreter, also after a step activated another virtualenv or conda environment, so `source other/bin/activate; pip install ...` installed into the launcher's environment (Unsloth Studio's setup script activates its own venv this way). The functions now fall back to PATH once VIRTUAL_ENV or CONDA_PREFIX differs from the launcher's, in the step and in any bash it starts.
…e process Under native FSDP2 model.generate runs inside native_fsdp2_unsharded, which all-gathers on every rank, so every rank has to make the same number of generate calls. A sweep of every python_scripts base for loops and retries whose generate count follows each rank's own output found four scripts that break that: - gpt_oss_(20B)_Reinforcement_Learning_GRPO_Minesweeper_Game_BF16: the eval callback and play_full_game sample a move per turn and stop when a rank's move does not parse or its game ends. GRPO seeds per device, so ranks stop after different numbers of calls. - Spark_TTS_(0_5B): samples again after a codec shape error, which can hit one rank only. - Openenv_wordle_grpo: play_wordle and the training rollouts stop when each rank's own Wordle game is done. - Qwen3_(4B)-Tool_Calling and Qwen3_(4B)-GRPO-Tool_Use: run_tools samples a reply and returns once a rank's reply has no tool call. Sesame_CSM_(1B)-TTS and Spark_TTS_(0_5B) also write each rank's own sampled speech to the same WAV files with sf.write, which can mix samples from several ranks. Run them and their AMD- / Kaggle- / HuggingFace Course- copies in one process. The mirror test gains two AST checks: one flags audio / image writes to a path the script spells out (literal, f-string or a name bound to one; in-memory buffers are left out), and one flags a generate, fast_generate, infer or generate_rollout_completions call, followed through the script's own helpers, inside a while loop, a loop with a break / continue / return, or a try whose handler generates again. A for loop over a fixed list with no early exit stays allowed. Each check has its own self-test.
Summary
Adds a generated
fsdp2_scripts/folder: one launcher perpython_scripts/<name>.py(467 today, AMD-, Kaggle- and HuggingFace Course- copies included) that runs the same script on every visible GPU with Unsloth's native FSDP2:Until now multi-GPU meant remembering
UNSLOTH_FSDP2=1 torchrun --standalone --nproc-per-node N "python_scripts/<name>.py", and 83 of the 467 mirrors still carry liveget_ipython()calls (!wget,%%capture,%%bash,%%file,%env,%cd), so they stop withNameError: name 'get_ipython' is not definedunder plainpythonortorchrun.How it works
scripts/fsdp2_launch.py(stdlib only, hand maintained) is the single implementation. The generator copies it tofsdp2_scripts/unsloth_fsdp2_launch.pyand writes one ~12 line launcher per script that callsmain(__file__, "<name>.py").python -m torch.distributed.run --standalone --nproc-per-node=N <launcher> [args]andUNSLOTH_FSDP2=1(unless the caller set it;UNSLOTH_FSDP2=0gives plain DDP). An argv list, no shell, so names with spaces and parentheses are safe. More processes than visible GPUs is refused; 0 or 1 GPU runs the script single process.torchrun) it runspython_scripts/<name>.pyunchanged throughrunpyas__main__, with the working directory,sys.argv[0]andsys.path[0]as forpython "python_scripts/<name>.py".get_ipythonstand-in covers exactly the calls the mirrors use (system,x = !cmd,%%capture,%%bash,%%file,%env,%cd). Shell commands, file writes and!cmdcaptures run once per node on local rank 0 after every local rank has reached the same step; the others then read its return code, output or error, and a rank that reached a different command fails with a clear divergence error instead of pairing the wrong steps.%env/%cdapply on every rank. Unknown magics raise instead of being skipped. Step markers live in a private (0700, owner checked) per-launch directory, so they are never reused across runs or writable by other users.--python,--target,--prefix,uv venv) always run.UNSLOTH_FSDP2_RUN_INSTALLS=1always runs them,0always skips them. Diagnostics print the command template with URLs, tokens and index / header values redacted.FSDP2_SINGLE_PROCESSinupdate_all_notebooks.pywith a one-line reason each: NeMo-Gym, Meta-Synthetic-Data, Synthetic_Data_Hackathon, Deepseek_OCR evaluation, Llama3 Ollama and Gemma3N inference (17 launchers with their AMD- / Kaggle- copies). Their launchers run single process with that reason printed;UNSLOTH_FSDP2_FORCE=1overrides. A scan test flags any new mirror that starts a server or shells out from Python without being on the list.LOCAL_RANKin the shell no longer hangs a plain run, SLURM / MPI multi-task steps are refused instead of nesting torchrun groups,%runfrom IPython uses a subprocess instead of replacing the kernel, spawn / forkserver children re-run the target with the stand-in, and marker waits have a bounded timeout (UNSLOTH_FSDP2_WAIT_TIMEOUT).push_to_hub, generation and prints are left alone: Unsloth's own collective saves pick the writer rank, and a rank guard around them would hang the collective.Generator
update_all_notebooks.py:generate_fsdp2_scripts()runs right after bothconvert_folder(..., "python_scripts", ...)calls (full pass, and the AMD-only pass scoped toAMD-), under the same--disable_convert_to_scriptswitch. It stages into a temp dir, writes only changed files, removes stale launchers in scope, and_assert_fsdp2_mirror()checks the name sets in both directions.fsdp2_scripts/README.mdis generated too (usage, env vars, effective batch size, install skipping).Tests (all wired into
notebooks-tests-ci.yml)tests/test_fsdp2_scripts_mirror.py(47): name sets equal both ways, each launcher equals the generator's render, helper copy equals its source byte for byte, idempotent regeneration, stale cleanup, AMD-only scope preserved.tests/test_fsdp2_launch.py(CPU torch, 104 tests, own CI job): command construction, flag precedence, refusal above the visible GPU count, no second launch inside torchrun, 0 / 1 GPU single process,UNSLOTH_FSDP2default and explicit 0,--dry-run, plus real two-processtorch.distributed.runruns on CPU checking that every shell step ran exactly once, both ranks saw the file / env var / cwd, return codes reach both ranks, unknown magics raise, installs are skipped by default and run once with the env var, and a failing rank fails the launch.tests/test_fsdp2_adapter_covers_scripts.py(no torch): every liveget_ipython()call inpython_scripts/, including inside the 30 live%%capturecells, maps to a supported form, so a new notebook with a new magic fails here instead of at run time.scripts/lint_exec_literals.pypasses; the stand-in's oneeval(IPython style{expr}expansion) and oneexec(a%%capturecell body) are recorded in the baseline with reasons.Not covered yet
torchrunon the same scripts) is still to do; this is a draft until it lands.%%capturecell live inpython_scripts/becauseremove_unwanted_sectionneeds a later### Unslothheading. The launcher only runs those installs when something is missing; fixing the mirrors themselves is a separate change.