MiniMax-H3: `references` argument ignored and documented multi-GPU example fails with CUDA device mismatch

[jyx13121802323]

Describe the bug

Environment

• OS: CentOS 7
• GPUs: 8 × NVIDIA A100 80GB
• Python: 3.11
• PyTorch: 2.6.0+cu124
• CUDA Driver: 550.xx (CUDA 12.4)
• Diffusers branches tested:
  • minimax-h3
  • minimax-h3-refactor

The same issues occur on both branches.

---

Description

I tried to reproduce the MiniMax-H3 Reference-to-Video example from the documentation:

https://github.com/huggingface/diffusers/blob/minimax-h3/docs/source/en/api/pipelines/minimax_h3.md

I encountered two independent issues.

---

Issue 1: references argument is ignored

Calling the pipeline with

state = pipe(
    prompt="...",
    references=[
        MiniMaxH3Reference(image=subject),
    ],
)

produces

UserWarning:
Unexpected input 'dict_keys(['references'])' provided.
This input will be ignored.

As a result, all reference inputs are ignored by the pipeline.

This happens on both minimax-h3 and minimax-h3-refactor.

---

Issue 2: Official multi-GPU example fails with device mismatch

Following the official documentation, I place the Qwen3-VL text encoder on GPU1 while keeping the transformer on GPU0.

import torch

from diffusers import ComponentsManager, ModularPipeline
from transformers import Qwen3VLForConditionalGeneration

manager = ComponentsManager()

pipe = ModularPipeline.from_pretrained(
    "MiniMaxAI/MiniMax-H3",
    components_manager=manager,
)

text_encoder = Qwen3VLForConditionalGeneration.from_pretrained(
    "MiniMaxAI/MiniMax-H3",
    subfolder="text_encoder",
    dtype=torch.bfloat16,
    device_map={"": "cuda:1"},
)

pipe.update_components(
    text_encoder=text_encoder,
)

pipe.load_components(dtype=torch.bfloat16)

pipe.transformer.to("cuda:0")
pipe.vae.to("cuda:0")
pipe.audio_vae.to("cuda:0")

The pipeline initializes successfully, but fails during denoising with

RuntimeError:
Expected all tensors to be on the same device,
but found at least two devices, cuda:1 and cuda:0

The traceback points to

transformer_minimax_h3.py

rotary_emb = self.rope(position_ids)

freqs = position_ids.unsqueeze(-1) * self.inv_freq.view(...)

Apparently,

• position_ids is located on cuda:1
• self.inv_freq is located on cuda:0

No automatic activation transfer happens between the text encoder and the transformer.

The full traceback ends with

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

---

Expected behavior

According to the documentation,

1. references should be accepted by the pipeline.
2. The documented two-GPU loading strategy should work without additional user modifications.
3. Intermediate tensors produced by the text encoder should be transferred automatically (or otherwise handled) before entering the transformer.

---

Questions

1. Is the current documentation ahead of the implementation?
2. Is the references argument currently supported in ModularPipeline?
3. Is the documented multi-GPU example expected to work, or is this feature still under development?
4. Is there an officially recommended way to run MiniMax-H3 on two 80GB GPUs without CPU offloading?

Thanks for the great work on MiniMax-H3 support. I'd be happy to test any proposed fixes or provide additional logs if needed.

Reproduction

import torch

from diffusers import ComponentsManager, ModularPipeline
from transformers import Qwen3VLForConditionalGeneration

from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3Reference
from diffusers.utils import load_image, load_video
from diffusers.utils.export_utils import encode_video

-------------------------

Pipeline

-------------------------

manager = ComponentsManager()

pipe = ModularPipeline.from_pretrained(
"MiniMaxAI/MiniMax-H3",
components_manager=manager,
)

print("Loading Qwen3-VL on GPU1...")

text_encoder = Qwen3VLForConditionalGeneration.from_pretrained(
"MiniMaxAI/MiniMax-H3",
subfolder="text_encoder",
dtype=torch.bfloat16,
#device_map={"": "cuda:1"},
)

pipe.load_components(dtype=torch.bfloat16)

manager.enable_auto_cpu_offload(
device="cuda",
memory_reserve_margin="12GB",
)

pipe.update_components(
text_encoder=text_encoder,
)

print("Loading remaining components...")

pipe.load_components(dtype=torch.bfloat16)

print("Moving transformer to GPU0...")

pipe.transformer.to("cuda:0")

pipe.vae.to("cuda:0")

pipe.audio_vae.to("cuda:0")

print("Pipeline Ready.")

subject = load_image(
"extracted_images/img_row8_col3_12.png"
)

product = load_image(
"extracted_images/img_row23_col3_48.png"
)

state = pipe(
prompt="subject_definitions: <Subject 1> is the elderly female host in <Picture 1>, sitting in a cozy streaming setup with a dignified and wise appearance. <Subject 2> is the health supplement in <Picture 2>, a premium wellness product with trustworthy packaging and clear health-focused branding. summary: [reference generation] The target video shows <Subject 1> at her streaming desk in a fixed camera livestream, carefully picking up and presenting <Subject 2> to the audience with a caring and authoritative manner, emphasizing the product's health benefits. retention_analysis: <Subject 1>: fully_preserved - the host's dignified elderly appearance and wise demeanor are retained throughout. <Subject 2>: fully_preserved - the health product's trustworthy packaging and branding are retained. detailed_description: The target video is a fixed-camera livestream shot in a warm and authoritative health style with soft, warm lighting and a comfortable, home-like background. The camera remains stationary throughout. A medium shot shows <Subject 1>, the elderly woman, sitting at her streaming desk with a composed and dignified expression. She carefully picks up <Subject 2> from the display area. She holds the product steadily at a clear viewing height, presenting its label side to the camera. Her hands cradle the product with care, and her expression conveys deep personal endorsement. She nods gently, reinforcing the trustworthiness of her recommendation. overall_soundscape: Warm, quiet room tone with a gentle, calming atmosphere. non_diegetic_music: N/A",
references=[
MiniMaxH3Reference(image=subject),
MiniMaxH3Reference(image=product),
],
num_frames=124,
)

encode_video(
state["videos"][0],
fps=24,
output_path="result.mp4",
audio=state["audio"][0],
audio_sample_rate=state["sampling_rate"],
)

Logs

System Info

• 🤗 Diffusers version: 0.40.0.dev0
• Platform: Linux-4.18.0-2.6.8.kwai.x86_64-x86_64-with-glibc2.17
• Running on Google Colab?: No
• Python version: 3.11.0
• PyTorch version (GPU?): 2.6.0+cu124 (True)
• Huggingface_hub version: 1.26.0
• Transformers version: 5.14.1
• Accelerate version: 1.14.0
• PEFT version: not installed
• Safetensors version: 0.8.0
• xFormers version: not installed
• Accelerator: NVIDIA A100-SXM4-80GB, 81920 MiB
  NVIDIA A100-SXM4-80GB, 81920 MiB
  NVIDIA A100-SXM4-80GB, 81920 MiB
  NVIDIA A100-SXM4-80GB, 81920 MiB
  NVIDIA A100-SXM4-80GB, 81920 MiB
  NVIDIA A100-SXM4-80GB, 81920 MiB
  NVIDIA A100-SXM4-80GB, 81920 MiB
  NVIDIA A100-SXM4-80GB, 81920 MiB
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: yes

Who can help?

No response