Fix orthogonal_ init for low-precision dtypes (bf16/fp16)

[janbernloehr]

What does this PR do?

torch.nn.init.orthogonal_ relies on torch.linalg.qr, whose geqrf kernel is only implemented for float32/float64 — on both CUDA ("geqrf_cuda" not implemented for 'BFloat16') and CPU ("geqrf_cpu" not implemented for 'BFloat16'). Since v5 materializes weights in the requested dtype before _init_weights runs, constructing any model that uses orthogonal init (e.g. RWKV) with dtype=torch.bfloat16/float16 crashes:

with torch.device("cuda"):
    model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16)  # NotImplementedError

This PR makes the orthogonal_ wrapper in src/transformers/initialization.py run the QR-based init in a temporary float32 tensor (same device) and copy the result back when the target dtype is a floating dtype below 32 bits. Full-precision behavior is unchanged, and the _is_hf_initialized guard is preserved. Since guard_torch_init_functions patches the torch modules with these wrappers, remote-code models calling torch.nn.init.orthogonal_ directly are covered as well.

Added tests/utils/test_initialization.py: verifies bf16/fp16 init succeeds and produces finite, orthogonal rows scaled by gain (fails with NotImplementedError without the fix), that the fp32 path exactly matches torch.nn.init.orthogonal_, and that _is_hf_initialized is respected. Also validated end-to-end: RWKV from_config in bf16 now constructs on CPU and CUDA (H100), and trains with decreasing loss (see issue).

Fixes #47225

Disclosure: this fix was implemented with the assistance of a code agent (Claude), per my proposal in the linked issue, and reviewed and validated by me.

Before submitting

• Did you read the contributor guideline and the Pull Request checks?
• Was this discussed/approved via a Github issue? orthogonal_ weight init crashes with geqrf_cuda not implemented for BFloat16 under v5 bf16 on-device init (RWKV and any QR-based init) #47225
• Did you write any new necessary tests?

Who can review?

@Cyrilvallez (author of the v5 weight-initialization refactor in #42191)

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 29099244966
Result: success | Grafana metrics are not available yet.

[Rocketknight1]

LGTM! Can we drop the new test file, though, and instead just add a small test into an existing file? With the comment this is unlikely to regress too, so even dropping the test entirely is fine I think.

[github-actions]

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

• The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
• A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
• A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.