Skip to content

[CI] RAPIDS XGBoost packages block 26.12 conda environments #8583

Closed

Description

@csadorf

Summary

Python conda jobs fail during environment creation because strict channel priority selects RAPIDS’ xgboost 3.3.0 packages, whose librmm constraints are incompatible with libnvforest 26.12.

Failing test/component: Conda test environment creation

Failure observed in:

Environment

  • OS: Rocky Linux 8; Ubuntu 22.04, 24.04, and 26.04
  • Python version: 3.12, 3.13, and 3.14
  • CUDA version: 12.2.2, 12.9.2, 13.0.3, and 13.3.0
  • GPU: L4, H100, and RTX PRO 6000
  • Dependencies: latest-deps; nightly 26.12 conda packages; strict channel priority

Test Details

  • Test file: N/A (environment creation fails before tests run)
  • Test name: conda-python-* and conda-notebook-tests
  • Error message:
    error    libmamba Could not solve for environment specs
    
    libcuml 26.12 requires libnvforest 26.12
    libnvforest 26.12 requires librmm 26.12.*
    
    xgboost 3.3.0 requires libxgboost 3.3.0
    libxgboost 3.3.0 requires librmm >=26.8,<26.9
    or librmm >=26.10,<26.11
    

Root Cause Analysis

cuML requests xgboost>=3.3.0. With rapidsai-nightly and rapidsai ahead of conda-forge under strict channel priority, conda selects RAPIDS’ XGBoost packages.

The available RAPIDS libxgboost 3.3.0 builds require an older RMM release:

  • stable builds require librmm >=26.8,<26.9
  • nightly builds require librmm >=26.10,<26.11

The same environment requires libnvforest 26.12, which requires librmm 26.12.*. No librmm version satisfies both constraints.

The published CUDA constraints for the 26.12 libnvforest and librmm packages are correct. The CUDA conflicts shown by libmamba are consequences of exploring incompatible package variants.

Removing or relabeling RAPIDS’ XGBoost packages would allow strict channel priority to resolve standard XGBoost from conda-forge. That migration is tracked in:

rapidsai/build-planning#312

Activity

  1. added theissue type on Sep 9, 2026
  2. changed the title [-][CI] 26.12 conda test environments are unsatisfiable[/-] [+][CI] RAPIDS XGBoost packages block 26.12 conda environments[/+] on Sep 9, 2026
  3. jcrist commented on Sep 9, 2026

    @jcrist
    Member

    This same issue is also affecting our 26.10 builds, see https://github.com/NVIDIA/cuml/actions/runs/34382331536/job/102574688739?pr=8575

  4. jakirkham commented on Sep 9, 2026

    @jakirkham
    Contributor

    @gforsyth submitted this fix an hour ago: #8596

    Also XGBoost packages nightly have been removed, which ensure they are not being picked up.

    This should fix the CI issues currently.


    Additionally we need to stop publishing RAPIDS XGBoost packages from the integration repo. These are currently being published nightly. I have this PR open to remove them: rapidsai/integration#868

    This should provided a persistent fix for CI issues.

  5. chyunsu3 commented on Sep 18, 2026

    @chyunsu3
    Contributor

    Resolved in #8658

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingci

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions