Skip to content

[Bug] CUDA sparse-conv backend fails to compile with CUDA 12.8 / torch 2.8 / GCC 11+ #638

Description

@WhaleK1123

[Bug] CUDA sparse-conv backend fails to compile with CUDA 12.8 / torch 2.8 / GCC 11+

Summary

MinkowskiEngine master (0.5.4, commit 02fc608) does not compile its CUDA
backend on a modern toolchain: CUDA 12.8 (thrust/CCCL), torch 2.8, GCC 11 or 13.
A set of small compatibility fixes is available (see below) that lets it build and
run the sparse-conv backend on an RTX 5070 Ti (sm_120).

Environment

  • OS: Ubuntu 22.04
  • Host compiler: GCC 11.4 (also reproduced on GCC 13)
  • CUDA: 12.8 (nvcc 12.8.93, supports compute_120)
  • PyTorch: 2.8.0+cu128
  • Build: SETUPTOOLS_USE_DISTUTILS=stdlib, TORCH_CUDA_ARCH_LIST="12.0",
    CC=gcc-13 CXX=g++-13

Steps to reproduce

git clone https://github.com/NVIDIA/MinkowskiEngine.git
cd MinkowskiEngine
pip install torch==2.8.0+cu128 numpy<2 setuptools wheel
export CUDA_HOME=/usr/local/cuda-12.8 TORCH_CUDA_ARCH_LIST="12.0" SETUPTOOLS_USE_DISTUTILS=stdlib
export CC=gcc-13 CXX=g++-13
python setup.py build_ext --inplace

Errors encountered (in order after each previous fix)

  1. namespace "thrust" has no member "sort_by_key" (also unique, sequence,
    reduce, etc.) in src/spmm.cu, src/coordinate_map_gpu.cu, pooling kernels.
    New thrust in CUDA 12.8 requires explicit includes; MinkowskiEngine relied
    on the old transitive includes.

  2. identifier "CUDF_FUNC_RANGE" is undefined in
    src/3rdparty/concurrent_unordered_map.cuh. The bundled cudf nvtx macro was
    disabled when its headers are removed.

  3. std::__to_address overload ambiguity in <bits/shared_ptr_base.h>
    (both GCC 11 and GCC 13), triggered by
    concurrent_unordered_map::create() returning a std::unique_ptr that is
    converted to std::shared_ptr. This is the hard blocker.

Root causes

  • MinkowskiEngine (2021) predates the CUDA 12.x thrust/CCCL change that made
    algorithm headers non-transitive.
  • The bundled nvtx3.hpp/ranges.hpp collide with torch's nvtx3.
  • default_allocator / c10_allocator were only minimally declared (no
    pointer/rebind/operator== etc.), which modern libstdc++ shared_ptr
    fails on.
  • concurrent_unordered_map::create() returns unique_ptr; the
    unique_ptr -> shared_ptr conversion hits a libstdc++ std::__to_address
    overload-ambiguity bug on both GCC 11 and 13.

Proposed fix (verified, builds on RTX 5070 Ti)

  • Add explicit thrust includes (<thrust/sort.h>, <thrust/unique.h>,
    <thrust/sequence.h>, <thrust/reduce.h>, <thrust/remove.h>,
    <thrust/count.h>, <thrust/copy.h>, <thrust/fill.h>,
    <thrust/device_vector.h>, <thrust/iterator/zip_iterator.h>,
    <thrust/tuple.h>, <thrust/execution_policy.h>).
  • No-op the bundled nvtx3 headers and add #define CUDF_FUNC_RANGE() ((void)0).
  • Make both allocators fully conformant (pointer, const_pointer, size_type,
    difference_type, rebind, operator==/!=).
  • Change concurrent_unordered_map::create() to return std::shared_ptr<Self>
    (pointer + deleter) instead of std::unique_ptr.

After the fix, a real sparse convolution runs on GPU:

>>> MinkowskiEngine 0.5.4 GPU sparse conv LIVE on RTX 5070 Ti (CUDA 12.8, sm_120)
>>> is_cuda_available: True, cuda_version: 12080

A ready-made patch is in the fix/build-cuda12.8-gcc13 branch (commit 6fd7c64)
of my fork. I verified it compiles and runs on CUDA 12.8 / torch 2.8 / sm_120.

Notes

  • This affects downstream users: FUSER (Jiang-HB/FUSER) pins
    MinkowskiEngine==0.5.4 and, on a Blackwell GPU, needs torch>=2.7 — which
    then requires this compatibility patch to compile. See the related
    FUSER requirements.txt issue #1.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions