Repository navigation
[Bug]: [NPU][ACL Graph] A3(910C) GLM-5.2-w8a8 fails to launch with enable_graph=true #2284
Description
Your environment
Hardware: A3 with ARM
xLLM version: master
docker: quay.io/jd_xllm/xllm-ai:xllm-dev-a3-arm-cann9-20260605
startup parameters:
--max_memory_utilization=0.9
--max_tokens_per_batch=4096
--max_seqs_per_batch=16
--block_size=128
--enable_prefix_cache=true
--enable_chunked_prefill=true
--communication_backend="hccl"
--enable_schedule_overlap=true
--enable_graph=true
--ep_size=16
--dp_size=2
--nnodes=16
🐛 Describe the bug
ATB graph warmup crashes at capture_end with error code 107025 ("capture model contains a stream that was not joined to the original stream") on multi-die 910C. The same code, torch_npu, ATB, driver, image, and config run fine on single-die 910C.
The crash is on the first warmup bucket (bucket=1), ~1 second after warmup start. Forward completes successfully; the failure is at capture_end → aclmdlRICaptureEnd. The cleanup capture_end then fails with 107029 ("stream is not in capture status").
frame #12 of the crash stack lands in libhccl.so — the unjoined stream is created by HCCL during the MoE dispatch/combine collectives inside the captured forward, and is not joined back to the capture stream before capture_end.
Always happens on A3(910C).
Log from rank-0
I20260821 17:32:25.836668 84235 profile_manager.cpp:1283] Graph warmup started: bucket_count=1, configured_max_batch_size=2, allocatable_sequences=2, decode_seq_len=16
I20260821 17:32:25.937453 84730 acl_graph_executor_impl.cpp:384] [DIAG-107025] before capture: bucket=1 current_stream=0x564ec800 default_stream=0x564ec800 capture_stream=0x5645f200 graph_stream_=0x564ec800 current==default=1 current==capture=0
I20260821 17:32:25.937512 84730 acl_graph_executor_impl.cpp:408] [DIAG-107025] capture stream resolved: bucket=1 entered_switch_branch=1 graph_stream_=0x5645f200 capture_stream=0x5645f200 need_restore_stream=1
I20260821 17:32:25.937529 84730 acl_graph_executor_impl.cpp:422] [DIAG-107025] before capture_begin: bucket=1 current_stream=0x5645f200 graph_stream_=0x5645f200
I20260821 17:32:25.943053 84730 acl_graph_executor_impl.cpp:431] [DIAG-107025] after capture_begin (in capture): bucket=1 current_stream=0x5645f200 graph_stream_=0x5645f200
I20260821 17:32:26.286935 84730 acl_graph_executor_impl.cpp:442] [DIAG-107025] after forward (before capture_end): bucket=1 current_stream=0x5645f200 graph_stream_=0x5645f200 current==graph_stream=1
E20260821 17:32:26.321024 84730 acl_graph_executor_impl.cpp:464] ACL graph capture_end during cleanup failed: capture_end:torch_npu/csrc/core/npu/NPUGraph.cpp:247 NPU function error: c10_npu::acl::AclmdlRICaptureEnd(capture_stream_, &model_ri), error code is 107029
[ERROR] 2026-08-21-17:32:26 (PID:84235, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EE9999: Inner Error!
EE9999[PID: 84235] 2026-08-21-17:32:26.320.711 (EE9999): rtStreamEndCapture execution failed, reason=stream is not in capture status[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
TraceBack (most recent call last):
end capture stream failed, runtime result = 107029[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
Exception raised from capture_end at torch_npu/csrc/core/npu/NPUGraph.cpp:247 (most recent call first):
frame #0: c10::Error::Error(...) + 0xb0 (0xffffa942c700 in /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(...) + 0x68 (0xffffa93ca860 in /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10_npu::NPUGraph::capture_end() + 0xbec (0xffffb7e899ec in /usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x149f64c]
frame #4: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x14a3798]
frame #5: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x148d670]
frame #6: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13daaf4]
frame #7: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13dbff8]
frame #8: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13d9804]
frame #9: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13bd118]
frame #10: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0xe6dd58]
frame #11: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0xe6ea50]
frame #12: <unknown function> + 0x289da4 (0xffffb5e39da4 in /usr/local/Ascend/ascend-toolkit/latest/lib64/libhccl.so)
frame #13: <unknown function> + 0x7fbb4 (0xffffa65afbb4 in /lib64/libc.so.6)
frame #14: <unknown function> + 0xe7f4c (0xffffa6617f4c in /lib64/libc.so.6)
terminate called after throwing an instance of 'c10::Error'
what(): capture_end:torch_npu/csrc/core/npu/NPUGraph.cpp:247 NPU function error: c10_npu::acl::AclmdlRICaptureEnd(capture_stream_, &model_ri), error code is 107025
[ERROR] 2026-08-21-17:32:26 (PID:84235, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EE9999: Inner Error!
EE9999[PID: 84235] 2026-08-21-17:32:26.320.038 (EE9999): rtStreamEndCapture execution failed, reason=capture model contains a stream that was not joined to the original stream[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
TraceBack (most recent call last):
end capture stream failed, runtime result = 107025[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
Exception raised from capture_end at torch_npu/csrc/core/npu/NPUGraph.cpp:247 (most recent call first):
frame #0: c10::Error::Error(...) + 0xb0 (0xffffa942c700 in /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(...) + 0x68 (0xffffa93ca860 in /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10_npu::NPUGraph::capture_end() + 0xbec (0xffffb7e899ec in /usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x149edb8]
frame #4: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x14a3798]
frame #5: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x148d670]
frame #6: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13daaf4]
frame #7: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13dbff8]
frame #8: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13d9804]
frame #9: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13bd118]
frame #10: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0xe6dd58]
frame #11: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0xe6ea50]
frame #12: <unknown function> + 0x289da4 (0xffffb5e39da4 in /usr/local/Ascend/ascend-toolkit/latest/lib64/libhccl.so)
frame #13: <unknown function> + 0x7fbb4 (0xffffa65afbb4 in /lib64/libc.so.6)
frame #14: <unknown function> + 0xe7f4c (0xffffa6617f4c in /lib64/libc.so.6)
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
910B (A2) with the identical config and software stack does NOT reproduce.
910C (A3) Always happens.
Activity
[-][Bug]: A3(910C) GLM-5.2-w8a8 fails to launch with enable_graph=true[/-][+][Bug]: [NPU][ACL Graph] A3(910C) GLM-5.2-w8a8 fails to launch with enable_graph=true[/+]XuZhang99 commentedon Aug 27, 2026
add
export HCCL_OP_EXPANSION_MODE="AIV"when launching xllm server.banxiaduhuo commentedon Aug 27, 2026
I tried HCCL_OP_EXPANSION_MODE=AIV — it did not help. It does not change how HCCL allocates/joins its internal communication streams during graph capture, which is where the crash occurs.
I also noticed vllm-ascend has reported a similar 107025 issue: vllm-project/vllm-ascend#13792
bluryar commentedon Sep 10, 2026
same issue on GLM5.3 W8A8
XuZhang99 commentedon Sep 10, 2026
change
nnodesto 16 for 8xA3 machine and use the latest main branch.XuZhang99 commentedon Sep 11, 2026
for glm5.2 and glm5.3 model, we recommend to use PyTorch-based implementation.
# from source python setup.py installand then start xllm server add:
banxiaduhuo commentedon Sep 18, 2026
Thanks for the suggestion — tried --model_impl=python, it works.