Decord causes segfaults on teardown #1432

python3 -m lmms_eval --model vllm --model_args model=google/gemma-3n-E2B-it,tensor_parallel_size=1,pipeline_parallel_size=1,dtype=bfloat16,trust_remote_code=True,max_model_len=8192,gpu_memory_utilization=0.85 --tasks pope --batch_size 16 --limit 2000 --gen_kwargs temperature=0 --output_path ./results

INFO 07-31 16:31:50 [api_utils.py:273] non-default args: {'trust_remote_code': True, 'dtype': 'bfloat16', 'seed': 1, 'max_model_len': 8192, 'gpu_memory_utilization': 0.85, 'model': 'google/gemma-3n-E2B-it'}
INFO 07-31 16:31:53 [model.py:638] Resolved architecture: Gemma3nForConditionalGeneration
INFO 07-31 16:31:53 [model.py:1875] Using max model len 8192
[transformers] `SiglipImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `SiglipImageProcessor` instead.
INFO 07-31 16:31:56 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 07-31 16:31:56 [vllm.py:1145] Asynchronous scheduling is enabled.
INFO 07-31 16:31:56 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
WARNING 07-31 16:31:56 [xpu.py:298] XPU Graph is disabled by environment variable, please set VLLM_XPU_ENABLE_XPU_GRAPH=1 to enable it.
INFO 07-31 16:31:56 [xpu.py:358] XPU platform: set server shutdown_timeout=5.
(EngineCore pid=1868) INFO 07-31 16:32:31 [core.py:121] Initializing a V1 LLM engine (v0.26.1rc1.dev181+g0f1739456) with config: model='google/gemma-3n-E2B-it', speculative_config=None, tokenizer='google/gemma-3n-E2B-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=xpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=1, served_model_name=google/gemma-3n-E2B-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=1868) [transformers] `SiglipImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `SiglipImageProcessor` instead.
(EngineCore pid=1868) INFO 07-31 16:32:35 [parallel_state.py:1640] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.4:43969 backend=xccl
(EngineCore pid=1868) INFO 07-31 16:32:35 [parallel_state.py:1977] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
2026:07:31-16:32:35: 1868 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
2026:07:31-16:32:35: 1868 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
(EngineCore pid=1868) INFO 07-31 16:32:36 [xpu_worker.py:108] Using V2 Model Runner
(EngineCore pid=1868) INFO 07-31 16:32:37 [model_runner.py:307] Loading model from scratch...
(EngineCore pid=1868) INFO 07-31 16:32:37 [vllm.py:1145] Asynchronous scheduling is enabled.
(EngineCore pid=1868) INFO 07-31 16:32:37 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=1868) WARNING 07-31 16:32:37 [xpu.py:298] XPU Graph is disabled by environment variable, please set VLLM_XPU_ENABLE_XPU_GRAPH=1 to enable it.
(EngineCore pid=1868) INFO 07-31 16:32:37 [xpu.py:194] Using Flash Attention backend.
(EngineCore pid=1868) INFO 07-31 16:32:37 [flash_attn.py:789] Using FlashAttention version 2
(EngineCore pid=1868) INFO 07-31 16:32:38 [weight_utils.py:867] Filesystem type for checkpoints: EXT4. Checkpoint size: 10.13 GiB. Available RAM: 962.04 GiB.
(EngineCore pid=1868) INFO 07-31 16:32:38 [weight_utils.py:890] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=1868)
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
(EngineCore pid=1868)
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:00<00:01,  1.19it/s]
(EngineCore pid=1868)
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:02<00:01,  1.45s/it]
(EngineCore pid=1868)
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:03<00:00,  1.08s/it]
(EngineCore pid=1868)
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:03<00:00,  1.12s/it]
(EngineCore pid=1868)
(EngineCore pid=1868) INFO 07-31 16:32:42 [default_loader.py:430] Loading weights took 3.37 seconds
(EngineCore pid=1868) INFO 07-31 16:32:42 [model_runner.py:328] Model loading took 10.28 GiB and 6.009605 seconds
(EngineCore pid=1868) INFO 07-31 16:32:42 [interface.py:635] Setting kv cache block size to 64 for FLASH_ATTN backend.
(EngineCore pid=1868) INFO 07-31 16:32:54 [encoder_runner.py:80] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 32 image items of the maximum feature size.
(EngineCore pid=1868) INFO 07-31 16:33:06 [backends.py:1094] Using cache directory: /root/.cache/vllm/torch_compile_cache/00e927453a/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=1868) INFO 07-31 16:33:06 [backends.py:1155] Dynamo bytecode transform time: 8.81 s
(EngineCore pid=1868) [rank0]:W0731 16:33:07.799000 1868 torch/_inductor/utils.py:3146] get_device_tflops: no Triton fallback available for non-CUDA devices. Returning 0.0; roofline estimates will use memory bandwidth only.
(EngineCore pid=1868) INFO 07-31 16:35:01 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 114.32 s
(EngineCore pid=1868) INFO 07-31 16:35:06 [backends.py:920] collected artifacts: 31 entries, 31 artifacts, 182288683 bytes total
(EngineCore pid=1868) INFO 07-31 16:35:06 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/2aec16846b1b4cb88def68aa4de930fbc01712a11279cfd4acdb9ec0ceb17799/rank_0_0/model
(EngineCore pid=1868) INFO 07-31 16:35:06 [monitor.py:53] torch.compile took 129.06 s in total
(EngineCore pid=1868) INFO 07-31 16:35:06 [monitor.py:81] Initial profiling/warmup run took 0.13 s
(EngineCore pid=1868) INFO 07-31 16:35:08 [gpu_worker.py:563] Available KV cache memory: 7.69 GiB
(EngineCore pid=1868) INFO 07-31 16:35:08 [kv_cache_utils.py:2235] GPU KV cache size: 200,424 tokens
(EngineCore pid=1868) INFO 07-31 16:35:08 [kv_cache_utils.py:2236] Maximum concurrency for 8,192 tokens per request: 24.47x
(EngineCore pid=1868) INFO 07-31 16:35:08 [gpu_worker.py:707] Compile and warming up model for size 8192
(EngineCore pid=1868) INFO 07-31 16:35:09 [cutedsl_warmup.py:100] Skipping CuTeDSL warmup on non-CUDA platform.
(EngineCore pid=1868) WARNING 07-31 16:35:09 [model_runner.py:754] Skipping CUDA graph capture. To turn on CUDA graph capture, ensure `cudagraph_mode` was not manually set to `NONE`
(EngineCore pid=1868) INFO 07-31 16:35:09 [gpu_worker.py:789] Free memory on device (29.38/30.3 GiB) on startup. Desired GPU memory utilization is (0.85, 25.75 GiB). Actual usage is 11.66 GiB for consumed memory (weights + non-torch), 6.4 GiB for peak activation, and 0.0 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=8103856743` (7.55 GiB) to fit into requested memory, or `--kv-cache-memory=12004325376` (11.18 GiB) to fully utilize gpu memory. Current kv cache memory in use is 7.69 GiB.
(EngineCore pid=1868) INFO 07-31 16:36:35 [jit_monitor.py:79] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=1868) INFO 07-31 16:36:36 [core.py:348] init engine (profile, create kv cache, warmup model) took 233.51 s (compilation: 129.06 s)
(EngineCore pid=1868) INFO 07-31 16:36:36 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])

...
...
...

Postprocessing:   0%|          | 0/2000 [00:00<?, ?it/s]
Postprocessing:  62%|██████▏   | 1241/2000 [00:00<00:00, 12405.28it/s]
Postprocessing: 100%|██████████| 2000/2000 [00:00<00:00, 12475.70it/s]
(EngineCore pid=1868) INFO 07-31 16:38:07 [core.py:1332] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=1868) INFO 07-31 16:38:07 [core.py:1468] [shutdown] EngineCore: start mode=drain timeout=5s
(EngineCore pid=1868) INFO 07-31 16:38:07 [core.py:1499] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=1868) INFO 07-31 16:38:07 [core.py:1345] [shutdown] EngineCore: exiting busy loop
(EngineCore pid=1868) INFO 07-31 16:38:07 [xpu_worker.py:167] XPUWorker shutdown: cleaning up (rank=0, local_rank=0)
(EngineCore pid=1868) INFO 07-31 16:38:08 [xpu_worker.py:177] XPUWorker shutdown: done (rank=0, local_rank=0)
(EngineCore pid=1868) WARNING 07-31 16:38:08 [parallel_state.py:2187] torch._C._host_emptyCache() only available in Pytorch >=2.5
vllm (model=google/gemma-3n-E2B-it,tensor_parallel_size=1,pipeline_parallel_size=1,dtype=bfloat16,trust_remote_code=True,max_model_len=8192,gpu_memory_utilization=0.85), gen_kwargs: (temperature=0), limit: 2000.0, offset: 0, num_fewshot: None, batch_size: 16

LMMs-Eval: Probing Intelligence in the Real World
> Good evaluation shapes what we build next.

branch: detached
commit: v0.6-173-g057bcac7

|Tasks|Filter|n-shot|    Metric    |   |Value |   |Stderr|Stderr_CLT|
|-----|------|-----:|--------------|---|-----:|---|------|---------:|
|pope |none  |     0|pope_accuracy |↑  |0.8345|±  |N/A   |    0.0083|
|pope |none  |     0|pope_f1_score |↑  |0.8397|±  |N/A   |    0.0083|
|pope |none  |     0|pope_precision|↑  |0.8141|±  |N/A   |    0.0083|
|pope |none  |     0|pope_recall   |↑  |0.8670|±  |N/A   |    0.0083|
|pope |none  |     0|pope_yes_ratio|↑  |0.5000|±  |N/A   |    0.0083|

...
...
...

WARNING 07-31 16:38:12 [core_client.py:725] [shutdown] MPClient: engine core exited unexpectedly; starting cleanup
INFO 07-31 16:38:12 [core_client.py:686] [shutdown] MPClient: start timeout=default
INFO 07-31 16:38:12 [core_client.py:688] [shutdown] MPClient: stopping engine manager
INFO 07-31 16:38:12 [core_client.py:690] [shutdown] MPClient: engine manager stopped
INFO 07-31 16:38:12 [core_client.py:691] [shutdown] MPClient: cleaning up background resources
INFO 07-31 16:38:12 [core_client.py:693] [shutdown] MPClient: complete
!!!!!!! Segfault encountered !!!!!!!