[VL] Fix native SIGSEGV in ~ObjectStore when release() races with store destruction

[yikf]

What changes are proposed in this pull request?

~ObjectStore() checked aliveObjects_.empty() outside the lock and then read aliveObjects_.rbegin() inside the lock. While a store is being destructed, another thread can still call the static ObjectStore::release() (e.g. a script-transformation feed thread closing its output iterator during task teardown), which erases entries from aliveObjects_ under the lock. A concurrent erase could empty the map between the unlocked check and rbegin(), so rbegin() dereferences an empty tree and crashes the JVM with a native SIGSEGV in std::_Rb_tree_decrement.

Fix: move the empty-check under the lock so it is atomic with popping the last object.

Surfaced as a flaky native crash in CI running GlutenSparkScriptTransformationSuite:

• Failing job: https://github.com/apache/gluten/actions/runs/31084950810/job/92564768040?pr=12714

From hs_err_pid*.log:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f3ae8f62f7e, pid=7077, tid=7655
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6+9) (build 17.0.6-ea+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6-ea+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libgluten.so+0xcdbf7e]  std::_Rb_tree_decrement(std::_Rb_tree_node_base*)+0xe
#
# Core dump will be written. Default location: Core dumps may be processed with "/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %d" (or dumping to /__w/gluten/gluten/gluten-ut/spark40/core.7077)
#
# If you would like to submit a bug report, please visit:
#   https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-17-openjdk
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: -Dlog4j.configurationFile=file:src/test/resources/log4j2.properties -Dbasedir=/__w/gluten/gluten/gluten-ut/spark40 -Dspark.test.home=/opt/shims/spark40/spark_home/ -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dfile.encoding=UTF-8 org.scalatest.tools.Runner -R /__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/classes /__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/test-classes -l org.apache.spark.tags.ExtendedSQLTest org.apache.spark.tags.SlowHiveTest org.apache.gluten.tags.UDFTest org.apache.gluten.tags.SkipTest -w org.apache.spark.sql.execution -w org.apache.spark.sql.catalyst -w org.apache.spark.sql.errors -w org.apache.spark.sql.extension -o -u /__w/gluten/gluten/gluten-ut/spark40/target/surefire-reports/.

Host: AMD EPYC 7763 64-Core Processor, 4 cores, 15G, CentOS Stream release 8
Time: Thu Aug  6 08:53:09 2026 UTC elapsed time: 23.585455 seconds (0d 0h 0m 23s)

---------------  T H R E A D  ---------------

Current thread (0x00007f3b5000a060):  JavaThread "Executor task launch worker for task 1.0 in stage 12.0 (TID 25)" daemon [_thread_in_native, id=7655, stack(0x00007f3abb3b6000,0x00007f3abb4b7000)]

Stack: [0x00007f3abb3b6000,0x00007f3abb4b7000],  sp=0x00007f3abb4b4548,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libgluten.so+0xcdbf7e]  std::_Rb_tree_decrement(std::_Rb_tree_node_base*)+0xe

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 19294  org.apache.gluten.runtime.RuntimeJniWrapper.releaseRuntime(J)V (0 bytes) @ 0x00007f3b6d3363e9 [0x00007f3b6d3363a0+0x0000000000000049]
J 19293 c1 org.apache.gluten.runtime.Runtime$RuntimeImpl.release()V (63 bytes) @ 0x00007f3b66c0bffc [0x00007f3b66c0ba60+0x000000000000059c]
J 14720 c1 org.apache.spark.task.TaskResourceRegistry$$Lambda$6960+0x00000008029b9890.apply$mcV$sp()V (8 bytes) @ 0x00007f3b663dc504 [0x00007f3b663dc3c0+0x0000000000000144]
J 15357 c2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007f3b6d0a315c [0x00007f3b6d0a3120+0x000000000000003c]
J 14722 c1 org.apache.spark.task.TaskResourceRegistry.exclusiveLock(Lscala/Function0;)Ljava/lang/Object; (43 bytes) @ 0x00007f3b663dbe0c [0x00007f3b663dbc60+0x00000000000001ac]
J 13925 c1 org.apache.spark.task.TaskResourceRegistry$$Lambda$6959+0x00000008029b94c0.apply(Ljava/lang/Object;)Ljava/lang/Object; (12 bytes) @ 0x00007f3b6627ef0c [0x00007f3b6627e940+0x00000000000005cc]
J 12394 c2 scala.collection.immutable.List.foreach(Lscala/Function1;)V (32 bytes) @ 0x00007f3b6cf578a0 [0x00007f3b6cf57820+0x0000000000000080]
J 18048 c1 org.apache.spark.task.TaskResourceRegistry.$anonfun$releaseAll$3(Lorg/apache/spark/task/TaskResourceRegistry;Lscala/Tuple2;)V (56 bytes) @ 0x00007f3b662c05a4 [0x00007f3b662bf6e0+0x0000000000000ec4]
J 15272 c1 org.apache.spark.task.TaskResourceRegistry$$Lambda$6958+0x00000008029b90f0.apply(Ljava/lang/Object;)Ljava/lang/Object; (12 bytes) @ 0x00007f3b6657e1dc [0x00007f3b6657e040+0x000000000000019c]
J 12394 c2 scala.collection.immutable.List.foreach(Lscala/Function1;)V (32 bytes) @ 0x00007f3b6cf578a0 [0x00007f3b6cf57820+0x0000000000000080]
j  org.apache.spark.task.TaskResourceRegistry.$anonfun$releaseAll$1(Lorg/apache/spark/task/TaskResourceRegistry;)V+31
j  org.apache.spark.task.TaskResourceRegistry$$Lambda$6956+0x00000008029b8a48.apply$mcV$sp()V+4
J 15357 c2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007f3b6d0a315c [0x00007f3b6d0a3120+0x000000000000003c]
J 12463 c1 org.apache.spark.task.TaskResourceRegistry.lock(Lscala/Function0;)Ljava/lang/Object; (22 bytes) @ 0x00007f3b661f6fec [0x00007f3b661f6e60+0x000000000000018c]
j  org.apache.spark.task.TaskResourceRegistry.releaseAll()V+7
j  org.apache.spark.task.TaskResources$$anon$3.onTaskCompletion(Lorg/apache/spark/TaskContext;)V+38
j  org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1(Lorg/apache/spark/TaskContextImpl;Lorg/apache/spark/util/TaskCompletionListener;)V+2
j  org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1$adapted(Lorg/apache/spark/TaskContextImpl;Lorg/apache/spark/util/TaskCompletionListener;)Ljava/lang/Object;+2
j  org.apache.spark.TaskContextImpl$$Lambda$6955+0x00000008029b8678.apply(Ljava/lang/Object;)Ljava/lang/Object;+8
j  org.apache.spark.TaskContextImpl.invokeListeners(Ljava/util/Stack;Ljava/lang/String;Lscala/Option;Lscala/Function1;)V+87
j  org.apache.spark.TaskContextImpl.invokeTaskCompletionListeners(Lscala/Option;)V+15
j  org.apache.spark.TaskContextImpl.markTaskCompleted(Lscala/Option;)V+32
j  org.apache.spark.TaskContext.runTaskWithListeners(Lorg/apache/spark/scheduler/Task;)Ljava/lang/Object;+39
j  org.apache.spark.scheduler.Task.run(JILorg/apache/spark/metrics/MetricsSystem;ILscala/collection/immutable/Map;Lscala/Option;)Ljava/lang/Object;+258
j  org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Lorg/apache/spark/executor/Executor$TaskRunner;Lscala/collection/immutable/Map;Lscala/runtime/BooleanRef;)Ljava/lang/Object;+37
j  org.apache.spark.executor.Executor$TaskRunner$$Lambda$7513+0x0000000802ad9248.apply()Ljava/lang/Object;+12
J 18605 c1 org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object; (225 bytes) @ 0x00007f3b66a10414 [0x00007f3b66a103a0+0x0000000000000074]
J 16272 c1 org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(Lorg/apache/spark/util/SparkErrorUtils;Lscala/Function0;Lscala/Function0;)Ljava/lang/Object; (7 bytes) @ 0x00007f3b6676d6ec [0x00007f3b6676d640+0x00000000000000ac]
j  org.apache.spark.util.Utils$.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object;+3
j  org.apache.spark.executor.Executor$TaskRunner.run()V+590
J 17287 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V java.base@17.0.6-ea (187 bytes) @ 0x00007f3b651089c4 [0x00007f3b651080a0+0x0000000000000924]
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@17.0.6-ea
J 14300 c1 java.lang.Thread.run()V java.base@17.0.6-ea (17 bytes) @ 0x00007f3b66319f84 [0x00007f3b66319e40+0x0000000000000144]
v  ~StubRoutines::call_stub

How was this patch tested?

flaky test, existed test to verify.

Was this patch authored or co-authored using generative AI tooling?

Yes, AI-assisted, Generated-by: Claude claude-opus-4-8.

[Copilot]

Pull request overview

Fixes a native SIGSEGV in gluten::ObjectStore::~ObjectStore() caused by a TOCTOU race where aliveObjects_.empty() was checked outside mtx_ and aliveObjects_.rbegin() was used inside mtx_, allowing a concurrent ObjectStore::release() to empty the map in-between and trigger UB.

Changes:

• Move aliveObjects_.empty() check under mtx_ in ~ObjectStore() to make the empty-check + last-element pop atomic.
• Add an in-code concurrency explanation documenting the race and crash mechanism.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cpp/core/utils/ObjectStore.cc:52

• There is trailing whitespace at the end of this comment line, which can cause unnecessary diffs and may fail whitespace/formatting checks.

      // The empty-check and the pop of the last alive object must be atomic under `mtx_`. 

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.