Fix native SIGSEGV when hold() races with release() on NativeMemoryManager

[yikf]

What changes are proposed in this pull request?

NativeMemoryManager.hold() called the native hold(handle) JNI method without checking whether the manager had already been released. When a task tears down its runtime (release() frees the native handle) while another thread is still closing an output iterator via ColumnarBatchOutIterator.close0() -> memoryManager().hold(), hold() dereferences a freed handle and crashes the JVM with a SIGSEGV (SEGV_MAPERR = use-after-free).

Surfaced as a flaky native crash in CI running GlutenSparkScriptTransformationSuite (its TRANSFORM ... USING tests close the columnar output iterator on a feed thread concurrently with task teardown):

• Failing job: https://github.com/apache/gluten/actions/runs/31071855879/job/92523319304?pr=12697

From hs_err_pid*.log :

# C  [libgluten.so+0x4fe59e]  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR)

Current thread: JavaThread "Thread-SparkScriptTransformationWriterThread-Feed"
C  [libgluten.so+0x4fe59e]  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
j  org.apache.gluten.memory.NativeMemoryManager$Impl.hold()V
j  org.apache.gluten.vectorized.ColumnarBatchOutIterator.close0()V
j  org.apache.gluten.iterator.ClosableIterator.close()V
...
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.run()V

How was this patch tested?

flaky test, existed test to verify.

Was this patch authored or co-authored using generative AI tooling?

Yes, AI-assisted, Generated-by: Claude claude-opus-4-8.

[yikf]

@zhztheplayer could you please take a look if you have time, thanks.

[Copilot]

Pull request overview

This PR fixes a race in NativeMemoryManager where hold() could call into native code after release() had freed the underlying native handle, leading to a JVM crash (SIGSEGV) under concurrent task teardown and iterator close.

Changes:

• Add an instance-level lock to make hold() and release() mutually exclusive around native JNI calls.
• Make hold() a safe no-op when the memory manager has already been released, preventing use-after-free.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zhztheplayer]

@yikf Thanks. I thought .hold() should never be called after .release() is called (Because all ColumnarBatchOutIterator instances should be closed before the Runtime is destroyed at the end of the Spark task). Is there a way to repeat the failure model from CI locally or in test?

[yikf]

Thanks @zhztheplayer. You're right that in the normal flow the iterator is closed before the Runtime is destroyed — but the ordering guarantee only holds within the task thread, and script transformation breaks it by closing the iterator from a second thread.

resIter.close() (which calls hold()) can be triggered from two places which are recycler and hasNext, both funneled through IteratorCompleter.tryComplete() and guarded by a CAS so it runs exactly once:

• the TaskResources recycler (priority 100), run on the task thread at task completion;
• IteratorCompleter.hasNext returning false, run on whatever thread is driving the iterator.

In the normal single-threaded path both run on the task thread, and releaseAll() executes resources by descending priority, so the iterator recycler (100) holds before the Runtime (30) releases — hold() always precedes release().

SparkScriptTransformationExec.processIterator starts BaseScriptTransformationWriterThread (a setDaemon(true) "Feed" thread) and never joins it; that thread drains the Gluten input iterator via iter.foreach. The race:

1. Feed thread: hasNext hits end-of-stream → tryComplete() wins the CAS (marks completed) but gets descheduled before running resIter.close().
2. Task thread: main thread finishes reading the script's stdout → task completes → the recycler's tryComplete() finds the CAS already taken and no-ops, so the task thread never holds and proceeds straight to Runtime.release() → nmm.release(), freeing the native handle.
3. Feed thread: resumes, runs resIter.close() → ColumnarBatchOutIterator.close0() → hold() on the now-freed handle → SIGSEGV.

So hold() runs exactly once, but on the feed thread, with no happens-before against the task thread's release(). The hs_err stack matches — hold() under BaseScriptTransformationWriterThread.run().

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f01a868459e, pid=7037, tid=7720
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6+9) (build 17.0.6-ea+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6-ea+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libgluten.so+0x4fe59e]  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
#
# Core dump will be written. Default location: Core dumps may be processed with "/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %d" (or dumping to /__w/gluten/gluten/gluten-ut/spark40/core.7037)
#
# If you would like to submit a bug report, please visit:
#   https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-17-openjdk
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: -Dlog4j.configurationFile=file:src/test/resources/log4j2.properties -Dbasedir=/__w/gluten/gluten/gluten-ut/spark40 -Dspark.test.home=/opt/shims/spark40/spark_home/ -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dfile.encoding=UTF-8 org.scalatest.tools.Runner -R /__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/classes /__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/test-classes -l org.apache.spark.tags.ExtendedSQLTest org.apache.spark.tags.SlowHiveTest org.apache.gluten.tags.UDFTest org.apache.gluten.tags.SkipTest -w org.apache.spark.sql.execution -w org.apache.spark.sql.catalyst -w org.apache.spark.sql.errors -w org.apache.spark.sql.extension -o -u /__w/gluten/gluten/gluten-ut/spark40/target/surefire-reports/.

Host: AMD EPYC 9V74 80-Core Processor, 4 cores, 15G, CentOS Stream release 8
Time: Thu Aug  6 05:15:16 2026 UTC elapsed time: 22.784636 seconds (0d 0h 0m 22s)

---------------  T H R E A D  ---------------

Current thread (0x00007f021007c970):  JavaThread "Thread-SparkScriptTransformationWriterThread-Feed" daemon [_thread_in_native, id=7720, stack(0x00007f0180ed1000,0x00007f0180fd2000)]

Stack: [0x00007f0180ed1000,0x00007f0180fd2000],  sp=0x00007f0180fcff60,  free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libgluten.so+0x4fe59e]  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
j  org.apache.gluten.memory.NativeMemoryManager$Impl.hold()V+4
j  org.apache.gluten.vectorized.ColumnarBatchOutIterator.close0()V+9
j  org.apache.gluten.iterator.ClosableIterator.close()V+13
j  org.apache.gluten.backendsapi.velox.VeloxIteratorApi.$anonfun$genFinalStageIterator$2(Lscala/Function1;Lorg/apache/gluten/metrics/IteratorMetricsJniWrapper;Lorg/apache/gluten/vectorized/ColumnarBatchOutIterator;)V+13
j  org.apache.gluten.backendsapi.velox.VeloxIteratorApi$$Lambda$7598+0x0000000802b4afc8.apply$mcV$sp()V+12
J 18857 c1 org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext()Z (23 bytes) @ 0x00007f0226a9bed4 [0x00007f0226a9b780+0x0000000000000754]
J 18316 c1 org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext()Z (14 bytes) @ 0x00007f0225f5a3f4 [0x00007f0225f5a2c0+0x0000000000000134]
j  org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext()Z+4
j  org.apache.gluten.execution.VeloxColumnarToRowExec$$anon$1.hasNext()Z+4
J 15005 c2 scala.collection.Iterator$$anon$10.hasNext()Z (76 bytes) @ 0x00007f022d0418d0 [0x00007f022d0417c0+0x0000000000000110]
J 18859 c1 org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext()Z (122 bytes) @ 0x00007f0226a9d26c [0x00007f0226a9cc80+0x00000000000005ec]
J 18857 c1 org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext()Z (23 bytes) @ 0x00007f0226a9b894 [0x00007f0226a9b780+0x0000000000000114]
J 963 c2 scala.collection.Iterator$$anon$9.hasNext()Z (10 bytes) @ 0x00007f022c94b1e8 [0x00007f022c94b1a0+0x0000000000000048]
J 10777 c2 scala.collection.IterableOnceOps.foreach(Lscala/Function1;)V (36 bytes) @ 0x00007f022ce7bf70 [0x00007f022ce7bea0+0x00000000000000d0]
J 6133 c1 scala.collection.AbstractIterator.foreach(Lscala/Function1;)V (6 bytes) @ 0x00007f0225794dec [0x00007f0225794d40+0x00000000000000ac]
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.processRowsWithoutSerde()V+21
j  org.apache.spark.sql.execution.SparkScriptTransformationWriterThread.processRows()V+1
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.$anonfun$run$1(Lorg/apache/spark/sql/execution/BaseScriptTransformationWriterThread;)V+13
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread$$Lambda$8729+0x0000000802dd1da8.apply$mcV$sp()V+4
J 14838 c2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007f022d02585c [0x00007f022d025820+0x000000000000003c]
j  org.apache.spark.util.Utils$.logUncaughtExceptions(Lscala/Function0;)Ljava/lang/Object;+1
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.run()V+9
v  ~StubRoutines::call_stub
V  [libjvm.so+0x843dba]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x32a
".//spark-test-spark40-group2-test-log/gluten-ut/spark40/hs_err_pid7037.log" 2057L, 177036B

[yikf]

btw, i found this case at other PR ci pipeline, https://github.com/apache/gluten/actions/runs/31071855879/job/92523319304?pr=12697

[yikf]

@zhztheplayer friendly re-ping, please take a look again if you have time.

[zhztheplayer]

@yikf Would you help check the failed CI?

[yikf]

@zhztheplayer It also failed due to similar issues. I submitted a PR: #12740