[Bug] SIGTERM during INIT_FAILED publication can escape forked child main

[doraemonmj]

Platform

All / Unknown

Runtime Variant

All / Unknown

Description

The device-free Ubuntu UT

tests/ut/py/test_worker/test_startup_readiness.py::TestNextLevelStartupFailure::test_second_child_failure_reaps_first

has intermittently failed twice while reclaiming a next-level child after another child reports an injected init failure. The test body observes the expected injected inner init failure, but its finally: w4.close() then raises:

RuntimeError: child next pid <pid> still alive; shm not freed

There is a signal race in _forked_child_main consistent with this failure. During setup() failure it does:

except BaseException as e:
    _write_error(...)
    _mailbox_store_i32(state_addr, _INIT_FAILED)
    os._exit(1)

The parent can observe _INIT_FAILED immediately and start rollback by sending SIGTERM. The child's SIGTERM handler raises _StartupCancelled. If SIGTERM arrives after the _INIT_FAILED store but before os._exit(1), _StartupCancelled is raised from inside the except BaseException suite. The sibling except _StartupCancelled does not catch an exception raised inside another except suite, so it can escape _forked_child_main and unwind into the forked copy of the caller's startup/test frames.

That violates _forked_child_main's documented load-bearing invariant: a forked child must never unwind into the copied parent startup frames and must always terminate through os._exit.

The CI symptom is consistent with the escaping child not becoming reapable before the shared startup-abort deadline expires. It is then placed in the cleanup journal, and the subsequent nonblocking waitpid(..., WNOHANG) in close() can still observe it alive. The controlled repro below proves the exception escape; the exact final still alive outcome remains scheduler-dependent.

Observed CI failures on two consecutive PR heads whose diff does not touch python/simpler/worker.py, this test, or the UT workflow:

• run 32438975068, Ubuntu UT job, head 9f5cb11babb3a548f1467d857bfc1b9884b0f36c
• run 32439970394, Ubuntu UT job, head c9ab22a5d417db35072d7f5bd320621948aef872

A third rerun passed, which matches an intermittent timing race:

• run 32442207474, Ubuntu UT job, head 2e19f5bcd3e4bd491eaa52357209b9e48ccb237b

Steps to Reproduce

The unmodified test is low probability: it passed 20/20 consecutive iterations on a local Linux aarch64 host.

A controlled device-free repro widens only the interval after publishing _INIT_FAILED. It reliably shows _StartupCancelled escaping _forked_child_main:

PYTHONPATH=tests/ut/py .venv/bin/python - <<'PY'
import time
import simpler.worker as worker
from test_worker.test_startup_readiness import TestNextLevelStartupFailure

original_store = worker._mailbox_store_i32


def delayed_store(addr, value):
    original_store(addr, value)
    if value == worker._INIT_FAILED:
        time.sleep(0.25)


worker._mailbox_store_i32 = delayed_store
TestNextLevelStartupFailure().test_second_child_failure_reaps_first()
PY

The relevant output is:

During handling of the above exception, another exception occurred:

  File "python/simpler/worker.py", line 4371, in _forked_child_main
    _mailbox_store_i32(state_addr, _INIT_FAILED)
  File "<stdin>", line 10, in delayed_store
  File "python/simpler/worker.py", line 4359, in _on_cancel
    raise _StartupCancelled()
simpler.worker._StartupCancelled

The traceback continues into the forked copy of test_second_child_failure_reaps_first, demonstrating the forbidden unwind. On a fast local host the parent usually still reaps the process in time, so the controlled command may exit successfully despite printing the child traceback.

Expected Behavior

• A setup failure publishes its error and the forked child always terminates through os._exit.
• SIGTERM during any point of child initialization or failure publication cannot unwind past _forked_child_main.
• Startup rollback followed by close() is convergent and does not fail because a just-terminated child has not yet become reapable.

Actual Behavior

In CI, rollback leaves the next-level child journaled as a survivor:

[worker pid=...] WARN: startup child reclaim failed (continuing best-effort):
child process(es) [...] did not exit within the close budget

Then w4.close() replays the journal cleanup and fails:

python/simpler/worker.py:3650: RuntimeError
RuntimeError: child next pid <pid> still alive; shm not freed

The child stderr contains the expected injected init failure immediately before the reclaim warning.

Git Commit ID

3b578e30a2a9e2859d19908b2647393ffdecc543 (main base containing the affected Worker code); also observed at the PR heads listed above, where the affected files are unchanged from this base.

CANN Version

N/A — device-free UT

Driver Version

N/A — device-free UT

Host Platform

Linux (x86_64) in CI; controlled exception-escape repro also confirmed on Linux (aarch64).

Additional Context

Related but not duplicates:

• [Bug] Forked worker children survive parent death and spin at 100% CPU forever #1493 covers children surviving an abrupt parent death. Here the parent is alive and executing normal startup rollback.
• [Bug] Worker.close() intermittently fails to reap fork children within the close budget on a multi-card a5 pool #1562 covers normal device-worker close and a SHUTDOWN-clobber race fixed by Fix: fence register/unregister and make shutdown sticky #1663. This failure is device-free and occurs during init-failure rollback.
• [Bug] Worker.close() still leaves child processes after sticky shutdown fix on a2a3 #1809 covers device/serving teardown after successful execution and was closed as belonging to another repository.
• fix(worker): bound hierarchical startup with readiness protocol #1396 introduced the readiness protocol, _forked_child_main, and this regression test.
• Add: cooperative init cancellation and retryable resource cleanup journal #1673 added cooperative init cancellation and retained _StartupCancelled as the signal-delivered BaseException for forked children.
• Fix retryable worker teardown convergence #1689 added the cleanup-journal survivor path where this now surfaces.

A fix should preserve the invariant at the outermost child boundary even if SIGTERM is delivered while handling another exception. Independently, the journal replay path may need a bounded reap retry after termination rather than a single immediate WNOHANG check, but that is secondary to preventing the child entry point from unwinding.

[ChaoWao]

Two more occurrences, this time reproducible locally rather than only in CI.

On an aarch64 dev box (Python 3.10.9), the full `pytest tests/ut` sweep failed this test on 2 of 2 consecutive runs, with the same `RuntimeError: child next pid still alive; shm not freed` out of `w4.close()`. Run alone, `pytest tests/ut/py/test_worker/test_startup_readiness.py` passes 70/70.

The branch it reproduced on is a mechanical identifier rename whose diff is byte-identical to `upstream/main` across `python/`, `tests/ut/py/`, `conftest.py` and `simpler_setup/` (git diff upstream/main...HEAD -- python/ tests/ut/py/ conftest.py simpler_setup/ is empty), so it adds no evidence about a cause — but it does say the trigger is not CI-specific and does not need the Ubuntu runner.

The load correlation is consistent with the race in the issue body: alone the child has the machine to itself and exits before the abort deadline; inside a 1897-test sweep (this box also had all 16 NPUs busy with other users at the time) it does not, so it lands in the cleanup journal and the `WNOHANG` `waitpid` in `close()` still sees it alive. If a repro under artificial load would help, "run the full ut sweep on a busy machine" appears to be enough.

[ChaoWao]

Another occurrence, same signature, on a diff that touches none of the implicated files.

• Run: https://github.com/hw-native-sys/simpler/actions/runs/32692441293/job/97328853916 (ut (ubuntu-latest, 3.10))
• PR Refactor: drop the scope-stats plumbing hbg cannot reach #1976, head f944c4a92 — the diff is hbg C++ scope-stats plumbing plus one troubleshooting doc; it does not touch python/simpler/worker.py, this test, or the UT workflow.
• RuntimeError: child next pid 9515 still alive; shm not freed

Two extra data points that fit the analysis in the issue body:

• The macOS sibling on the same commit passed its Python UTs. Its ut job reports fail only because the C++ step was cancelled by matrix fail-fast — gh api repos/.../actions/jobs/97328853893 shows "Run Python unit tests": success. So the same code passed on one runner and failed on the other, which is what a scheduler-dependent race looks like rather than a regression.
• 20/20 local passes of that node on the PR head (aarch64, unloaded), so it does not reproduce off a loaded GitHub-hosted runner.

That now makes it three PR heads with unrelated diffs. Not adding anything to the diagnosis — just recording the sighting so the frequency is visible.