Repository navigation
Fix SubscriptionManager shutdown hang when a publish worker spins on the cancel thread - #5007
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the publish-loop change that lets workers detect manager shutdown directly.
No open issues 🎉
What changed in this PR
Fixes an intermittent SubscriptionManager.DisposeAsync() hang by letting publish workers detect manager shutdown directly, without waiting for cancellation to propagate to their own token.
Changes:
- Check both manager and worker cancellation tokens in the publish loop to prevent spinning on the cancelling thread.
- Document the callback-ordering scenario behind the hang.
| File | Description |
|---|---|
src/Opc.Ua.Client/Subscription/SubscriptionManager.cs |
Adds the manager shutdown check to the publish worker loop and explains why it is needed. |
Review effort: Balanced
SubscriptionManager.DisposeAsync could hang when a Publish worker was blocked on a full ingress buffer. Admission links the manager's token directly, so cancelling the manager can resume the worker inline, inside the manager's callback list, before the callback that cancels the worker's own linked token has run. The worker then saw ct not cancelled, and TryBeginPublishRequest kept returning false because m_disposed was set, so the loop spun without awaiting on the cancelling thread. The spin normally ends when the controller disposes the worker. On net48 CancelAsync is Task.Run(Cancel), so the controller can wake from m_publishControl before the token is flagged and park again. Its wake-up is then also behind the spinning callback, and shutdown never finishes. The worker loop now also checks the manager's shutdown token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code coverage✅ Coverage gate passed.
Coverage is above the recorded baseline - consider ratcheting Thresholds live in |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5007 +/- ##
==========================================
- Coverage 83.29% 83.28% -0.02%
==========================================
Files 3589 3607 +18
Lines 520073 521985 +1912
Branches 88220 88650 +430
==========================================
+ Hits 433182 434718 +1536
- Misses 57533 57736 +203
- Partials 29358 29531 +173
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Description
Fixes an intermittent hang in
SubscriptionManager.DisposeAsync()when a Publish worker is blocked on a full ingress buffer. It showed up asManagerDisposalCancelsSaturatedMessageAdmissionAsynctiming out on net48 (about 1 run in 50 locally, andtest-windows-net48 (2/19)on the CI run for #5005).Root cause
The worker's token is linked to the manager's token. Backpressured admission in
MessageProcessor.OnPublishReceivedAsyncalso links the manager's token directly, through its own registration. When the manager is cancelled:ctnot yet cancelled.TryBeginPublishRequest()returns false becausem_disposedis set, so the loop runscontinuewithout ever awaiting. It spins on the cancelling thread, and the worker's callback can't run.This spin is normally brief, because the publish controller wakes and disposes the worker, which cancels
ctfrom another thread. On net48 that escape can be lost:CancelAsyncis aTask.Run(Cancel)polyfill, som_publishControl.Set()can wake the controller before the token is flagged. The controller then parks again, and its next wake-up also sits behind the spinning callback, so shutdown never finishes.On .NET 8+
CancelAsyncflags the token synchronously, so from the code the controller should always see it. That explains why this appears on net48, but I did not measure the unfixed rate on net10.0.Fix
The worker loop now also checks the manager's
ShutdownToken, so it exits instead of spinning, whichever order the callbacks run in.Evidence
A temporary stack trace at the spin showed
ctCancelled=False shutdown=True, withPublishWorkerAsyncrunning underCancellationTokenSource.Cancel(). The failing runs had this, and so did many passing runs, where the controller broke the spin.Each row ran as parallel loops on a machine shared with other load. Further runs on the fixed code:
FullyQualifiedName~SubscriptiontestsThere is no evidence that #5000 is involved: nothing it changed appears on the spinning stack. The earlier 0/20 result on the commit before it is consistent with the low failure rate.
Not done
CancelAsyncpolyfill's timing.UA.slnxsuite was not run locally, only the tests above, so that checklist item is unchecked.Related Issues
No issue filed. Found while investigating CI on #5005.
Checklist
🤖 Generated with Claude Code