Skip to content

fix: renew tasks lock so long jobs are not stolen - #3629

Open
pplulee wants to merge 1 commit into
postalserver:mainfrom
pplulee:fix/renew-tasks-lock-heartbeat
Open

fix: renew tasks lock so long jobs are not stolen#3629
pplulee wants to merge 1 commit into
postalserver:mainfrom
pplulee:fix/renew-tasks-lock-heartbeat

Conversation

@pplulee

@pplulee pplulee commented Aug 31, 2026

Copy link
Copy Markdown

Summary

  • The :tasks role is only renewed at the start of each run_tasks cycle. If a scheduled task (retention, CheckAllDNS, …) runs longer than 5 minutes, another worker steals the lock and runs the same work concurrently.
  • Renew acquired_at every 30 seconds while tasks are running, using a stable locker name captured on the tasks thread.
  • Treat Galera/MySQL StatementInvalid on lock updates as a failed acquire instead of crashing the tasks thread.

Addresses the lock-steal race described in #3595.

Test plan

  • Run two workers and a long scheduled task (> 5 minutes).
  • Confirm the second worker does not log acquired task role by stealing it from a lazy worker while the first is still running tasks.
  • Confirm worker_roles.acquired_at keeps moving forward during the long task.
  • Stop the holder; after ~5 minutes another worker can steal the role.

Made with Cursor

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants