Skip to content

feat(api): require lock ownership for reapply with force takeover#593

Draft
aparajon wants to merge 7 commits into
aparajon/reapply-apifrom
aparajon/reapply-lock-ownership
Draft

feat(api): require lock ownership for reapply with force takeover#593
aparajon wants to merge 7 commits into
aparajon/reapply-apifrom
aparajon/reapply-lock-ownership

Conversation

@aparajon

@aparajon aparajon commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Why

A reapply must not race a concurrent operation on the same database. It needs to hold the database lock before requeuing, and give the operator a way to take over a stale lock.

What

  • Acquire the database lock before requeuing; reject when another owner holds it.
  • Add --force semantics to take over a held lock (release-then-acquire is non-atomic but fails closed — a third caller winning the gap makes the re-acquire reject).
  • Record the acquired lock_id on the requeue. The concurrency guard for re-activating a terminal apply is the active-apply exclusivity re-check under the apply-target advisory lock, not lock-row validation.

Risk Assessment

Medium — adds a fail-closed lock gate to the reapply path: without acquiring the lock the reapply is rejected, and force takeover is explicit. Builds directly on the base endpoint.

PR 3 of 4. Stacked on the API PR.


PR authored with Claude Code (Opus 4.8).

@aparajon
aparajon force-pushed the aparajon/reapply-lock-ownership branch from bef6c63 to bb4ad1e Compare June 29, 2026 21:08
@aparajon
aparajon force-pushed the aparajon/reapply-api branch from 0d650f9 to 7d41999 Compare June 29, 2026 21:08
@aparajon
aparajon force-pushed the aparajon/reapply-lock-ownership branch from bb4ad1e to c4f6c54 Compare June 29, 2026 21:51
@aparajon
aparajon force-pushed the aparajon/reapply-api branch from 7d41999 to 402785c Compare June 29, 2026 21:51
@aparajon
aparajon force-pushed the aparajon/reapply-lock-ownership branch from c4f6c54 to 528c77e Compare June 30, 2026 14:30
@aparajon
aparajon force-pushed the aparajon/reapply-api branch from 402785c to 5f5e465 Compare June 30, 2026 14:30
@aparajon
aparajon force-pushed the aparajon/reapply-lock-ownership branch from 528c77e to 510e858 Compare June 30, 2026 15:00
aparajon and others added 4 commits July 2, 2026 18:09
POST /api/reapply re-plans a failed apply's stored desired schema and, when the
fresh plan still matches the remaining work, requeues the failed apply. Rejects
stale failures, non-reappliable operations, and changed plans. Lock-ownership
enforcement (and force takeover) lands in the next change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…int errors

Split the failed-task work-selection guard so a task pointing at a missing
operation row is logged and skipped distinctly from a task whose operation
isn't failed, rather than conflating the two. Return an error from the plan
equivalence check when a fingerprint fails to marshal, so an internal marshal
failure surfaces as an error instead of masquerading as plan drift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…responses

- Fail closed when a failed task references a missing operation row (a storage
  inconsistency) instead of silently dropping it, which could requeue a partial
  reapply or report no failed work while the task stays failed.
- Canonicalize empty plan artifacts to nil in the equivalence fingerprint so a
  nil vs empty map (null vs {}) doesn't reject an otherwise-identical re-plan.
- Use a "reapply rejected" prefix on 4xx responses (matching the log and the
  other control endpoints) instead of always "reapply failed".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve the caller through resolveCaller like every other control
operation, so a direct-API reapply is audited to the verified identity
rather than the client-supplied caller string.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aparajon
aparajon force-pushed the aparajon/reapply-api branch from 00d8618 to 90fd136 Compare July 2, 2026 22:14
aparajon and others added 3 commits July 2, 2026 18:16
Reapply now acquires the database lock before requeuing the failed apply and
rejects when another owner holds it, so a reapply can't race a concurrent
operation. A --force request takes over a held lock. Threads the lock through
ReapplyFailed so the requeue is conditional on ownership.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Reword the ReapplyFailed contract: the lock_id is recorded for association and
  the concurrency guard is the active-apply re-check under the apply-target
  advisory lock, not lock-row validation — so the doc no longer implies the
  requeue write is conditional on still holding the lock.
- Document that the force release-then-acquire window is intentionally
  non-atomic and fails closed (a third caller winning the gap makes the
  re-acquire reject), so an operator isn't left wondering during an incident.
- Require caller up front in ExecuteReapply, before any apply/plan reads, rather
  than deep inside lock acquisition.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reapply lock's owner, ownership checks, and release now use the
resolved caller — the authenticated identity when the request carries
one — so lock ownership matches the audited identity instead of the
client-supplied caller string. Caller validation happens on the
resolved identity, since with API auth enabled the request caller may
legitimately be empty.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aparajon
aparajon force-pushed the aparajon/reapply-lock-ownership branch from 510e858 to 818907a Compare July 2, 2026 22:20
@aparajon
aparajon force-pushed the aparajon/reapply-api branch 4 times, most recently from 6b22230 to 7110ec1 Compare July 9, 2026 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant