fix(storage): terminal apply states are immutable outside explicit reapply

[aparajon]

Why this matters

Once an apply settles — completed, failed, stopped, cancelled — that final verdict is what operators and the PR merge gate trust. But the general update method wrote whatever state the caller had in memory, last writer wins. A caller holding a stale snapshot (a slow goroutine, a raced handler) could overwrite a settled apply back to "running": the verdict silently disappears, the merge gate reasons from a state that no longer exists, and nobody can tell what actually happened.

What it does

1. The database now refuses to move a settled apply back to an active state. The update SQL only matches rows that are still active, so writing "running" over "completed" is rejected with a distinct error (ErrApplyTerminalStateImmutable) instead of silently succeeding. Settled verdicts stay settled.

2. When a write is refused, the caller learns exactly why. A guarded write that matched no rows is re-read to distinguish three cases: the row is gone (not-found), the row is settled (the immutability refusal), or nothing needed to change (a harmless no-op). The re-read is a locking read, so it sees the latest committed state rather than the transaction's stale snapshot.

3. Legitimate reopenings still work. Restarting a stopped apply and reapplying a failed one go through their own dedicated, compare-and-swap transitions — those are unchanged. Rewriting one terminal state to another (a same-state refresh or verdict correction) also stays allowed.

4. The stop-before-start cleanup no longer clobbers a driver's progress. When a start request finds an earlier stop that already took effect on the data plane, the handler records "stopped" only if the row still holds the exact state that decision was based on. If a driver advanced the apply while the handler was checking, the handler backs off, reloads the row, and leaves the stop request for the current owner — the start is then rejected by the pending-stop gate instead of proceeding over live work.

Before / after

before: apply settles as completed → stale caller writes "running"
        → verdict silently erased, apply resurrected ❌

after:  apply settles as completed → stale caller writes "running"
        → write refused (ErrApplyTerminalStateImmutable), verdict intact ✅

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR enforces “terminal apply state immutability” as a storage-layer safety invariant, preventing stale writers from moving an apply from a terminal verdict back into the active lifecycle, and hardens the stop-before-start normalization path with a CAS write to avoid clobbering concurrent state advances.

Changes:

• Add a storage guard in Applies().Update that refuses terminal → active transitions with a distinct storage.ErrApplyTerminalStateImmutable.
• Disambiguate “0 rows affected” outcomes for guarded active-state updates by re-reading the row to distinguish missing vs terminal vs idempotent no-op.
• Update the stop-before-start handler to use UpdateDerivedState (CAS) and add coverage for the concurrent-advance scenario.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/storage/storage.go	Documents new Update() behavior around terminal immutability.
pkg/storage/mysqlstore/applies.go	Implements terminal→active guard + zero-rows disambiguation helper.
pkg/storage/mysqlstore/applies_test.go	Adds/updates tests for terminal immutability and Update() missing-row behavior.
pkg/storage/errors.go	Introduces ErrApplyTerminalStateImmutable.
pkg/api/handlers_test.go	Extends staticApplyStore test double with UpdateDerivedState CAS semantics.
pkg/api/control_handlers.go	Uses UpdateDerivedState to CAS the stop-before-start normalization write.
pkg/api/control_handlers_test.go	Adds tests covering the CAS stop-before-start behavior and concurrent advance handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.