fix(tern): cancel never overwrites a terminal apply; task-state writes propagate errors

[aparajon]

Why this matters

A cancel racing an apply's completion could rewrite a fully-applied schema change as cancelled — restamping completed_at and telling the merge gate and operators that a deployed change never ran. Separately, task-state writes that failed were logged and swallowed, letting the drive proceed with in-memory state diverging from the stored rows that crash recovery re-drives from.

What it does

• markApplyCancelled refuses to overwrite an apply that already recorded a final outcome (completed, failed, reverted). The policy is a named predicate, cancelPreservesApplyOutcome.
• Cancel-of-stopped remains supported: stopped is terminal but explicitly cancellable (a driver may cancel a stopped apply to abandon its checkpoint), and a repeated cancel stays idempotent.
• transitionTaskState and markTasksRunning now return write errors instead of logging and continuing; every caller handles the failure explicitly.
• Stop/Cancel control paths fail closed: the operator is only told the stop or cancel succeeded once storage reflects it.
• Drive paths (engine task launch, terminal poll transitions, grouped progress sync, resume preparation) exit for operator retry (taskAbort / propagated error) instead of finalizing from task rows storage never accepted.
• Best-effort finalizers (failApplyWithTasks, pending-task cancellation) and per-tick progress writes log a definitive message and continue — each iteration is independent or rewritten by the next poll.

Before / after

apply completes ──▶ state=completed, completed_at stamped
late cancel     ──▶ overwrites: state=cancelled, completed_at restamped ❌
                    (merge gate told a deployed change never ran)
task write fails ──▶ log + swallow ──▶ drive keeps going ⛔
                    (memory ≠ storage; a crash re-drives stale task rows)

apply completes ──▶ state=completed, completed_at stamped
late cancel     ──▶ refused, final outcome preserved ✅
cancel of stopped ─▶ still cancels (abandon checkpoint) ✅
task write fails ──▶ control path: RPC fails, operator retries ✅
                 └─▶ drive path: owner exits, recovery re-drives from stored state ✅

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR hardens SchemaBot’s local tern drive state persistence so stored apply/task rows remain authoritative under races and storage write failures, preventing late control actions (cancel) or in-memory-only transitions from producing an incorrect merge gate/operator view.

Changes:

• Prevent Cancel from overwriting a terminal apply outcome by guarding markApplyCancelled with cancelPreservesApplyOutcome.
• Propagate task-state persistence errors (instead of logging-and-continuing) across stop/cancel, sequential drive, grouped drive progress sync, and resume preparation paths.
• Add/extend tests to cover cancel-after-terminal-outcome behavior and fail-closed behavior when task writes fail.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pkg/tern/local_control.go	Stop/cancel now fail closed if task terminal-state persistence fails; adds cancelPreservesApplyOutcome and guards cancel state overwrites.
pkg/tern/local_control_test.go	Adds coverage for cancel preserving final apply outcomes and for stop/cancel/drive behavior when task writes fail.
pkg/tern/local_control_resume.go	Makes resume preparation and certain resume transitions fail/abort on task write errors instead of proceeding with diverged in-memory state.
pkg/tern/local_client_test.go	Updates tests to handle new error-returning resume helpers.
pkg/tern/local_client_integration_test.go	Updates integration test to handle new error-returning resume helper.
pkg/tern/local_apply.go	transitionTaskState now returns errors; stale-task conflict resolution stops “unblocking” if it can’t persist the resolution.
pkg/tern/local_apply_sequential.go	Sequential drive now aborts on failed task terminal-state writes (and returns actions from task failure helpers).
pkg/tern/local_apply_grouped.go	Grouped drive now exits on failed progress sync writes; markTasksRunning now returns an error and is handled.
pkg/tern/local_apply_failure.go	Best-effort finalizers now log definitive messages when task write persistence fails.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.