After a reboot the kernel stays down: restore a lost kernel on daemon start

[haralez]

What I'm trying to do, and what stops me

After a host reboot I want rig start (or the daemon coming back under a service manager) to bring back the kernel: the operator, advisor and queue worker. Today it never does:

• The daemon skips it. bootKernelIfNeeded returns skipped ("kernel rig already managed") whenever a rig named kernel exists (kernelAlreadyManaged only checks the name). After a reboot the rig exists, but none of its seats are running.
• rig start doesn't restore it either. It treats skipped as fine and leaves kernel out of its restore candidates (if (rig.name === "kernel") continue; in start.ts).
• The status says nothing useful. /api/kernel/status and rig status report skipped whether the kernel is up or gone, because the tracker never looks at liveness in that state.

So the kernel only comes back if someone knows to run rig up kernel --existing by hand. #1060 makes rig start say so. This proposal is about not needing it.

Evidence

On one Linux host running OpenRig 0.6.6, after an orderly reboot:

• The kernel didn't come back. The daemon started under its service manager, but the kernel stayed down until it was restored by hand with rig up kernel --existing. We now run that from a separate boot step, before rig start --last, as a workaround.
• The status doesn't track liveness. On 0.6.7, with the kernel up, GET /api/kernel/status returns kernel_state: "skipped" and detail: "kernel rig already managed". That's the same answer as when it's down.
• On main (0.6.9) the code paths are unchanged. That covers kernelAlreadyManaged, the tracker's skipped state, and the kernel exclusion in rig start.

The fact that makes this safe

The daemon can already tell a lost kernel from a stopped one, and it knows which before kernel boot runs:

• rig down (rig-teardown.ts) marks the rig's sessions exited.
• A reboot or crash leaves them running in the database. Startup reconciliation, which runs before bootKernelIfNeeded, finds no tmux session and marks them detached (reconciler.ts).

Proposed behaviour

In bootKernelIfNeeded, when a kernel rig exists:

1. Some kernel seat is live (a running session after reconciliation): skip as today, but report it truthfully, as ready, partial_ready or booting derived from the seats (the tracker already does this aggregation after a cold boot).
2. No live seat, and its latest occupants are detached (lost, not stopped): restore it the way rig up kernel --existing does, fired and not awaited like the cold boot, and tracked as booting through ready, partial_ready or bootstrap_failed.
3. No live seat, and its latest occupants are exited (someone ran rig down kernel): stay skipped, with a detail that says the kernel is stopped and how to start it.
4. OPENRIG_NO_KERNEL=1 and test mode: unchanged, never restore.

Why it stays safe

• A deliberate stop is respected. Rule 3 keeps rig down kernel sticky across daemon restarts, so this never resurrects a kernel someone stopped.
• No double kernel. Restore only runs when reconciliation found no live kernel seat. The restore path already refuses live or ambiguous occupants, and already pins the bound tmux session name.
• Conversations are kept. It resumes the seats' saved sessions through the same restore path as rig up kernel --existing, rather than cold-booting a new kernel over the old rig.
• Nothing else is restored. Other rigs still come back only through rig start or rig up. Restoring all rigs on daemon start is a separate question and not part of this.

Cost to everyone else

• One new daemon-start path for an existing-but-lost kernel. A host whose kernel was lost to a reboot will start agent sessions on daemon start without a command, which is what happens on first boot today.
• /api/kernel/status gives more truthful values in the already-managed case (ready and so on, instead of always skipped). Anything that treats skipped as "kernel fine" keeps working, because it will see ready instead.
• No new setting. If you'd rather keep this opt-in, a setting that defaults to off would also work for us.

Open questions for maintainers

• Is "latest occupants all detached" the right lost-versus-stopped signal, or is there a better rig-level marker?
• Should the status for an already-managed live kernel come from the same seat aggregation fix(kernel): report a recovered kernel as ready, and list each kernel seat once #1044 introduced for recovery?
• Would you rather rig start include the kernel as a restore candidate (CLI-side) than have the daemon do it? The daemon side covers service-manager restarts where nobody runs rig start.

I'm happy to send a PR for whichever shape you prefer.

[lab1207]

Verified all three code claims on current main, and the proposal is sound. Do this:

Confirmed in source. kernelAlreadyManaged (kernel-boot.ts:136) is a pure name check; the already-managed branch (:97) returns skipped without looking at liveness; start.ts:287 excludes the kernel from restore candidates. The status conflation you measured (same skipped up or down) is exactly what the code does.

On your three questions:

1. Detached-vs-exited is the right lost-vs-stopped signal — it's already the codebase's own vocabulary (reconciliation marks detached, teardown marks exited), so rule 3 speaks the existing language instead of inventing a marker.
2. Yes, reuse the fix(kernel): report a recovered kernel as ready, and list each kernel seat once #1044 recovery aggregation for the live-kernel status rather than building a second one. One aggregation, two callers.
3. Daemon side, not CLI side. Your reason is dispositive: service-manager restarts have no CLI invocation, so a CLI-only fix leaves the reported case (reboot under a service manager) broken. Keep rig start as is.

One test to demand of the PR: reboot-simulation with a stopped kernel (teardown-marked) asserting no resurrection — rule 3 is the load-bearing behavior and the easiest to regress. Confirm the above by reading bootKernelIfNeeded top to bottom; it's ~70 lines.

[mvschwarz]

Thanks, @haralez, for the report from your Linux host: after an orderly reboot under a service manager, the kernel stayed down until it was restored by hand, and its status said skipped whether it was up or gone. Thanks also, @lab1207, for checking the source. Your reading is that kernelAlreadyManaged checks only the name, that the already-managed branch returns skipped without looking at liveness, and that rig start leaves the kernel out of its restore candidates. We haven't reproduced the report ourselves.

We'd welcome the PR you offered, kept focused and with regression tests. For your three questions, please have it take a position on each and give its reasons; the review will decide. The tests should keep today's behavior where it's intended: a kernel someone stopped with rig down kernel stays stopped across daemon restarts, and running with no kernel (OPENRIG_NO_KERNEL=1) never restores one. Your #1060, still open, would have rig start say why it skipped the kernel and how to bring it back. With #1060 merged, that message would be the fallback for any case a fix doesn't cover. We'll review the new PR once it's open.

[haralez]

PR: #1081. It takes a position, with reasons, on each of the three open questions. Detached-versus-exited is the signal, tightened to require each seat to still be bound to its session, because unclaiming a seat also marks it detached. Status reuses the existing seat aggregation. The fix is daemon-side, with no new setting. Its regression tests include a kernel stopped with rig down kernel staying stopped across daemon restarts, and OPENRIG_NO_KERNEL=1 never restoring one.