Client gives up on a daemon that is still starting: 'did not become healthy after 3 spawn attempts' while embedded zccache takes 40-47 s to become ready

[zackees]

What happened

fbuild deploy (via FastLED bash autoresearch) intermittently fails before any compile:

ERROR fbuild::output: daemon error: daemon did not become healthy after 3 spawn attempts (10s each).
This is a daemon problem, not a defect in the code being built — no compilation was attempted.

Two occurrences on 2026-09-25 against an otherwise idle host.

Evidence (~/.fbuild/prod/daemon/daemon.log, UTC)

02:14:59.467  fbuild daemon starting on port 63518
02:14:59.478  listening on 0.0.0.0:63518                       <- port bound, health not yet served
02:15:00.175  artifact index loaded loaded=25576
02:15:00.270  metadata cache file not found, starting empty  path=.../zccache/v1.13.1/metadata.bin
02:15:00.273  system include cache restored entries=20
02:15:12.854  fbuild daemon starting ... another fbuild-daemon already owns 0.0.0.0:63518; yielding
02:15:27.032  fbuild daemon starting ... yielding (single-flight)
          (client gives up here, ~38 s after first spawn)
02:15:47.028  zccache backend ready (embedded)                 <- 46.8 s of silence before this
02:15:48.913  Daemon is idle; self-eviction in 30s

daemon start	zccache backend ready	startup	client result
01:53:56	01:54:08	12 s	ok
01:45:15	01:45:55	40 s	failed
02:14:59	02:15:47	47 s	failed

The daemon is healthy, just late. Single-flight works correctly: every respawn yields to the running daemon. The client's total budget (3 spawn attempts, about 10–13 s each) is shorter than a slow embedded-zccache startup, so the whole build is reported as failed even though the daemon comes up moments later.

Every start logs metadata cache file not found, starting empty for zccache/v1.13.1/metadata.bin. The file is never persisted, so each daemon start may pay a full metadata rebuild. That is a likely source of the 12–47 s variance.

Ask

1. The client should wait on "a daemon that owns the port is starting" (bind succeeded, backend initialising) rather than respawn and count down. For example, serve a starting health state as soon as the port is bound and poll it with a longer, progress-aware deadline.
2. Log what the embedded zccache backend is doing between system include cache restored and backend ready, so the ~45 s is attributable.
3. Check why metadata.bin is never found or persisted across daemon restarts. The daemon self-evicts after 30 s idle, so cold starts are frequent in agent workflows, and each one hits this path.

Related

• fbuild-daemon grows to ~3.9GB while idle then fails health check; heap-profile entry points not wired #1360 (daemon memory growth failing health check) is a different health failure.
• Daemon lets a second deploy preempt an in-flight deploy on the same port (no per-port lease); contention misreported as 'serial driver may be wedged' #1461 is the same session's port-contention bug.

fbuild 2.5.27 (PyPI wheel in FastLED's .venv), Linux 6.18 NixOS.

[zackees]

Root cause on the zccache side is tracked in zackees/zccache#1652 (embedded backend bring-up of 12–47 s, and metadata.bin is never persisted). This fbuild issue stays open for the client-side part: wait on a daemon that owns the port and is still starting, instead of respawning and giving up.