Symptom
While its loader was completely stuck (see companion issue), petapico advertised:
Nanopub-Query-Status: READY
Nanopub-Query-Load-Counter: 87675
Nanopub-Query-Loaded-Nanopub-Count: 87675
Nanopub-Query-Registry-Nanopub-Count: 87686
Nanopub-Query-Loader-Last-Success-Age-Seconds: 1481
READY, while 11 nanopubs behind and having made no progress for ~25 minutes. The instance had also flipped from LOADING_UPDATES to READY during the stall, so the status got less accurate as the problem persisted.
Why it matters
The liveness header exists precisely so a stalled instance is visible from outside, and here it reported healthy throughout. The only usable signal was Loader-Last-Success-Age-Seconds climbing past 1400 — a field you have to know to look at, and to have a threshold for.
This is the same failure mode as #139: an instance quietly serving stale data while claiming to be caught up. A load balancer, a fleet health check, or a person glancing at the status page would all have concluded the instance was fine.
Observed context
- ingestion had been failing continuously, with the loader's circuit breaker active after 10 consecutive batch failures
- so the app knew both that batches were failing repeatedly and that the registry count exceeded its own — neither fact reached the status
- queries were healthy, so nothing else hinted at a problem
Suggested directions
READY should require more than "the loader's own counter matches its own last commit": consecutive batch failures and last-success age are already tracked and could gate it- a distinct state (e.g.
DEGRADED / STALLED) when the loader has failed N consecutive batches or has not committed for longer than a threshold
- consider whether
Loaded-Nanopub-Count trailing Registry-Nanopub-Count by more than a small margin should ever be reported as READY
- the fleet consistency script compares instances against each other; a single instance that is behind and says
READY is exactly what it cannot catch
Symptom
While its loader was completely stuck (see companion issue), petapico advertised:
READY, while 11 nanopubs behind and having made no progress for ~25 minutes. The instance had also flipped fromLOADING_UPDATEStoREADYduring the stall, so the status got less accurate as the problem persisted.Why it matters
The liveness header exists precisely so a stalled instance is visible from outside, and here it reported healthy throughout. The only usable signal was
Loader-Last-Success-Age-Secondsclimbing past 1400 — a field you have to know to look at, and to have a threshold for.This is the same failure mode as #139: an instance quietly serving stale data while claiming to be caught up. A load balancer, a fleet health check, or a person glancing at the status page would all have concluded the instance was fine.
Observed context
Suggested directions
READYshould require more than "the loader's own counter matches its own last commit": consecutive batch failures and last-success age are already tracked and could gate itDEGRADED/STALLED) when the loader has failed N consecutive batches or has not committed for longer than a thresholdLoaded-Nanopub-CounttrailingRegistry-Nanopub-Countby more than a small margin should ever be reported asREADYREADYis exactly what it cannot catch