Repository navigation
P2: Add structured metrics/logging for analysis initialization lock lifecycle #869
Description
Problem
The analysis_zip.lock / analysis_zip.receipt coordination mechanism is opaque. There is no structured observability for lock acquisition, wait duration, reclaim attempts, or timeout exits. This made the 2026-08-04 stale-lock incident hard to detect early and difficult to quantify during response.
Proposed solution
Emit structured log events (key=value pairs or JSON) for each stage of the initialization lock lifecycle:
| Event | Key fields |
|---|---|
| Lock acquired | analysis_id, lock_path, ts |
| Receipt found | analysis_id, wait_duration_s |
| Waiting for receipt | analysis_id, lock_age_s, elapsed_s (periodic) |
| Stale lock detected | analysis_id, lock_age_s, action |
| Timeout exit | analysis_id, elapsed_s, reason |
These events should be parseable by log aggregators (Vector, Loki, etc.) and ideally expose a counter/gauge for alerting on initialization stalls.
Acceptance criteria
- All lock lifecycle stages emit at least one structured log line
- Log lines include enough context to correlate across workers and analyses
- No sensitive data in log output
- Existing log output is not made significantly more verbose under normal conditions
Context
- Incident: 2026-08-04 OpenStudio Server queue stall
Activity