How observability and alerting work
The problem: an autonomous nightly platform fails in two very different ways — loudly (an exception) and quietly (everything exits 0 and produces nothing). Classic monitoring catches the first kind. The second kind once ran for two days straight here: zeros across the fleet, not a single alert. That incident shaped this design.
Layer 1 — three primitives that make silence structurally hard
Section titled “Layer 1 — three primitives that make silence structurally hard”All error handling funnels through three small primitives:
| Primitive | For | Guarantee |
|---|---|---|
capture_exception | Any caught error | Never raises; logs + metric + durable error_event row (secrets stripped) in one call |
guarded(...) | One-shot batch operations | Catches everything, captures, returns None — a batch step cannot take the process down |
supervise(...) | Long-running background loops | On crash: capture + restart with backoff — a loop cannot die silently |
The third one is the important lesson: the projector, watchdogs, health monitor and autoscaler all run under supervision, because a background loop that dies quietly turns into “the platform looks fine and does nothing” — the worst failure class of an autonomous system.
The sink is queryable (GET /api/v1/errors — see
How to read errors); the write is
best-effort by design, so the structured log line remains the last-resort
record for a crash before the event loop exists.
Layer 2 — the “zero is a failure” alert doctrine
Section titled “Layer 2 — the “zero is a failure” alert doctrine”The nightly run ends with a self-assessment that distinguishes three bad endings — and the third is the doctrine:
all_zero is the interesting one: nothing crashed, every exit code is
clean — but a platform whose whole night produced zero useful output is
failing at its job. Infrastructure monitoring cannot see this; only an
outcome check can.
Layer 3 — the dead-man’s switch
Section titled “Layer 3 — the dead-man’s switch”Alert delivery itself can die (the host, the network, the messaging credentials — or the same cloud account that runs production). So the final tier is inverted: the nightly job pings an external service on success and on failure, and the absence of any ping is what raises the alarm — from infrastructure deliberately independent of the platform’s own account. A provider-level kill switch that takes down production cannot take down the thing watching production.
The trade-offs, honestly
Section titled “The trade-offs, honestly”The full typed-alert pipeline (severity levels, dedup keys, a rules-driven dispatcher, distributed tracing) is designed and approved but parked — what runs today is the Phase-0 belt above: three outcome modes, the error sink, and the dead-man’s switch. The page will change when the rest ships; until then, this is the honest inventory. There is also no metrics-scrape endpoint yet — counters exist, dashboards do not.
See it in two minutes
Section titled “See it in two minutes”GET /api/v1/errors?limit=5 shows the sink live; the nightly report (the
cycles domain) carries the outcome assessment of the latest run.
Specs: SPEC-060 (error sink + primitives), SPEC-091 (alerting doctrine; full dispatcher designed, Phase-0 live).