---
title: How observability and alerting work
description: Three error primitives that make silence impossible, a durable error sink, and an alerting philosophy built on one rule — "zero useful output" is a failure even when nothing crashed.
locale: en
version: 0.2.0
canonical: https://enterium-developer-docs.pages.dev/developers/how-it-works/observability-alerting/
---

> Documentation index: https://enterium-developer-docs.pages.dev/developers/llms.txt — every page of this portal has a Markdown twin at <route>.md; the frozen API contract is at https://enterium-developer-docs.pages.dev/developers/api/openapi.json.

**The problem:** an autonomous nightly platform fails in two very
different ways — loudly (an exception) and *quietly* (everything exits 0
and produces nothing). Classic monitoring catches the first kind and is
blind to the second — a fleet can produce zeros all night with every
dashboard green. This design starts from that second kind.

## Layer 1 — three primitives that make silence structural­ly hard

All error handling funnels through three small primitives:

| Primitive | For | Guarantee |
| --- | --- | --- |
| `capture_exception` | Any caught error | Never raises; logs + metric + durable `error_event` row (secrets stripped) in one call |
| `guarded(...)` | One-shot batch operations | Catches everything, captures, returns None — a batch step cannot take the process down |
| `supervise(...)` | Long-running background loops | On crash: capture + restart with backoff — a loop cannot die silently |

The third one is the important lesson: the projector, watchdogs, health
monitor and autoscaler all run under supervision, because a background
loop that dies quietly turns into "the platform looks fine and does
nothing" — the worst failure class of an autonomous system.

```mermaid
flowchart TB
    ERR[Any caught error] --> CAP[capture_exception]
    CAP --> LOG[structured log]
    CAP --> MET[error counter]
    CAP --> DB[(error_event<br/>durable sink)]
    subgraph loops [Supervised background loops]
        P[projector] --- W[watchdog] --- H[health monitor] --- A[autoscaler]
    end
    loops -- crash --> CAP
    loops -- restart with backoff --> loops
```

The sink is queryable (`GET /api/v1/errors` — see
[How to read errors](../../guides/reading-errors/)); the write is
best-effort by design, so the structured log line remains the last-resort
record for a crash before the event loop exists.

## Layer 2 — the "zero is a failure" alert doctrine

The nightly run ends with a self-assessment that distinguishes three bad
endings — and the third is the doctrine:

```mermaid
flowchart LR
    RUN[Nightly run ends] --> Q{Assess}
    Q -- half of collects failed --> C[collapse:<br/>exit 1 → alert]
    Q -- generated > 0, published 0 --> PF[publish_fail:<br/>exit 1 → alert]
    Q -- all green but zero qualified --> Z[all_zero:<br/>exit 0, still alarmed<br/>via dead-man's switch]
    Q -- normal --> OK[report + success ping]
```

**`all_zero` is the interesting one:** nothing crashed, every exit code is
clean — but a platform whose whole night produced zero useful output is
failing at its job. Infrastructure monitoring cannot see this; only an
outcome check can.

## Layer 3 — the dead-man's switch

Alert delivery itself can die (the host, the network, the messaging
credentials — or the same cloud account that runs production). So the
final tier is inverted: the nightly job **pings an external service on
success and on failure**, and *the absence of any ping* is what raises
the alarm — from infrastructure deliberately independent of the
platform's own account. A provider-level kill switch that takes down
production cannot take down the thing watching production.

## Trade-offs

The full typed-alert pipeline (severity levels, dedup keys, a rules-driven
dispatcher, distributed tracing) is designed and approved but parked —
what runs today is the Phase-0 belt above: three outcome modes, the error
sink, and the dead-man's switch. The page will change when the rest
ships; until then, this is the honest inventory. There is also no
metrics-scrape endpoint yet — counters exist, dashboards do not.

## See it in two minutes

`GET /api/v1/errors?limit=5` shows the sink live; the nightly report (the
`cycles` domain) carries the outcome assessment of the latest run.