--- title: How the run engine works description: Every pipeline execution as a stream of typed events — Redis Streams transport, a single idempotent DB writer, self-healing watchdogs, and concurrency that adapts to the provider in real time. locale: en version: 0.2.0 canonical: https://enterium-developer-docs.pages.dev/developers/how-it-works/run-engine/ --- > Documentation index: https://enterium-developer-docs.pages.dev/developers/llms.txt — every page of this portal has a Markdown twin at .md; the frozen API contract is at https://enterium-developer-docs.pages.dev/developers/api/openapi.json. **The problem:** a pipeline execution is a long, failure-prone process touching LLM providers, the database and the site builder. If its state lives in the memory of whichever process happens to run it, a crash loses the truth of what happened — and two processes writing "the state" produce two truths. ## Layer 1 — one entry, streams in between, one writer Run creation is a single seam: duplicate-guard, budget preflight, version resolution from the catalog, a `pipeline_run` row — then a job message onto a Redis Stream. Execution talks back **only in events**. ```mermaid sequenceDiagram participant API as Platform API participant J as runs:jobs (stream) participant W as Native worker participant E as runs:events (stream) participant P as Projector participant DB as PostgreSQL participant SSE as SSE clients API->>J: enqueue job (XADD) J->>W: consumer group delivers W->>E: typed events: run_started,
step_completed, run_succeeded… E->>P: consumer group delivers P->>DB: apply_event (sole DB writer) P->>SSE: fan-out via Redis Pub/Sub ``` Two design choices carry the weight: - **Typed event factories with a JSON-Schema contract.** Workers never hand-build event dicts — factories stamp the schema version and validate fields, and the schema ships as a checked-in contract with fixtures for every event type. The web UI's mock emulator replays the same fixtures, so the frontend develops against the exact wire format. - **The projector is the only writer.** Everything else emits; one consumer applies. Idempotency is structural: step counts are computed as *distinct terminal steps*, not increments, so an at-least-once redelivery cannot double-count. A poison message is acknowledged and captured rather than retried forever. ## Layer 2 — the run's states, and who may move them ```mermaid stateDiagram-v2 [*] --> queued: create_run queued --> running: worker picks up queued --> cancelled: cancel running --> succeeded running --> failed: step error (fail-loud) running --> interrupted: watchdog: heartbeat lost ``` Honestly: today these transitions are enforced by convention across the writers (projector, watchdog, retry), not by a formal guard table — an explicit FSM with measured-recovery chaos tests is a designed, not-yet- accepted evolution. What *is* live is the self-healing pair: - the **watchdog** marks runs whose heartbeat went stale as `interrupted` (a crashed worker cannot leave a run "running" forever); - the **reaper** reclaims abandoned *job messages* from the consumer group, re-queues them up to a retry cap, then dead-letters them. The two never overlap: a run that failed on its own merits is terminal and left alone. Neither of those catches a run that is **blocked rather than dead**. A step waiting on a database lock keeps its heartbeat beating, so the watchdog sees a healthy run, and its job message was never abandoned, so the reaper has nothing to reclaim — the run simply stays `running`, holding its row lock and starving every contract queued behind it, while every surface reports health. That class is covered by a third, database-level sweep (`reap_idle_in_tx`), which measures the age of the *transaction* — not how long the connection has been idle, since a client issuing statements inside one long transaction never looks idle — and terminates only backends that are actually blocking others. ## Layer 3 — inputs are frozen before the first step A run's inputs (brand profile, author profile, collection config) are assembled **once** at start into a frozen, typed contract with secrets redacted — steps read `ctx.inputs`, never re-query the database mid-run. The rule this enforces: a clean boundary between *what you were given* and *what you produced*. It exists because both failure modes actually happened — brand guidelines silently truncated by an ad-hoc read, and a config field mutated by step 1 and consumed as "input" by step 6. ## Layer 4 — concurrency that senses the provider The global LLM concurrency cap is not a constant; it is a control loop: ```mermaid flowchart LR A[LLM adapter
reads x-ratelimit-* headers] -- telemetry --> R[(Redis)] R --> D[Autoscaler tick
C* = min of provider ·
DB pool · host · ceiling] D -- set cap --> R R -- read cap --> W[Workers acquire slots] W --> A ``` **Sense** — every live LLM response drops rate-limit telemetry into Redis (fail-soft: telemetry can never break the call). **Decide** — a periodic tick computes the cap as the *minimum of several limiters* and applies AIMD (raise slowly, cut sharply, instant cut on 429). **Actuate** — the cap is a Redis key with a TTL; if the control loop dies, workers fall back to the static setting on their own. Why several limiters, not just the provider: a live experiment found a *silent wall* — above a certain concurrency the **database pool** exhausted while the LLM provider and CPU looked green, and throughput went to zero. The pool limiter, not the provider, is the binding constraint in practice. Boundary hygiene note: the run context is forbidden by the import linter from touching the integrations internals, so Sense and Decide communicate strictly through Redis — the control loop respects the same module boundaries as everything else. ## Trade-offs Cancel applies to *queued* runs immediately; a run that is already executing is not interrupted mid-flight — it runs to completion or failure (the control channel for mid-run interruption is in place, its consumer is on the roadmap). Exactly-once is by-construction rather than by formal replay-test. Both are documented limits, not surprises. ## See it in two minutes `POST /api/v1/runs` launches a run; watch it live over the runs SSE channel, or read its `pipeline_run` / `run_step` rows — every claim above is visible in that data.