How the run engine works

The problem: a pipeline execution is a long, failure-prone process touching LLM providers, the database and the site builder. If its state lives in the memory of whichever process happens to run it, a crash loses the truth of what happened — and two processes writing “the state” produce two truths.

Layer 1 — one entry, streams in between, one writer

Run creation is a single seam: duplicate-guard, budget preflight, version resolution from the catalog, a pipeline_run row — then a job message onto a Redis Stream. Execution talks back only in events.

Two design choices carry the weight:

• Typed event factories with a JSON-Schema contract. Workers never hand-build event dicts — factories stamp the schema version and validate fields, and the schema ships as a checked-in contract with fixtures for every event type. The web UI’s mock emulator replays the same fixtures, so the frontend develops against the exact wire format.
• The projector is the only writer. Everything else emits; one consumer applies. Idempotency is structural: step counts are computed as distinct terminal steps, not increments, so an at-least-once redelivery cannot double-count. A poison message is acknowledged and captured rather than retried forever.

Layer 2 — the run’s states, and who may move them

Honestly: today these transitions are enforced by convention across the writers (projector, watchdog, retry), not by a formal guard table — an explicit FSM with measured-recovery chaos tests is a designed, not-yet- accepted evolution. What is live is the self-healing pair:

• the watchdog marks runs whose heartbeat went stale as interrupted (a crashed worker cannot leave a run “running” forever);
• the reaper reclaims abandoned job messages from the consumer group, re-queues them up to a retry cap, then dead-letters them. The two never overlap: a run that failed on its own merits is terminal and left alone.

Neither of those catches a run that is blocked rather than dead. A step waiting on a database lock keeps its heartbeat beating, so the watchdog sees a healthy run, and its job message was never abandoned, so the reaper has nothing to reclaim — the run simply stays running. On 2026-07-29 one such transaction held a row lock for six hours; two contracts produced nothing that night and no surface reported a failure. That class is covered by a third, database-level sweep (reap_idle_in_tx), which measures the age of the transaction — not how long the connection has been idle, since a client issuing statements inside one long transaction never looks idle — and terminates only backends that are actually blocking others.

Layer 3 — inputs are frozen before the first step

A run’s inputs (brand profile, author profile, collection config) are assembled once at start into a frozen, typed contract with secrets redacted — steps read ctx.inputs, never re-query the database mid-run. The rule this enforces: a clean boundary between what you were given and what you produced. It exists because both failure modes actually happened — brand guidelines silently truncated by an ad-hoc read, and a config field mutated by step 1 and consumed as “input” by step 6.

Layer 4 — concurrency that senses the provider

The global LLM concurrency cap is not a constant; it is a control loop:

Sense — every live LLM response drops rate-limit telemetry into Redis (fail-soft: telemetry can never break the call). Decide — a periodic tick computes the cap as the minimum of several limiters and applies AIMD (raise slowly, cut sharply, instant cut on 429). Actuate — the cap is a Redis key with a TTL; if the control loop dies, workers fall back to the static setting on their own.

Why several limiters, not just the provider: a live experiment found a silent wall — above a certain concurrency the database pool exhausted while the LLM provider and CPU looked green, and throughput went to zero. The pool limiter, not the provider, is the binding constraint in practice. Boundary hygiene note: the run context is forbidden by the import linter from touching the integrations internals, so Sense and Decide communicate strictly through Redis — the control loop respects the same module boundaries as everything else.

The trade-offs, honestly

Cancel-while-running is currently a no-op in practice (the control channel exists, but nothing consumes it — queued runs cancel instantly, running ones finish or fail). Exactly-once is by-construction rather than by formal replay-test. Both are known, designed-for gaps, not surprises.

See it in two minutes

POST /api/v1/runs launches a run; watch it live over the runs SSE channel, or read its pipeline_run / run_step rows — every claim above is visible in that data.

Specs: SPEC-007 (runs framework), SPEC-055 (typed events + mock emulator), SPEC-062 (idempotent projector), SPEC-071 (adaptive concurrency), SPEC-069/070 (frozen input contracts), SPEC-104 (formal FSM — designed).