How nightly orchestration works

The problem: “orchestration” invites over-engineering. This platform’s honest answer today is a deliberately boring one — a timer-driven batch script with explicit pins and layered safety nets — plus a documented target design (durable intent + reconciliation) it may grow into. This page describes both and says plainly which is which.

Layer 1 — the real thing: the nightly batch

A scheduled timer starts a plain batch script that walks the active contracts: Collect (explicitly pinned version) → generation under a queue-depth backpressure gate → publish → hub deploy → live-URL verification → stamp. No daemon, no workflow engine — an auditable script whose every phase leaves records.

Layer 2 — the safety nets are the architecture

The interesting design is not the happy path but the nets under it:

• Publish sweep — at the end of the same run, anything generated but not yet published gets a second chance immediately.
• Reconcile timers — two independent recurring jobs (publish-side and collect-side) re-drive stranded work every few hours; a crashed run’s leftovers don’t wait for tomorrow.
• Outcome watchdog — late morning, a separate timer checks that the night produced an outcome at all — the “zero is a failure” doctrine applied to the whole night.
• Run-level self-healing — the run engine’s watchdog and reaper handle stuck runs and abandoned jobs below this layer.

Redundancy here is deliberate: each net catches a failure class the others cannot see.

Layer 3 — the target design (status: designed, not built)

The documented next generation is reconciler-first: each contract’s day becomes a durable cycle record (intent), and a single loop continuously compares intent against observed facts, taking one idempotent step at a time — the pattern that would replace scattered safety nets with one convergence engine, add a proper admission gate and provider-aware submission breaker. Status: designed, not built. No stored cycle record (intent) exists yet — the cycles API domain that shows each night’s funnel derives it after the fact from run and publication records; the current admission control is the batch script’s own backpressure; a general cron-scheduler for individual runs is implemented but not enabled in production. This page will change when that changes.

Trade-offs

A batch script is single-shape: contracts run in a fixed pattern, and mid-run steering is limited (a running pipeline completes rather than stopping mid-step; see the run engine). The layered nets mean recovery logic lives in several places — exactly the dispersion the reconciler design wants to collapse. The platform chose to ship the boring version first and measure where it actually hurts; the design doc exists because those measurements now do.

See it in two minutes

The cycles API domain shows each night’s funnel — dispatched, completed, failed, recovered — which is this whole page in numbers.