How nightly orchestration works

The problem: “orchestration” invites over-engineering. This platform’s honest answer today is a deliberately boring one — a timer-driven batch script with explicit pins and layered safety nets — plus a documented target design (durable intent + reconciliation) it may grow into. This page describes both and says plainly which is which.

Layer 1 — the real thing: the nightly batch

A scheduled timer starts a plain batch script that walks the active contracts: Collect (explicitly pinned version) → generation under a queue-depth backpressure gate → publish → hub deploy → live-URL verification → stamp. No daemon, no workflow engine — an auditable script whose every phase leaves records.

Layer 2 — the safety nets are the architecture

The interesting design is not the happy path but the nets under it:

• Publish sweep — at the end of the same run, anything generated but not yet published gets a second chance immediately.
• Reconcile timers — two independent recurring jobs (publish-side and collect-side) re-drive stranded work every few hours; a crashed run’s leftovers don’t wait for tomorrow.
• Outcome watchdog — late morning, a separate timer checks that the night produced an outcome at all — the “zero is a failure” doctrine applied to the whole night.
• Run-level self-healing — the run engine’s watchdog and reaper handle stuck runs and abandoned jobs below this layer.

Redundancy here is deliberate: each net catches a failure class the others cannot see.

Layer 3 — the target design, honestly labeled

The documented next generation is reconciler-first: each contract’s day becomes a durable cycle record (intent), and a single loop continuously compares intent against observed facts, taking one idempotent step at a time — the pattern that would replace scattered safety nets with one convergence engine, add a proper admission gate and provider-aware submission breaker. Status: designed, not built. No cycle table exists yet; the current admission control is the batch script’s own backpressure; a general cron-scheduler for individual runs exists in code but is switched off and unused in production. This page will change when that changes.

The trade-offs, honestly

A batch script is single-shape: contracts run in a fixed pattern, and mid-run steering is limited (cancel of a running pipeline is effectively a no-op today). The layered nets mean recovery logic lives in several places — exactly the dispersion the reconciler design wants to collapse. The platform chose to ship the boring version first and measure where it actually hurts; the design doc exists because those measurements now do.

See it in two minutes

The cycles API domain shows each night’s funnel — dispatched, completed, failed, recovered — which is this whole page in numbers.

Specs: SPEC-108 (reconciler-first — designed), SPEC-021 (run scheduler — built, idle), SPEC-104 (run FSM — designed), plus the live daily-run tooling.