---
title: How nightly orchestration works
description: What actually drives the fleet every night — a deliberately boring batch script with layered safety nets — and the reconciler-first design it is converging toward.
locale: en
version: 0.2.0
canonical: https://enterium-developer-docs.pages.dev/developers/how-it-works/orchestration/
---

> Documentation index: https://enterium-developer-docs.pages.dev/developers/llms.txt — every page of this portal has a Markdown twin at <route>.md; the frozen API contract is at https://enterium-developer-docs.pages.dev/developers/api/openapi.json.

**The problem:** "orchestration" invites over-engineering. This platform's
honest answer today is a deliberately boring one — a timer-driven batch
script with explicit pins and layered safety nets — plus a documented
target design (durable intent + reconciliation) it may grow into. This
page describes both and says plainly which is which.

## Layer 1 — the real thing: the nightly batch

```mermaid
gantt
    dateFormat HH:mm
    axisFormat %H:%M
    title The nightly window (conceptual)
    section Per contract
    Collect (pinned version) :a, 00:00, 45m
    Generate (queue backpressure) :b, after a, 90m
    Publish + hub deploy :c, after b, 30m
    Live-URL verify + stamp :d, after c, 10m
    section Safety nets
    Publish sweep (same run) :e, after d, 15m
    Reconcile timers (every 4h) :f, 00:00, 24h
    Outcome watchdog (late morning) :g, 11:00, 10m
```

A scheduled timer starts a plain batch script that walks the active
contracts: Collect (explicitly pinned version) → generation under a
queue-depth backpressure gate → publish → hub deploy → live-URL
verification → stamp. No daemon, no workflow engine — an auditable
script whose every phase leaves records.

## Layer 2 — the safety nets are the architecture

The interesting design is not the happy path but the nets under it:

- **Publish sweep** — at the end of the same run, anything generated but
  not yet published gets a second chance immediately.
- **Reconcile timers** — two independent recurring jobs (publish-side and
  collect-side) re-drive stranded work every few hours; a crashed run's
  leftovers don't wait for tomorrow.
- **Outcome watchdog** — late morning, a separate timer checks that the
  night *produced an outcome at all* — the
  ["zero is a failure" doctrine](../observability-alerting/) applied to
  the whole night.
- **Run-level self-healing** — the [run engine's](../run-engine/)
  watchdog and reaper handle stuck runs and abandoned jobs below this
  layer.

Redundancy here is deliberate: each net catches a failure class the
others cannot see.

## Layer 3 — the target design (status: designed, not built)

The documented next generation is **reconciler-first**: each contract's
day becomes a durable *cycle record* (intent), and a single loop
continuously compares intent against observed facts, taking one
idempotent step at a time — the pattern that would replace scattered
safety nets with one convergence engine, add a proper admission gate and
provider-aware submission breaker. **Status: designed, not built.** No
stored cycle *record* (intent) exists yet — the `cycles` API domain that
shows each night's funnel derives it after the fact from run and
publication records; the current admission control is the batch
script's own backpressure; a general cron-scheduler for individual runs
is implemented but not enabled in production. This page will change when
that changes.

## Trade-offs

A batch script is single-shape: contracts run in a fixed pattern, and
mid-run steering is limited (a running pipeline completes rather than
stopping mid-step; see [the run engine](../run-engine/#trade-offs)). The layered nets mean recovery logic lives in several
places — exactly the dispersion the reconciler design wants to collapse.
The platform chose to ship the boring version first and measure where it
actually hurts; the design doc exists because those measurements now do.

## See it in two minutes

The `cycles` API domain shows each night's funnel — dispatched,
completed, failed, recovered — which is this whole page in numbers.