Pipeline catalog

All content on the platform is produced by pipelines — multi-step programs that run against a contract’s data. The catalog file pipelines/catalog.yaml is the single source of truth for dispatch: which pipelines exist, which variants they have, and which version is the current default.

The pipelines

Pipeline	Purpose	Runs against
Collect (link_ranker)	Source discovery: harvests candidate source articles for a contract’s niche, ranks them by semantic relevance to the contract’s ontology, and qualifies the best ones for generation. The v3 line is vector-based: candidates are ranked on embedding cosine similarity rather than keyword matching.	a sales contract
Generate (generate)	Article production: takes a qualified source, grounds it in the contract’s research corpus (RAG), writes the article through a staged LLM pipeline with deterministic quality gates, renders the final layout and publishes it.	a generated article
Localize (localize)	Language versions: produces localized variants of an article for multilingual instances.	a sales contract

How the catalog works

Each pipeline has variants (prod, dev) and a list of versions, with a latest pointer naming the current default:

pipelines:
  generate:
    display_name: "Generate (контент-пайплайн)"
    entity: generated_articles
    variants:
      prod:
        latest: v3.1.x        # current default
        versions:
          v3.0.0: { ... }     # every version stays listed — and runnable
          v3.1.x: { ... }

Three rules follow from this design:

1. Adding a version is a one-line edit in the catalog plus a new folder under pipelines/{id}/{variant}/{version}/. Nothing else registers it.
2. Old versions stay runnable. A run can be pinned to any listed version; latest only sets the default for new dispatches.
3. The manifest is the contract. Each version folder carries its own manifest.yaml describing steps, required environment and entrypoints — the dispatcher reads the manifest, never the code, to decide how to run it.

Where pipelines are triggered from

• The nightly daily-run — the production scheduler dispatches Collect → Generate → publish across the whole contract fleet on a timer. The run is visible as pipeline_run / run_step records via the API.
• The API — individual runs can be launched per contract or per article.
• Version pins — the daily-run pins explicit pipeline versions, so a freshly-landed version does not reach production traffic until the pin is moved deliberately.