How source discovery works

The problem: everything downstream — generation, quality, cost — is bounded by what Collect finds. Feed it noise and the best writing pipeline in the world produces polished noise. Discovery therefore optimizes for qualification density, not raw harvest volume.

Layer 1 — three source classes, deliberately unequal

• Curated feeds — a tiered, hand-picked list per contract: high precision, low recall; the trusted backbone.
• Search harvest — the workhorse: a search API queried on both its evergreen and news surfaces, driven by curator-authored queries from the contract’s ontology — not blind scraping; the query set is an editorial instrument.
• Legacy news API — an older, lower-precision harvester kept beside the new one as a fallback.

The contract’s own site pages are deliberately not a candidate source — they feed ontology construction (what the niche is about), never the article queue.

Layer 2 — the funnel of independent gates

There is no single “drop reason” field; rejection is distributed across gates that each own one concern:

Two gates deserve attention:

• Topic dedup defers, never deletes. A candidate too similar to already-committed topics (or to a peer in the same run) is marked deferred — the only reversible verdict in the funnel, because topic space frees up as content ages.
• The export gate enforces a hard daily generation budget. Each contract has a per-day budget — fleet default of 5 articles, overridable per contract (0 is a legal full stop). The remainder is counted against a database day-window, so a restarted run cannot double-spend the limit. Within the budget, candidates are ranked by expected impact: (rank_deep / 100) × yield_ratio(topic), where yield_ratio is the topic’s measured impressions-per-article relative to the contract median (clamped 0.25–4.0), fed by the topic feedback loop. Topics without data get a neutral weight — with no SEO signal the ranking degrades to plain rank order, but the cap always holds (the two axes fail independently). An explore floor reserves part of the budget for new topics; candidates the budget does not reach stay in the pool and compete again the next night. Below the rank floor a candidate simply never becomes an article; absence is the rejection record.

The historical asymmetries are closed: the old auto-pass band no longer bypasses the cap, and the legacy priority path (an operator override that skipped the rank floor) is ignored by budget selection — nothing in the platform writes that flag anymore, and the field is slated for removal.

Layer 3 — the cascade, honestly

The economic idea is cheap before expensive: kill candidates before they cost real LLM money. Today’s production cascade achieves this with a cheap LLM screener ahead of the deep analyzer — every surviving candidate costs at least one small LLM call. A vector pre-filter (embed candidates, recall-rank them by cosine before any LLM sees them) exists as a measured next generation: on a live run it cut the screening workload by ~5.7× (harvest 714 → recall pool 646 → 38 screened-in → 55 selected). It is owner-gated and not yet the production default — the catalog’s version pinning is exactly what lets both generations coexist.

Layer 4 — discovery also feeds the corpus

Collect does double duty: qualified source pages are mined into research facts through the same admission pipeline generation uses (content-hash dedup → provenance and relevance gates → conflict arbitration). Discovery is thus the RAG flywheel’s intake, not just the article queue’s.

The trade-offs, honestly

Curated queries and feeds mean recall depends on curation quality — a stale query set starves the funnel silently, which is why harvest-novelty is monitored as a signal. And competitor filtering deliberately does not live here: brand defense operates downstream (generation and publish), where context exists to judge intent — a URL alone is not evidence of promotion.

See it in two minutes

A contract’s funnel is visible in its run records: harvested → screened → ranked → deferred → seeded, step by step, for every nightly run.

Specs: SPEC-077/082 (collect architecture + ontology), SPEC-080 (vector recall — measured, gated), SPEC-118 (collect-side topic dedup), SPEC-101 (corpus intake).