--- title: How source discovery works description: Where article candidates come from — three source classes, a funnel of independent drop-gates, and the honest state of the cheap-to-expensive cascade. locale: en version: 0.2.0 canonical: https://enterium-developer-docs.pages.dev/developers/how-it-works/source-discovery/ --- > Documentation index: https://enterium-developer-docs.pages.dev/developers/llms.txt — every page of this portal has a Markdown twin at .md; the frozen API contract is at https://enterium-developer-docs.pages.dev/developers/api/openapi.json. **The problem:** everything downstream — generation, quality, cost — is bounded by what Collect finds. Feed it noise and the best writing pipeline in the world produces polished noise. Discovery therefore optimizes for *qualification density*, not raw harvest volume. ## Layer 1 — three source classes, deliberately unequal ```mermaid flowchart LR subgraph SRC [Source classes] F[Curated feeds
tiered, trusted] Q[Search harvest
web + news surfaces,
curator-authored queries] L[Legacy news API
kept as fallback] end F --> H[Harvest pool] Q --> H L --> H ``` - **Curated feeds** — a tiered, hand-picked list per contract: high precision, low recall; the trusted backbone. - **Search harvest** — the workhorse: a search API queried on both its evergreen and news surfaces, driven by **curator-authored queries** from the contract's ontology — not blind scraping; the query set *is* an editorial instrument. - **Legacy news API** — an older, lower-precision harvester kept beside the new one as a fallback. The contract's *own* site pages are deliberately **not** a candidate source — they feed ontology construction (what the niche is about), never the article queue. ## Layer 2 — the funnel of independent gates There is no single "drop reason" field; rejection is distributed across gates that each own one concern: ```mermaid flowchart TB H[Harvest pool] --> D1[URL dedup
vs everything already seen] D1 --> D2[Per-domain cap
no single source floods the funnel] D2 --> S[Cheap LLM screener
pass / fail] S --> DA[Deep LLM analysis
rank 0–100] DA --> TD[Topic dedup
cosine vs committed topics —
DEFER, reversible] TD --> EX[Export gate
daily generation budget ·
impact-ranked top-N] EX --> P[Seeded as pending article] ``` Two gates deserve attention: - **Topic dedup defers, never deletes.** A candidate too similar to already-committed topics (or to a peer in the same run) is marked deferred — the only *reversible* verdict in the funnel, because topic space frees up as content ages. - **The export gate enforces a hard daily generation budget.** Each contract has a per-day budget — fleet default of 5 articles, overridable per contract (0 is a legal full stop). The remainder is counted against a database day-window, so a restarted run cannot double-spend the limit. Within the budget, candidates are ranked by **expected impact**: `(rank_deep / 100) × yield_ratio(topic)`, where `yield_ratio` is the topic's measured impressions-per-article relative to the contract median (clamped 0.25–4.0), fed by the [topic feedback loop](../topic-feedback-loop/). Topics without data get a neutral weight — with no SEO signal the ranking degrades to plain rank order, but the cap always holds (the two axes fail independently). An explore floor reserves part of the budget for new topics; candidates the budget does not reach stay in the pool and compete again the next night. Below the rank floor a candidate simply never becomes an article; absence is the rejection record. The historical asymmetries are closed: the old auto-pass band no longer bypasses the cap, and the legacy **priority path** (an operator override that skipped the rank floor) is ignored by budget selection — nothing in the platform writes that flag anymore, and the field is slated for removal. ## Layer 3 — the cascade today The economic idea is *cheap before expensive*: kill candidates before they cost real LLM money. Today's production cascade achieves this with a **cheap LLM screener** ahead of the deep analyzer — every surviving candidate costs at least one small LLM call. A **vector pre-filter** (embed candidates, recall-rank them by cosine before any LLM sees them) exists as a measured next generation: on a live run it cut the screening workload by ~5.7× on that run (714 harvested → 646 in the recall pool → 38 screened in). It is owner-gated and not yet the production default — the catalog's version pinning is exactly what lets both generations coexist. ## Layer 4 — discovery also feeds the corpus Collect does double duty: qualified source pages are mined into **research facts** through the same admission pipeline generation uses (content-hash dedup → provenance and relevance gates → conflict arbitration). Discovery is thus the [RAG flywheel's](../rag-flywheel/) intake, not just the article queue's. ## Trade-offs Curated queries and feeds mean recall depends on curation quality — a stale query set starves the funnel silently, which is why harvest-novelty is monitored as a signal. And competitor filtering deliberately does *not* live here: brand defense operates downstream (generation and publish), where context exists to judge intent — a URL alone is not evidence of promotion. ## See it in two minutes A contract's funnel is visible in its run records: harvested → screened → ranked → deferred → seeded, step by step, for every nightly run.