How quality gates work

The problem: an autonomous pipeline that publishes without human review needs something on the publish path that says no. The obvious 2024-era answer — “have an LLM judge the article” — was tried here and measured. It failed.

The measurements that set the design

Two results shaped everything:

1. Cheap LLM judges anti-correlate with human quality ratings on this workload — articles the judge scored highest, humans scored lower. A gate built on that signal would have inverted quality.
2. Naive corpus-grounding produced ~73% false positives when used to detect fabricated claims — checking a sentence against the corpus flags mostly innocent phrasing, not invention.

The conclusion is the platform’s quality doctrine: deterministic gates decide; LLMs only advise. Where an LLM verdict is unavoidable, it must be evidence-anchored: a verbatim quote from the text is required for any non-zero score, and the quote is mechanically checked to actually occur as a substring — a judge cannot praise what it cannot cite.

The gates on the publish path

• Topic dedup — the candidate is embedded and compared against the contract’s already-published topics and its batch peers. A duplicate is deferred, not deleted: the verdict is “not now”, because topic space frees up as content ages. The same comparator also runs earlier, on the Collect side, so most duplicates are never generated at all — one core, two seams.
• Claim verification — numerical and normative claims are checked against retrieved sources, not against the corpus alone (“verify against the world, not against yourself” — the lesson of the 73% false-positive measurement). An article with unverifiable claims is quarantined.
• Quality scoring — deterministic signal battery (brand-safety, language consistency, structural checks) stamped onto every article: telemetry that feeds audits and remediation.

The failure philosophy: quarantine, not regeneration

A gate failure does not trigger regeneration. Measured on live data, the overwhelming majority of blind retries fail the same gate again — burning money to re-lose the same argument. A failed article goes to quarantine and returns only when something material changed (the corpus gained relevant facts, the topic freed up). Every re-entry path is enumerated; there is no silent retry loop.

The trade-offs, honestly

Deterministic gates have false positives too — a defer verdict on a genuinely fresh angle, a grounding block on a claim whose source phrasing differs in units. The design accepts this: on an autonomous publish path, a wrongly-held article costs a day; a wrongly-published fabrication costs trust. Signals are tuned against measured false-positive rates, and every gate’s verdict is recorded with the article, so a human can overrule with full context.

See it in two minutes

Read the gate verdicts on any article via the API (quality/publish domains), or see where the gates sit in Content lifecycle.

Specs: SPEC-064/074 (two-tier doctrine), SPEC-065 (evidence-anchored judging), SPEC-115 (relevance), SPEC-116/118 (dedup, publish + collect), SPEC-117 (claim verification), SPEC-126 (regeneration suppression).