How the scoring systems work
The problem: a platform that decides autonomously needs numbers — but a single “quality score” blending unrelated concerns is a number that lies. The design principle here: one score, one question, one consumer — and safety verdicts never hide inside averages.
The inventory
Section titled “The inventory”| Score | Question it answers | Formula idea | Consumer |
|---|---|---|---|
| Candidate rank | ”Is this source worth an article?” | Structured ontology-overlap (tiered term comparison + domain affinity) blended with a deep LLM analysis into 0–100 | The export gate’s floor and bands |
| Dedup cosine | ”Have we already covered this?” | Embedding cosine of the candidate/article against committed topics and batch peers; one calibrated scale for both the collect-side and publish-side gates | Defer verdicts |
| Corpus coverage | ”Does the corpus know enough to write this niche?” | Weighted blend: claim-value coverage (facts carrying usable numbers) and slot coverage (ontology-derived topic slots filled) | The research-floor and enrichment planning |
| Quality stamp | ”How well-made is the artifact?” | Deterministic automatic categories (structure, AI-marker absence, technical SEO) scaled, minus severity-weighted defect penalties | Audits, remediation queues, reporting |
| Brand-safety battery | ”May this ship under our brand?” | A set of independent regex-precise signals (competitor domains, names in titles, recommendation-verbs, third-party promo, buyers-guide shapes, outbound links, placeholder leaks) | The verdict — reject/rework — never the number |
| Topic yield | ”Does this topic earn attention?” | Impressions per article over a maturity-filtered window, with sample-size guards | The topic feedback loop |
| Relevance | ”May this fact be cited here?” | Cited-first membership — the references list contains only what the text actually used; a cross-encoder reranker orders candidates | The writing framework’s retrieval |
The two rules that keep scores honest
Section titled “The two rules that keep scores honest”1. Calibrate against labels, not vibes. The dedup threshold moved only after a manual labeling study (~a hundred pairs): the old threshold measured at ~0.49 precision — a coin flip — the new one at ~0.83, and one notch higher started losing true same-batch clones. Similarly, the corpus coverage match threshold was recalibrated after the original value saturated at 1.000 on virtually every run — a score that always says “perfect” measures nothing.
2. Verdicts don’t hide in averages. Brand-safety and grounding failures flip the article’s verdict directly. A “9/10 quality” article with one competitor-promotion hit is not a 8.5 — it is a rejected article with a high stamp, and both facts stay visible.
The trade-offs, honestly
Section titled “The trade-offs, honestly”Threshold calibration is perishable: change the embedding model and every cosine threshold silently means something else (an offline recalibration protocol exists for exactly this migration). And regex-precise safety signals trade recall for auditability — each signal is narrow and explainable, at the cost of needing new signals as new failure shapes appear; the signal battery grows by incident, deliberately.
See it in two minutes
Section titled “See it in two minutes”Any article row carries its stamp, signals and dedup scores in its
telemetry; any candidate carries its rank; the corpus coverage of a
contract is a first-class read on the research domain.
Specs: SPEC-082 (candidate rank), SPEC-116/118 (dedup calibration), SPEC-101 (coverage), SPEC-106 (quality stamp), SPEC-115 (relevance), SPEC-125 (topic yield).