Source MIT Technology Review · Published · Open the original ↗
Policy
AI’s hidden storage bill: What inference leaves behind
Ask most executives to picture the cost of AI and they’ll likely see GPUs, racks of accelerators with gigawatts of power, and a waitlist for silicon. But that picture leaves something out: Data storage costs.
What happens to data after a model responds? Much of the back-end data that’s generated, from billions of daily prompts, images, and videos, needs to be stored. But data storage is one of the harder infrastructure costs to plan for, and it’s often overlooked in initial AI infrastructure planning.
Traditionally, AI infrastructure conversations have centered on the compute-intensive process of training a model. Training is finite: a run starts, burns through compute, and ends. But inference runs continuously, every time a model answers a prompt. For infrastructure planning, that difference matters: A training run is a project you can budget for and retire. Inference doesn’t retire; every inference request leaves something behind.
The opening of the report, quoted unchanged from MIT Technology Review. The full text continues at the source.