Why AI infrastructure is becoming a memory game

While discussions around the cost of artificial intelligence infrastructure typically focus on Nvidia and graphics processing units (GPUs), memory is an increasingly important part of the picture. As hyperscalers prepare to build billions of dollars’ worth of new data centers, the price for DRAM (Dynamic Random Access Memory) chips has jumped roughly 7x in the last year. This price jump is turning AI infrastructure into a memory game, where managing how data is stored and retrieved is vital to business viability.

The complexity of this memory game is visible in how AI developers price their services. In an interview with semiconductor analyst Doug O’Laughlin, Val Bercovici, the chief AI officer at data platform company Weka, points to Anthropic’s prompt caching documentation as a key indicator of this shift. Bercovici noted that six or seven months ago, Anthropic’s pricing page for prompt caching was very simple. Today, it has evolved into what he describes as “an encyclopedia of advice on exactly how many cache writes to pre-buy.” The system now features 5-minute tiers and 1-hour tiers, which require customers to navigate arbitrage opportunities. In practice, this means companies must balance “cache writes”—the cost of saving prompt data into temporary memory—against “cache reads,” which are the queries made when retrieving that already-stored data. Managing this window is difficult because adding new data to a query can bump existing information out of the cache.

This shift is driving a new focus on memory orchestration across the technology stack. At the software level, companies that master memory orchestration will be able to make the same queries using fewer tokens, directly lowering their operational costs. This dynamic is creating opportunities for startups; for instance, in October, TechCrunch covered Tensormesh, a startup focused on cache optimization. Meanwhile, at the hardware level, data centers are optimizing how they use different memory types, such as determining when to use DRAM chips instead of HBM (High Bandwidth Memory). Higher up the stack, users are also structuring their model swarms to take advantage of shared cache. Ultimately, companies that manage these memory tiers successfully will be able to run applications that previously did not seem profitable.

Why it matters

As AI infrastructure costs shift from pure compute to memory management, companies that optimize how they handle data in cache will gain a significant competitive advantage by reducing token consumption and improving profitability.