Squeezing More Out of Every Byte
Google Fellow Dave Patterson had the line of AI Infra Summit 2026: We’re entering a “memory-centric era.” The pricing backs him up. Analyst Jim Handy of Objective Analysis tells us that DRAM average price increased 250% from 2Q25 to 2Q26. Businesses are slowing down because they simply cannot get enough of it.
But everyone walked into the summit already knowing about the shortage. The real story was the response: engineering around memory, not just buying more of it. Here’s what I took away from the conference.
Prioritizing the Crunch
Patterson’s framing held up everywhere else on the agenda. It’s the kind of line that sounds abstract until you hear it backed up by the people actually buying and building the hardware. Amazon’s Peter DeSantis discussed in a talk with SemiAnalysis’ Dylan Patel how AWS’ Trainium chips now lead against every other accelerator in what Dylan called the single most important metric: memory-bandwidth-per-dollar. Getting there meant investing in the memory stack as deliberately as AWS invests in compute itself.
Samsung made a version of the same argument from the memory side. Their session on composable AI memory nodes laid out a tiering scheme, with something the company is calling HBM over optical sitting as a middle tier, and a fully composable memory pool that lives independent of any single GPU. The pitch, boiled down: not every byte of context needs to sit in the most expensive memory you own, and building a system that can route data to cheaper tiers without tanking performance is its own kind of capacity gain.
Our own Thad Omura framed this same problem from the silicon side in his talk.
Solving Compute and Memory Bottlenecks at the Same Time
Thad’s framing was that idle capacity, whether that’s a stalled GPU or memory that’s provisioned but not reachable, is an “invisible wall,” a constraint infrastructure teams have to actively design around rather than assume will take care of itself. Stranded memory and stalled compute, in his telling, are really the same problem wearing two different faces: A GPU sitting idle is usually just memory failing to get where it needs to be, fast enough. He pointed to a specific, practical response many hyperscalers are already taking: reusing the DDR4 memory sitting in servers that are being retired, rather than letting it go to waste. By our math, a single retiring server can carry roughly $30k dollars worth of memory that’s still perfectly usable, just sitting in the wrong generation of hardware.
Rakesh Vitta dug into exactly why that idle time happens in a technical breakdown blog we published the week of the summit. He walked through what happens when an inference server’s GPU memory fills up during a multi-turn agentic session: When the KV cache has to restore from CPU memory back to the GPU, that GPU sits stalled until the restore finishes, and across concurrent sessions, that stall time adds up fast. In his example, a single four-GPU server running ten concurrent agentic sessions at 128,000 tokens each needs one point two eight million tokens of KV cache capacity, more than the GPU’s own high bandwidth memory can hold on its own.
That’s the exact problem our Leo X-Series Smart Memory Controllers and Scorpio™ Fabric Switches are built to solve. By giving that KV cache a dedicated memory tier directly on the fabric instead of routing it back through CPU memory, testing in Rakesh’s post showed sixty two percent faster time to first token and twenty two percent more tokens per second.
The Takeaway
Nobody at AI Infra Summit 2026 was claiming victory over memory scarcity. Everyone was building around it, from hyperscalers reusing retired hardware to memory makers rethinking where data lives, to fabric-level fixes that get more out of every byte already on hand.
That’s probably the more durable story anyway. Supply isn’t loosening up on its own timeline anytime soon, and the industry spent several days in Santa Clara proving this crunch can be addressed through architecture rather than pure purchasing.