In the early days of chat-based LLMs, the memory wall was driven by linear context growth. For each individual inference workload, expanding context length pushed available memory demand beyond capacity.
But we are now in the agentic era. Agentic workloads operate on dynamic loops: They generate reasoning tokens, call external tools and APIs, parse returned data, and append state updates across dozens or hundreds of execution turns. Operating in parallel, the CPU tokenizes requests, performs RAG lookups, manages session state, and prepares prefill metadata while the GPUs serve the system’s many users. This continuous execution cycle fundamentally reshapes how the KV cache strains GPU and CPU memory.
As AI inference moves from isolated workloads to always-on agentic systems, the memory path — not just compute — determines how quickly users see a response.
Under these conditions, how do you create a memory path that:
- Can help orchestrate a massive amount of simultaneous agentic inference?
- Minimizes latency and reduces time to first token (TTFT)?
- Maximizes throughput to increase tokens per second?
- Efficiently uses CPU cores and attached memory?
The answer is an innovative architectural solution: a fabric-attached memory tier using PCIe® or platform-specific protocols made possible by Astera Labs’ Leo X-Series Smart Memory Controllers and Scorpio™ Smart Fabric Switches.
Traditional KV Cache Offloading Creates New Bottlenecks
As an example that illustrates the scale of the KV cache capacity gap for long contexts, let’s take:
- An AI server platform with four GPUs
- 10 concurrent inference sessions, each generating 128,000 tokens
The 10 concurrent sessions (without shared context) mean you need 1,280,000 tokens of KV capacity to process these operations without added latency from congestion and queuing. This demands significantly higher capacity than this system’s GPU high-bandwidth memory (HBM) holds.
When GPU HBM fills up, infrastructure architects typically turn to several strategies for offloading KV cache. However, these strategies suffer from a mix of capacity and latency issues.
- Host CPU DRAM: This solution does enable data rates fast enough for AI Inference workloads, but CPU DRAM capacity per GPU remains severely limited when running dense multi-GPU nodes and introduces latency and bandwidth bottlenecks when aggregating GPU requests to the CPU.
- NVMe/SSDs: These are highly scalable and lower cost than memory, but high I/O latency for this tier degrades TTFT. This can also lead to cache retrievals that are slower than recomputing tokens on the GPU, particularly for the decode phase which is inherently memory bottlenecked and can adversely impact tokens per second. Additionally, SSDs may run into challenges handling the large amount of daily writes which can exceed the rated endurance of the SSDs resulting in further performance degradation or repair events.
- Network solutions: Remote direct memory access (RDMA) and transmission control protocol (TCP) are suitable for distributed servers (e.g., vLLM cluster sharing), but transferring gigabytes of KV data across network interfaces creates congestion and high latency.
The standard solutions for AI inference have been KV offloading to CPU DRAM or to NVMe SSDs. Libraries like LMCache extend the KV pool into CPU memory and restore KV blocks from DRAM back to GPU as sessions advance.
This works well until CPU memory bandwidth becomes scarce. A typical KV restore for a 32,000-token agentic session might need to move 8GB of KV blocks from DRAM to GPU. When CPU DRAM bandwidth is shared between multiple GPUs and across other CPU workers to serve multiple agents simultaneously, arbitration overhead and queuing will limit the effective throughput even further.
GPU Utilization Suffers as Systems Scale
Time to first token variability compounds at rack-scale and beyond in ways that single-server benchmarks don’t reveal.
When LMCache restores a KV block from CPU memory to GPU HBM, the decode phase on that GPU stalls until the restore completes. Under CPU DDR5 memory, our 32,000-token sessions each take ~119 ms longer to start than they would with a dedicated memory tier for offloading outside the compute path. During those 119 ms, the GPUs can potentially be sitting completely idle.
Multiply this latency across 10 concurrent sessions and the GPU is effectively idle for a material fraction of every scheduling window. Effectively, the idle GPUs increase TTFT and reduce utilization, and CPU efficiency falls as well because contention to the same memory subsystem consumes bandwidth needed for tokenization, RAG, and orchestration.
Leo X-Series and Scorpio Break the KV Cache Wall With a Dedicated PCIe Memory Tier
To avoid the cascading latency and bandwidth bottlenecks experienced with conventional KV cache offloading to CPU DRAM, Leo X-Series – Astera Labs’ new fabric-attached memory controller – pairs with our Scorpio 320 lane X- and P-Series Smart Fabric Switches to connect a dedicated memory tier directly to the GPU scale-up or the host-side PCIe fabric.
Importantly, isolating KV cache offload in a dedicated memory tier is critical for a highly efficient bandwidth path. The dedicated bandwidth provided by the Leo X-Series Smart Memory Controller over PCIe and platform-specific protocols delivers sequential, cache-friendly restores of KV cache while avoiding CPU memory contention.
In this topology, a Scorpio P-Series Smart Fabric Switch provides high-performance PCIe peer-to-peer transfers to bypass the CPU root complex and isolates KV traffic from CPU memory arbitration, which preserves bandwidth and latency for concurrent dense-model transactions.
CPU DRAM vs. Leo X-Series PCIe-Attached Memory
| Dimension | CPU DRAM-backed KV cache | Leo X-Series PCIe-attached memory |
| Bandwidth path | Shared DDR5 path; can saturate under host and co-located workloads | Dedicated bandwidth over PCIe x16 with consistent memory throughput; isolated from host DRAM |
| Latency behavior | Latency rises with arbitration and queueing above high channel utilization | Consistent low-latency access under concurrent transfers |
| Dense-model traffic | KV transfers contend with dense-model traffic through the CPU memory subsystem | KV transfers over PCIe peer-to-peer transfers bypass the CPU root complex |
| Capacity economics | Competes for capacity in the host-memory pool | Supports higher capacity and low-cost options such as pooled memory |
How We Tested Dedicated PCIe KV Caching
That example AI server from earlier? We built it and tested this setup with 32,000-token, multi-turn sessions.
Hardware:
- System: 1S server, 128 logical CPUs (64 physical cores)
- GPU: 4× NVIDIA H200 143 GB (TP=4)
- CPU DDR5 Memory: 1 TB, NUMA node 0 (~200 GB/s peak bandwidth)
- Fabric-attached Memory: Astera Labs Leo X-Series Smart Memory Controller with 256GB of memory
- Kernel: Linux 6.8.0-138-generic
- PCIe Switch: Astera Labs Scorpio P-Series 320L Smart Fabric Switch
Software:
- vLLM: v1, tensor parallel degree 4
- LMCache: v0.4.2 + Patch (Patch to support offload to Leo X-Series memory)
- CPU Memory stressor: Each thread with two 4 GB buffers, traffic to emulate CPU utilization across Agentic workloads
- Telemetry: Intel PCM (pcm-memory) sampling DDR5 channel read/write BW and nvidia-smi sampling GPU utilization
Benchmark workload:
- Model: Qwen2.5-32B
- 10 user sessions, each with a 32k-token growing context every turn simulating an agentic workload
- Turns 2+: LMCache restores KV from the offload pool
- CPU memory stressors simulate CPU memory contention by tokenization, RAG and orchestration in agentic workload. Stressor worker threads continuously run traffic between two 4 GB buffers per thread.
Figure 1:KV Cache offload datapath with CPU memory and with Leo X-Series (see above for exact hardware details)
Latency Benefits: 62% Faster Time to First Token with Leo X-Series
For this configuration running the above workload for KV cache, with tensor parallelism of four, and 10 sessions with five turns, the average warm turn on a conventional CPU-DRAM test delivered a 246 ms TTFT.
When we routed the KV cache through PCIe-attached Leo X-Series Smart Memory Controllers, we reduced the warm TTFT to 127 ms. With 32 stressor threads added to the test to simulate a dense model for a multi-tenant environment, this gain was even more pronounced. These stressor threads running on the CPU caused a DDR5 memory bottleneck, pushing the CPU DRAM-based TTFT up even further to 413 ms while it held at just 158 ms for the Leo/Scorpio topology. The Leo/Scorpio PCIe memory tier thus resulted in 62% faster time to first token under these conditions.
Figure 2: Leo X-Series memory demonstrating 62% faster TTFT
Throughput Benefits: 22% More Tokens per Second With Leo X-Series
The approach we described here bypasses the periods of system idleness experienced with CPU DRAM KV cache offloading. This means the GPUs in the configuration can run much larger batch sizes, keeping compute pipelines fully saturated and eliminating “stranded” GPU cycles.
When the inference engine instantly offloads an agent’s KV cache over low-latency links to Leo X-Series memory, GPU HBM is freed up to process other active inference requests. Once the tool returns, the cache is swapped back quickly without stalling the GPU compute pipeline.
This approach results in 22% more tokens per second for a typical agentic workload (10x sessions, each session with 1 cold turn followed by 5 warm turns). For deeper agentic sessions (20–50 warm turns per cold), we continued to observe a 22% improvement in token throughput for the system. Under realistic multi-tenant DDR5 memory pressure, TPS improvement is expected to widen further across the same workload range.
PCIe-attached memory also removes the shared-resource coupling that causes unpredictable spikes in latency when CPU and GPUs need to access the same memory tier. Establishing predictability with a dedicated KV cache memory tier enables higher steady-state system utilization and throughput while improving user experience.
Figure 3: Leo X-Series demonstrating increased GPU utilization
Capacity Scaling and Optionality Benefits
The system we designed using one Leo X-Series Smart Memory Controller per GPU added 256 GB of memory for each of the 4 devices, available for KV cache. This is sufficient to handle multiple concurrent sessions of 128,000-token workloads, but what about future large-context workloads of 1 million tokens or more?
Scaling memory even further for workloads of any size is straightforward with Leo X-Series. You simply add more Leo X-Series devices on additional PCIe slots. With this topology, you can scale memory beyond the DDR5 DIMM slots and channel count fixed at board design time. Leo X-Series can support up to 2TB of expanded memory per device.
And in our DDR5 supply-constrained environment, Leo provides another important advantage: the 62% faster TTFT and 22% more TPS improvements were measured just by moving where the KV cache is allocated. This means system implementations can benefit from Leo X-Series without increasing total system memory capacity to ease cost and supply challenges.
Latency SLAs Become Deterministic With Leo X-Series
Latency SLAs are notoriously difficult to target for DDR5-backed KV offload. A 99th percentile TTFT is effectively “the worst-case memory pressure the server will see,” a number that can change significantly depending on co-tenant behavior, OS background activity, and system load.
With Leo-X series, the KV restore path has a fixed bandwidth envelope independent of CPU activity. This means that any variations in TTFT are limited to GPU-side factors (prefill compute, link transfer rate, scheduling jitter), which are bounded and predictable.
No Application Changes Required
To support Leo X-Series memory expansion, LMCache requires a small set of targeted additions at the memory allocator level but no changes to vLLM. These changes register the Leo-X series memory as IO memory, giving the GPU a virtual address that resolves directly to the Leo-X Series device’s physical address over PCIe.
Break Memory Bottlenecks for Next-Gen AI With Scorpio and Leo
AI infrastructure is entering a memory-scaling era to keep pace with the agentic era accelerating on the modeling side. Novel KV cache offloading solutions are necessary to bridge the widening gap between the tokens today’s workloads generate and GPU HBM capacity.
Together, Leo X-Series and Scorpio X-Series form a solution that enables:
- An efficient scale-up path: Scorpio enables PCIe peer-to-peer transfers that bypass the CPU root complex, helping match bandwidth to GPU demand while supporting concurrent dense-model transactions. The end result, 62% faster time to first token and 22% more tokens per second, meaning faster responses for users and higher rack-scale utilization for Agentic AI workloads.
- Scalable deployment: Leo X-Series provides a full software stack solution with COSMOS infrastructure, including GPU DMA operation, proven fleet management, data center RAS, and security features deployed at CSPs.
- Dynamic KV pool provisioning: This solution lends itself to scale-up fabrics by allowing memory to be dynamically sliced and assigned to specific GPU compute clusters on demand. If a node experiences a burst of high-concurrency context requests, the fabric can dynamically allocate gigabytes of pooled Leo memory to the node’s KV cache without physical hardware reconfiguration.
- Elimination of stranded capacity across hosts: Unused host memory on one side of the fabric can be borrowed by GPUs on another side, maximizing system-wide memory utilization across the entire rack.
- Proven RAS and security capabilities: Astera Labs’ COSMOS telemetry and management software suite, deployed across millions of servers worldwide, provides fleet-wide visibility, enhanced error reporting, and hyperscale-grade memory health management to extend memory service life.
As token sizes, contexts, agent steps, and users grow, each session adds KV state and each restore adds memory fabric pressure. A dedicated KV cache tier created with Leo and Scorpio turns that pressure into higher accelerator utilization, shorter and steadier TTFT, and consistent, efficient next-generation agentic AI performance.