AI infrastructure has entered the era of the “AI factory.” Inference workloads dominate AI compute and hyperscalers are increasingly focused on profitable tokens/s generation. Inference accounts for roughly two-thirds of all AI compute in 2026, up from about a third in 2023, and the trend line only points up.1
The AI fabric has a real impact on tokens per second, and the switch that connects multiple GPUs into a unit of compute is a significant conventional bottleneck to token throughput. When the fabric switch has enough radix and bandwidth to give every GPU single-hop, line-rate access to every other GPU, the interconnect allows compute to drive throughput with minimal overhead. When it doesn’t, the fabric itself becomes the bottleneck.
The high-radix, high-bandwidth Scorpio™ X-Series 320L Smart Fabric Switch breaks fabric bottlenecks by connecting larger numbers of GPUs in a single-hop topology than conventional state-of-the-art solutions can. To show how much a single high-radix switch can improve tokens/s performance:
- We built two equivalent 8-GPU platforms that differ only in the fabric switch topology, and
- We ran them head-to-head across three representative inference workload types.
This post walks through why a Scorpio X-Series-based topology wins across all of them.
What Types of Workloads Are Driving the Inference Era?
Not all inference traffic is identical. The ratio of input tokens to output tokens for a given request determines how each workload stresses the platform, particularly how that workload is bound by compute, by memory bandwidth, or by interconnect latency. We tested three workload profiles that span this range.
Decode-Heavy Workloads
Decode-heavy workloads take a short prompt and generate a long response. We tested a workload with a 1:8 input-to-output ratio (128 tokens in, 1,024 tokens out).
Where these show up: Conversational assistants, code and long-form drafting
Why they stress the platform:
- They’re latency-bound; the system runs one forward pass per output token
- Collective latency is paid per token, so scale-up interconnect inefficiency compounds across thousands of forward passes
Prefill-Heavy Workloads
Prefill-heavy workloads take a large input and return a very short answer. We tested a prefill-heavy workload with a 1000:1 input-to-output ratio (3,000 tokens in, 3 tokens out).
Where these show up: Content screening, classification and routing, extraction and scoring
Why they stress the platform:
- They’re both compute-bound and bandwidth-bound; the whole prompt is processed in parallel and saturates throughput
- Time-to-first-token effectively becomes end-to-end latency, since there are only a handful of output tokens
Balanced Workloads
Balanced workloads combine a large context with a substantial answer. We tested a balanced workload with a 7:1 input-to-output token ratio (3,500 tokens in, 500 tokens out).
Where these show up: Agentic workflows, retrieval-augmented answers, code assistants working over a repository
Why they stress the platform:
- They balance compute-heavy prefill with bandwidth-heavy decode in a single request
- Large context reads per request make cache reuse and interconnect bandwidth the deciding factors in effective throughput
- Prefill and decode contend with each other, so a long prompt entering the batch can stall other users’ token generation
Where Conventional Switches Fall Short
Conventional PCIe fabric switches max out at around 144 lanes. Building an 8-GPU cluster on that topology requires two switches, each connected to its own CPU host, with the eight GPUs split into two four-GPU domains. GPUs on one side of that split do not have a direct path to GPUs on the other side; traffic has to detour back through a CPU root complex adding an additional host-bridge hop and potential cross-NUMA node inter-socket transfer to reach its destination, limiting the bisectional bandwidth and adding latency to every transaction that crosses the boundary.
That communication overhead compounds as the number of concurrent inference requests grows. High concurrency exacerbates the very stresses described above: It adds latency to the per-token forward passes that decode-heavy workloads depend on, it competes with the interconnect bandwidth that balanced workloads need for large-context reads, and it slows the bulk, parallel data movement that prefill-heavy workloads require to hit throughput targets.
The Scorpio X-Series 320L Smart Fabric Switch provides single-hop, line-rate, peer-to-peer connectivity for multiple-GPU platforms. A single Scorpio X-Series switch maximizes the number of concurrent requests a platform can process without paying a multi-switch latency tax.
Head-to-Head: How We Tested Scorpio X-Series Against the Baseline Topology
To isolate the fabric’s impact on real inference throughput, we built two 8-GPU platforms that are equivalent everywhere except the fabric, then measured output tokens per second across a concurrency sweep from 1 to 512 requests, for each of the three workload profiles above, across four large language models.
Figure 1: Scorpio X-Series 320L Smart Fabric Switch vs. conventional 2x 144L fabric switch topologies
The Results: Scorpio Increases Tokens per Second Across the Board
Across every model, every workload type, and every concurrency level we tested, the Scorpio X-Series 320L platform delivered more output tokens per second than the dual-144L baseline. We saw gains of up to 89% depending on model and workload, with the most consistent advantage seen for the prefill-heavy workloads that saturate interconnect bandwidth.
At the highest concurrency level tested, Scorpio’s advantage held at 30% or better for every single model-workload combination, and the dense models (which push more of their I/O through all-gather and reduce-scatter collectives) saw the largest gains.
Decode-Heavy Workloads
For chat and reasoning-style traffic, where the fabric pays a latency cost on every one of the thousands of forward passes needed to generate a long response, Scorpio’s single-hop connectivity translated directly into more completed tokens per second.
Figure 2: Decode-heavy workload performance
Prefill-Heavy Workloads
Content screening and classification workloads process almost the entire prompt in parallel with only a handful of output tokens, so throughput is entirely a function of how fast the fabric can move bulk data across all 8 GPUs. This is where Scorpio’s advantage was most consistent, holding at 38% or better across the full concurrency sweep for every model.
Figure 3: Prefill-heavy workload performance
Balanced Workloads
Agentic and retrieval-augmented workloads place compute-heavy prefill and bandwidth-heavy decode in the same request, so any contention on the fabric shows up as lost throughput on both phases at once. Scorpio’s higher cross-sectional bandwidth kept large-context reads from stalling other users’ decode.
Figure 4: Balanced workload performance
What It Means for AI Factory Economics
These results confirm what the workload analysis above predicts: whenever a workload forces GPUs to move data across the fabric — whether that’s per-token latency in decode-heavy chat traffic, large-context bandwidth in agentic workflows, or bulk parallel reads in prefill-heavy classification — a higher-radix, single-hop switch turns directly into more tokens generated per second on the same GPUs, at the same power envelope. For hyperscalers and neoclouds counting every watt and every port against the cost of a token, that has a direct impact on the profitability of their AI factories.
This test is one entry in a broader pattern of results we’re seeing as we push every layer of the fabric — switches, interconnect, and memory — to keep GPUs fed and inference racks productive. For a look at how we improved time-to-first token and tokens per second by speeding up KV cache offload, read my colleague’s recent blog about creating a fabric-attached memory tier with Astera Labs Leo X-Series Smart Memory Controllers.
References:
1. Deloitte, “Deloitte 2026 Technology, Media & Telecommunications Predictions: Narrowing the Gap Between the Promise of AI and its Reality,” accessed October 2026, https://www.deloitte.com/us/en/about/press-room/deloitte-2026-tmt-predictions.html.