At a glance:
- In-Network Compute (INC) is the execution of collective operations at network switches.
- INC is most impactful for inference workloads, where performance, latency, and efficiency directly shape user experience and deployment cost.
- Scale-up is the best INC use case because the most demanding inference systems depend on tightly coupled, high-bandwidth, low-latency communication within the node or pod.
- Four factors must be accounted for before INC is widely adopted across the AI industry.
- Scorpio™ X-Series Smart Fabric Switches bring INC to open scale-up fabrics, improving tokens per watt performance.
As AI model sizes have grown to trillions of parameters and deployment has shifted from training to production inference, two trends are driving urgency for the network switch fabric to become the major driver of AI system performance.
- Inference workloads are surpassing training workloads in total volume1, and operations essential to completing them sit on the unavoidable critical execution path through the switch. Even microseconds of latency in these operations translates directly to token latency for the end user.
- Compute capability is outpacing interconnect’s ability to facilitate it.
Compute capability has scaled at a rate of 3x every two years, compared to a 1.4x increase in interconnect bandwidth over the same time frame.2 For legacy architectures, this means an inference latency gap grows with every GPU generation, and token latency cannot be meaningfully improved by faster and faster GPUs straining against this infrastructural bottleneck.
INC addresses this problem by improving communication latency, speeding up collective operations by nearly 2x in 8-GPU clusters today, with more significant gains possible in future larger clusters. Read on to learn why this is and how prepared the AI industry is to embrace this vital solution at scale.
What Is In-Network Compute?
In-network compute refers to the execution of collective operations within network switches.
This definition is intentionally precise. The switch is the topological point of convergence for network traffic. It’s the natural location for collective operations because all traffic must pass through it, it sees aggregate traffic from multiple endpoints simultaneously, and switch-resident compute eliminates redundant data traversal by reducing traffic at the point of aggregation.
INC is particularly effective for collective operations like AllReduce that are essential to today’s inference workloads. AllReduce is a two-step operation that coordinates matrix multiplication across the multiple accelerators that make up an AI cluster. When inference model matrices are very large, they must be distributed across a number of accelerators. To generate a token with a distributed model, aggregation called ReduceScatter is performed across all involved accelerators, and local copies of the output are distributed back to each of them in a step called AllGather. AllReduce is a ReduceScatter step followed by an AllGather step, resulting in each accelerator having an identical copy of the final result.
With INC, the switch can perform the reduction at the exact topological point where all inputs converge, using aggregate switch bandwidth far exceeding any individual GPU’s link bandwidth.
Why Is INC More Powerful for Inference Than Training?
The data-parallel gradient synchronization workloads that previously dominated for AI training were GB-scale, bandwidth-bound, and often overlap-able with computation.2 Conversely, inference AllReduce workloads are characterized by tensor-parallel activation reduction, which are much smaller message sizes (KB-scale), latency-bound, and occur on the critical path of every generated token.
As a result, all latency reduction enabled for inference AllReduce by INC directly impacts the end user’s experience.
For a detailed look at how we expect INC speedup to scale across message sizes and models, read our technical deep dive companion blog.
Why Does INC Excel in Scale-Up Architectures?
Scale-up switch fabrics are extensions of the compute’s domain of influence. Scale-out network switches are the network’s domain of influence.
In scale-up architectures (built on PCIe, NVLink, or UALink protocols), the GPU is the sovereign entity. The switch extends its reach and is managed by the GPU driver. In scale-out architectures (built on Ethernet or InfiniBand protocols), the switch is an independent entity managed by the network team.
This distinction has profound implications for INC deployment:
| Dimension | Scale-Up Switch INC | Scale-Out Switch INC |
| Management | GPU driver manages switch | Network OS manages independently |
| Trust domain | Single domain; hardware-assisted | Cross-domain; software-managed |
| Failure handling | Co-managed with compute | Independent fault domain |
| Operator ownership | Compute team | Network team |
Table 1: INC in scale-up switches vs. scale-out switches
INC is more tractable in scale-up because the compute node already owns the switch, trust is established, and failure semantics are unified.
The limited real-world adoption of scale-out INC proves this point. InfiniBand SHARP is a proven production switch INC system with up to 5.1x AllReduce latency reduction3, yet it remains underdeployed because only a handful of providers and production workloads have successfully enabled and tuned SHARP.4 This roadblock stems from operational complexity at the compute-network domain boundary, which is precisely the boundary that scale-up switch INC eliminates.
By contrast, INC is now available for every scale-up protocol, and it is entering every scale-up switch standard: NVLink (supported by NVSwitch products), PCIe 6.0 (supported by Scorpio X-Series 320 Lane Smart Fabric Switch), and UALink (published in spec 2.0).
- NVSwitch/NVLink represents the most mature production switch INC deployment. NVSwitch 4.0 (Blackwell, 2024) delivers 1.8 TB/s per GPU with in-switch AllReduce already reducing inference token latency in production B200 deployments.
- UALink (Ultra Accelerator Link) Switch is an open alternative. The UALink Consortium (now exceeding 115 member companies5) published UALink 2.0 specifications in April 2026, supporting up to 1,024 accelerators per switch domain at 212.5 GT/s with deterministic sub-microsecond latency. INC in the UALink Switch is a first-class design goal.5
- Astera Labs’ Scorpio X-Series 320L Smart Fabric Switch features in-network compute capability. It can offload AI collective communication inside the fabric to reduce accelerator I/O and improve performance-per-watt.
Is the Industry Ready for In-Network Compute?
INC-capable switches are here and more are on the way. The AllReduce fraction of inference latency is growing. But are we as an industry ready to utilize in-network compute to its fullest potential?
The SHARP deployment experience4 shows that hardware availability does not guarantee deployment readiness. Software and systems need to be ready before the hardware can be broadly adopted. While INC-capable switch hardware and firmware provide the underlying execution engine, successful deployment also requires updates across the surrounding software stack: collective communication libraries, GPU and memory-path integration, fabric management, and INC resource management. These layers connect AI frameworks and GPU workloads to the switch, configure the fabric, expose INC capabilities, and manage shared resources across jobs.
Enabling this software stack is a critical step for INC adoption because hardware and firmware alone do not make INC usable in production. Without the required software integration, workloads cannot reliably invoke INC operations, the system cannot allocate or monitor INC resources, and operators cannot deploy the capability at cluster scale with predictable performance, isolation, and lifecycle management.
Figure 1: Scale-up fabric software stack and INC support
INC-Aware Collective Libraries
INC-aware collective libraries are the software interface between AI frameworks and in-network compute. They expose familiar operations such as AllReduce, ReduceScatter, AllGather, and All-to-All while selecting algorithms that can offload supported communication and reduction work to the switch. The libraries should preserve compatibility with existing applications, provide graceful fallback paths, and expose controls for performance, topology, and workload-specific tuning.
GPU and Memory-Path Integration
GPU and memory-path integration enables collective data to move efficiently between accelerator memory and the INC-capable fabric. This layer must support GPU-resident buffers, GPUDirect-style transfers, address translation, registration, ordering, and completion semantics so data can reach the switch without unnecessary host-memory copies. It also defines the software and hardware interfaces needed to make INC operations transparent and reliable to GPU applications.
Fabric Management
Fabric management connects the software stack to the switch and turns INC capabilities on in the deployed topology. It configures ports, routes, address ranges, collective contexts, and feature-specific parameters, while validating link health and compatibility across the fabric. This layer should also provide operational visibility and safe reconfiguration so INC can be enabled without disrupting ordinary traffic. The fabric management, further, discovers the topology, allocates and reserves resources for jobs, tracks collective lifecycles, enforces isolation and policy, and releases capacity when work completes. This control plane orchestrator is essential for coordinating multiple concurrent workloads and for making INC performance predictable under production conditions.
Security
When INC is enabled, the data is no longer opaque to the switch and fabric. If the data is expected to be protected by encryption, then the security assumptions and solutions must be redesigned to keep the unencrypted data secure as it passes through the fabric with INC.
The Hardware and Firmware Are Ready; Are You?
Make your system software and fabric stack deployment-ready now by enabling the libraries, GPU buffer access, fabric controls, and INC management needed to turn INC capability into production-ready AI cluster performance. Astera Labs will partner with you to enable the INC requirements in your infrastructure.
Scorpio X-Series Enables INC to Meet Our Current Moment
The Scorpio X-Series 320L Smart Fabric Switch is the industry’s largest open, memory-semantic fabric switch, engineered for this moment as inference workloads and their growing AllReduce demands move to the fore. Its high-radix architecture simplifies scale-up topologies by reducing hops (up to 80 GPUs with single-hop), while In-Network Collective engines accelerate collective operations. Scorpio X-Series brings in-network compute to open scale-up fabrics, offloading collective communication to reduce accelerator I/O and improve performance-per-watt.
Want to see how else Scorpio accelerates token throughput for the workloads driving AI today? Read our blog post on the impact of Hypercast™ for mixture-of-experts models.
References:
1. McKinsey & Company. The Future of AI Workloads.” Analysis of global data center demand by workload type through 2030. [Online] 2026. https://www.mckinsey.com/featured-insights/week-in-charts/the-future-of-ai-workloads.
2. Du, Aojie Jiang and Kang Zhu and Zhiheng Zhang and Zhengxu Su and Juntao Liu and Yuan Du and Li. A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Network (SCIN). arXiv. [Online] 2026. https://arxiv.org/abs/2603.28239.
3. Ramesh, Bharath, et al. Scalable MPI Collectives Using SHARP: Large Scale Performance Evaluation on the TACC Frontera System. 2020 Workshop on Exascale MPI (ExaMPI) [Atlanta, GA, USA], 2020, pp. 11–20. https://doi.org/10.1109/ExaMPI52011.2020.00007.
4. SemiAnalysis. The GPU Cloud ClusterMAX Rating System: How to Rent GPUs. [Online] March 26, 2025.
5. UALink Consortium. Ultra Accelerator Link (UALink) Consortium Publishes Four Specifications Defining In-Network Compute, Chiplets, Manageability and 200G Performance. Press Release. April 7, 2026. https://ualinkconsortium.org/news/.