Inference demand is growing faster than the industry can build for it. Modularity at every level, from chip to grid, is how capacity gets built fast enough.
This theme ran through the 2026 OCP Global Summit. AMD CEO Dr. Lisa Su called 2026 “the year of inference,” with agentic traffic up 14x this year. Dr. Dimitrios Ziakas, CTO of Hyve Solutions, pointed to a 7x year-over-year jump in Google’s token consumption through May.
Workloads, silicon and data center infrastructure must now evolve at the same pace. A data center design fixed for 20 years, or even three, is now a liability. Astera Labs announced the Orion Rack-Scale Reference Architecture at OCP this year in direct response to this need: modular, adaptable inference compute that comes online at the speed and scale the workloads demand.
Why Modularity Matters to the Data Center
Over the past decade of cloud server deployments and into the AI era today, there have been three key benefits to designing for modularity:
1. Reusability
Modularity starts with common building blocks that work in many places and many configurations. For AI infrastructure, modular thinking means viewing hyperscale deployments as learning opportunities in which components are tested and proven for adaptability to additional configurations down the road.
2. Configuration Flexibility
Custom hardware tuned to one workload is efficient, but it does not scale easily to other workloads. Inference cloud providers, especially outside the hyperscalers, cannot start from scratch for every deployment. So modularity provides proven, flexible components that adapt quickly to new configurations. That built-in flexibility lets operators keep pace with fast-changing workloads, because hardware can’t change overnight.
3. Speed to Market
Together, reusability and flexibility let system designers move faster. Each successful modular deployment builds confidence in the next one. With that confidence, teams decide faster and spend less time re-validating each new build.
From Chip to Grid: How OCP Drives Modularity at Every Level
OCP sets standards from “chip to grid,” from the silicon, through the rack and the facility, to the utilities that serve them. This vision is impractical without modularity. No single vendor owns every layer between a chiplet and a data center campus, and each layer changes on its own schedule.
Modularity at each level enables shorter time-to-market and controlled TCO. Modular building blocks displayed at OCP 2026 can help the data center ecosystem rapidly adapt to new needs.
Chip Level: Modularity Inside the Package
Modularity for AI infrastructure starts inside the package. OCP’s Open Chiplet Economy creates repeatable, modular standards and interfaces at the silicon level similar to those that exist at the board level.
Establishing industry standards for the physical envelope and interfaces of an I/O chiplet enables vendor optionality at a crucial moment for the ecosystem when compute silicon is diversifying as fast as inference racks are multiplying. Many accelerators have strikingly similar interconnect requirements and implementations that could benefit from a common I/O chiplet while the rest of the accelerator remains differentiated.
Board and System Level: Compute-Agnostic Scale-Up Systems
OCP’s long-established standards for motherboard and adapter card footprints continue to evolve as well to meet AI’s changing needs. The DC-MHS and OCP Mezzanine card specifications are excellent examples of this. At OCP 2026, our own Christopher Blackburn spoke about the Open Adapter Card (OAC), the successor to the now-7-year-old Open Accelerator Module (OAM).
The OAC is a vertical card with a backplane connector, about 1.75x the PCB area of a full-height, full-length PCIe card, and up to 5 kW per module with direct liquid cooling. This new form factor is intended to provide for increasing amounts of HBM, compute dies, and on-board networking required by today’s workloads. At the same time, it once again enables a “common slot” system architecture that we achieved with the PCIe CEM card and OAM specifications.
Rack and Pod Levels: Multiple Standard Footprints for the Unit of AI Compute
The OCP Global Summit is at heart a data center systems conference, and for OCP member organizations who design the physical space these systems will fill, the rack itself is the most important modular unit as increasingly the rack has become the system.
While AI scale-up does continue to outgrow the rack, the rack itself is growing to keep up. AMD’s Helios and Helion racks are built on the OCP standard now called Open Rack Wide, and this standard itself is evolving to accommodate more power, cooling, and connectivity per rack.
Beyond the rack, the Open Data Center for AI initiative is standardizing pod dimensions, inter-rack cabling specs, and rack footprint spacing within the data center to reduce the number of time-consuming bespoke decisions that must go into each building blueprint. Innovations like Microsoft, Google, and Meta’s co-developed Mt. Diablo power rack sidecar look to both scale utilities for new data centers and to provide avenues to adapting legacy ones to evolving needs.
The interconnect between racks is typically modular by design in some fashion as well. Cables are by definition modular solutions, and the expanding array of connectivity options across both copper and optical interconnects provide crucial design optionality to optimize reach, speed, and power for next-generation signaling protocols.
Building Level: Campuses Deployable at Scale
Finally, the very building envelope itself for the next generation of inference data centers must be a modular, rapidly repeatable structure. This can take the form of reproducible blueprints for conventional building campuses, or in an even more extreme case, Meta’s announced plans for tent-like data center structures that they propose to stand up in only 3-5 months each.
The Software Stack: Total Visibility Across Modular Solutions
While modularity enables optionality and interchangeability, components must remain anchored to a common software layer to make the entire architecture operable and serviceable at scale. Software is the common pane of glass that data center operators must use to orchestrate, manage, and configure all these disparate parts.
This software stack can take multiple forms, from vendor-aligned turnkey solutions to API connectivity that delivers multiple data streams into existing platforms and deployment mechanisms. Either way, enabling visibility across the data center is essential for a thriving hardware ecosystem.
From Project to Product: Modular Hardware Demands Repeatable Implementation
In their talk “Modular by Design: Building AI Clusters from Factory-Built Units,” Alex Akesson and Bill Stube from Crusoe called on the industry to transform data center construction “from a project to a product.” Standing up inference capacity needs to be a repeatable, standardized framework rather than a one-off bespoke build.
To me, this is where the industry has the furthest still to go in embracing the modular data center. Every modular rack brought online needs a standardized playbook for scale-up physical implementation, from the scale-up fabric itself to the cables and connectors used to link it all together. Every modular rack needs a similar playbook for pod- and building-level deployment. Implementation at every level must be streamlined, predictable, and timely.
In his market analysis session, IDC’s Chris Drake argued that the industry’s emphasis must move from open building blocks to integrated reference architectures that bring different specifications together in one system, with success measured by interoperability among competing suppliers. I’d argue that the specific kind of reference architectures we need today are actually rack-scale reference implementations.
Orion: Purpose-Built for Modular Deployment
Our recently announced Orion Rack-Scale Reference Architecture is precisely the kind of robust, production-grade reference architecture that will be crucial to the next phase of AI inference.
- It’s a compute-agnostic architecture designed to accommodate varied XPU and CPU options that enable configuration flexibility to efficiently support a variety of workloads.
- We’ve provided reference designs for a set of modular components spanning compute trays, fabric trays, memory trays, compute boards, and link boards, all of which enable a system that scales from 8 to 160 GPUs.
- It’s built on our hyperscale-tested connectivity to deliver unmatched performance: the Orion Compute Board incorporates a Scorpio™ X-Series 320L Smart Fabric Switch to deliver up to 89% more tokens per second than comparable legacy architectures, while Hypercast™ and In-Network Compute accelerate collective operations by up to 2x.
- The Orion Memory Tray, built on the Leo X-Series Smart Memory Controller, establishes a fabric-attached memory tier that can improve tokens per second by a further 22% and cut time to first token by up to 62%.
- As AI signaling demands push data rates higher than ever, the Orion Link Board leverages industry-leading Aries Smart Retimers, proven across millions of hyperscale deployments, to close the toughest channels.
- COSMOS Rack and Fabric Manager provides that crucial unifying software layer for telemetry, topology discovery, and link diagnostics.
Built on open standards, Orion provides a comprehensive set of modular building blocks that can be used to quickly bring online performant, validated infrastructure for next-gen inference workloads.
Orion’s modular approach provides specific benefits to emerging neocloud and enterprise AI market segments, whose business models depend equally on compute flexibility, controlled deployment costs, and repeatable rapid implementations. To learn why, read about the vision behind our Orion release.