Compute Express Link: Why AI Runs Out of Memory

At a Glance

Concept: A universal, open-standard “data highway” protocol that lets processors and accelerators share memory seamlessly, as if they were a single, massive chip.
Why it matters: Generative AI models have grown so large that their parameter weights no longer fit inside the onboard memory of a single GPU. Without CXL, scaling AI means buying thousands of redundant processors strictly to access their attached memory, destroying capital efficiency.
Who uses it: Hyperscale cloud providers (AWS, Google, Meta), chip designers (NVIDIA, AMD, Intel, Astera Labs), and global memory manufacturers (Samsung, SK Hynix).
Biggest takeaway: CXL fundamentally alters server architecture. Instead of buying a server with a fixed ratio of compute and memory, data centers are migrating to “disaggregated” racks where a central pool of memory is dynamically assigned to whichever CPU or GPU needs it at that exact millisecond.

In Simple Words

Imagine a team of master chefs (processors) working in a massive kitchen.

Traditionally, each chef has their own personal cutting board (memory). If Chef A needs an ingredient that is currently sitting on Chef B’s cutting board, Chef B has to stop working, pack the ingredient into a box, and physically walk it over to Chef A. This manual transfer wastes immense amounts of time and severely slows down the kitchen.

Compute Express Link (CXL) turns the entire kitchen into one giant, shared cutting board.

Every chef can reach out and grab the exact ingredient they need instantly, without asking permission or boxing it up. More importantly, the system instantly updates every chef on what the others are doing, preventing them from accidentally using the same ingredient twice. By allowing every processor to look at the exact same pool of memory simultaneously, the “kitchen” can process massive AI recipes without ever stopping to pass ingredients back and forth.

Why This Matters

The artificial intelligence revolution is currently bottlenecked not by math, but by traffic.

Over the past decade, the computational speed of GPUs (measured in TFLOPS) has increased exponentially. However, the speed and capacity of the memory feeding those GPUs (measured in Gigabytes per second) have grown at a much slower rate. This disconnect creates the “Memory Wall”—a physical limit where a USD 40,000 GPU sits idle, burning power while waiting for data to arrive from RAM.

To bypass this wall, AI companies have been forced into highly inefficient architectures. Because a frontier Large Language Model (LLM) requires terabytes of memory, data centers must link thousands of GPUs together. They are not buying these GPUs because they need more processing power; they are buying them simply to access the High Bandwidth Memory (HBM) physically soldered next to the chip.

CXL breaks this expensive paradigm. It allows data center architects to install massive, cheap banks of standard DDR5 memory independently of the expensive GPUs, creating a flexible, high-speed data reservoir that drastically lowers the capital expenditure required to scale AI supercomputers.

The Big Picture

The true superpower of CXL is that it does not require the entire semiconductor industry to re-tool its manufacturing lines.

Historically, when the tech industry needed a faster data connection, they invented an entirely new plug and socket. CXL bypassed this friction. It is fundamentally a software and protocol layer designed to ride on top of Peripheral Component Interconnect Express (PCIe)—specifically starting with the PCIe Gen5 standard.

Because every modern motherboard on Earth already has standard PCIe slots, hardware engineers can adopt CXL instantly. They simply slide a CXL-enabled memory expansion card into a standard PCIe slot, and the server magically gains access to terabytes of cache-coherent RAM. This clever engineering choice allowed CXL to achieve unprecedented, universal adoption across rival manufacturers (Intel, AMD, and ARM) in record time.

How It Works

Eliminating latency between separate silicon chips requires mastering the physics of electrical signaling and processor memory hierarchies. Here is the first-principles breakdown of the CXL architecture.

1. The Fundamental Problem: Network Copy Overhead

When an external device (like an AI accelerator card) is connected to a traditional CPU via standard PCIe, the two chips do not share memory natively. If the CPU has data the accelerator needs, the system must invoke Direct Memory Access (DMA), copy the block of data, and send it across the PCIe bus to the accelerator’s local memory. This copying process adds severe latency and burns immense electrical power.

2. The Insufficiency of Traditional PCIe

Standard PCIe is an exceptional I/O (Input/Output) protocol, but it is deeply isolated. It was built to talk to storage drives and network cards, not to act as an extension of the CPU’s brain. It lacks the ability to maintain “coherency”—meaning if both the CPU and the accelerator are reading the same data block simultaneously, standard PCIe has no fast mechanism to ensure they are both looking at the most updated version of the numbers.

3. The Core Mechanism: Cache Coherency

CXL solves this through absolute Cache Coherency. Processors have ultra-fast internal memory called “cache.” CXL ensures that if the GPU updates a piece of data in its cache, the CPU’s cache is instantly invalidated and updated. Neither chip has to manually copy the data back and forth; they both simply read and write to the same shared address space. If the CPU updates a variable, the GPU instantly sees the new variable.

4. Technical Depth: Flits and the Three Sub-Protocols

To achieve this over existing physical wires, CXL multiplexes three specific sub-protocols into data packets called “Flits” (Flow Control Units) that stream continuously across the PCIe Gen5/Gen6 physical layer:

CXL.io: Operates almost exactly like standard PCIe, handling device discovery, configuration, and interrupts.
CXL.cache: Allows the external device (the accelerator) to securely request, access, and cache data directly from the host CPU’s memory.
CXL.mem: Allows the host CPU to access the memory attached to the external device as if it were local, native RAM.

5. Real-World Consequences: Memory Pooling and Disaggregation

Because devices can now share memory fluidly over physical distances, hyperscalers no longer need to build monolithic, static servers. They deploy Disaggregated Architecture. A data center rack can contain one chassis full of CPUs, one chassis full of GPUs, and a separate chassis completely filled with terabytes of CXL-enabled RAM. Using a CXL switch, the data center orchestration software instantly assigns 500 Gigabytes of RAM to a specific GPU for a heavy AI training workload, and then instantly reassigns that same RAM to a CPU for a database search a minute later, drastically reducing hardware waste.

Real-World Applications

CXL is actively restructuring the physical footprint of global cloud infrastructure.

In-Memory Databases (IMDBs): Cloud platforms running massive databases (like SAP HANA or Redis) require the entire database to fit inside active RAM to achieve sub-millisecond search times. Previously, if a database exceeded the physical RAM limit of a single server motherboard (e.g., 4 Terabytes), it had to be partitioned across multiple servers, killing performance. CXL memory expansion cards allow operators to plug petabytes of RAM directly into the PCIe slots of a single server, supercharging database performance.

Deep Learning Recommendation Models (DLRMs): Social media algorithms (like those at Meta or TikTok) are heavily memory-bound. They use massive “embedding tables” that map user preferences to content. These tables are too large to fit into a GPU’s native High Bandwidth Memory (HBM). Data centers are using CXL.mem to attach vast pools of external DDR5 memory directly to the AI accelerators, ensuring the recommendation algorithms never stall.

Composable Cloud Instances: Cloud providers (AWS, Azure) struggle with “bin packing.” If a customer requests a virtual machine with 16 CPU cores and 100GB of RAM, the cloud provider must find a physical server that matches those exact specs. CXL enables “composable infrastructure.” The cloud provider can dynamically stitch together 16 cores from Server A and 100GB of RAM from Server B using a CXL fabric switch, creating perfectly tailored, on-demand virtual machines instantly.

Economic & Strategic Impact

The financial implication of CXL is the elimination of Stranded Memory.

In a traditional data center, memory makes up roughly 30 to 40 percent of total hardware costs. However, because memory is permanently soldered or slotted into individual servers, up to 25 percent of it is “stranded” at any given time. If a server is running a compute-heavy task that maxes out its CPU but only uses 10 percent of its RAM, the other 90 percent is trapped. It cannot be used by the server sitting right next to it, even if that adjacent server is starving for memory and crashing.

By migrating to CXL memory pooling, a massive cloud provider can reduce their total necessary DRAM purchases by up to 15 percent across their entire fleet. When operating at a hyperscale level with millions of servers, eliminating stranded memory translates directly to billions of dollars in saved capital expenditure (CapEx), fundamentally shifting the economics of AI deployment.

Advantages

Breaks the Capacity Limit: Allows a single processor to address terabytes or petabytes of memory beyond what can physically fit in local DIMM slots.
Eliminates Data Copies: Cache coherency removes the need for software to manually package and transfer data between chips, slashing power consumption and software complexity.
Hardware Agnosticism: As an open standard supported by the entire industry consortium, CXL prevents vendor lock-in, allowing an Intel CPU, an AMD GPU, and Samsung memory to interoperate flawlessly.
Protects Existing Investments: By riding the standard PCIe physical layer, it allows data centers to adopt next-generation architectures without ripping out their legacy motherboard and rack infrastructure.

Limitations

Latency Overhead: While vastly faster than networking protocols (like Ethernet or InfiniBand), CXL still involves traversing a physical bus. Accessing memory across a CXL link incurs roughly 150 to 250 nanoseconds of latency—which is slower than accessing native, direct-attached DDR5 or HBM.
Switch ASIC Immaturity: To achieve true “Memory Pooling” (CXL 3.0+), the data center requires highly complex, ultra-low-latency CXL switches. The silicon for these specific switches is difficult to manufacture and has suffered developmental delays.
Software Orchestration: The hardware is ready before the software. Most traditional operating systems and hypervisors were designed assuming memory is a fixed, local asset. Rewriting the Linux kernel and cloud orchestration layers to dynamically assign memory pools on the fly remains a monumental software engineering challenge.

Common Misconceptions

Misconception: CXL replaces High Bandwidth Memory (HBM) on GPUs.

Reality: HBM is physically stacked directly on top of the GPU processor to provide explosive, immediate bandwidth for the most intense math. CXL cannot match HBM’s speed. CXL is used for capacity (holding the massive bulk of the AI model), while HBM is used for velocity (feeding the data instantly into the math engines). They are highly complementary.

Misconception: CXL is a physical cable you plug into the wall.

Reality: CXL is a digital protocol. The physical connection is almost exclusively standard PCIe lanes (traces on a motherboard, or internal riser cables within a server chassis).

Misconception: Memory pooling is already fully deployed everywhere.

Reality: The market is executing a phased rollout. CXL 1.1 and 2.0 (Direct Memory Expansion) are actively deployed today. CXL 3.0 and 3.1 (True Multi-Host Memory Pooling via switches) are still largely in the prototyping and early sampling phases as of 2026.

What Most People Miss

The ultimate endgame of the CXL protocol is Optical Interconnects.

Currently, CXL transmits electrical signals over copper PCIe traces. Copper is physically limited; driving high-frequency CXL Gen6 signals over copper generates severe heat and limits the maximum cable distance to a few inches. The semiconductor industry is aggressively developing Optical CXL. By integrating silicon photonics (converting the CXL electrical signal into light), data centers will be able to stretch the CXL fabric across entire buildings, completely separating the “Compute Hall” from the “Memory Hall” without suffering signal degradation, unlocking the final stage of hyperscale disaggregation.

Comparison Table

Feature	Standard PCIe (Gen4/Gen5)	CXL 1.1 / 2.0	CXL 3.0 / 3.1
Primary Function	Direct I/O connection.	Memory Expansion (Direct Attached).	Memory Pooling & Fabrics.
Cache Coherency	No.	Yes (Between host and specific device).	Yes (Switched and Fabric-level).
Memory Sharing	Requires slow data copying.	Direct host access to device memory.	True multi-host, dynamic pooling.
Network Topology	Point-to-Point.	Tree / Basic Switching.	Advanced Fabric / Daisy Chaining.
Physical Layer	Standard PCIe wires.	PCIe Gen5 physical layer.	PCIe Gen6 physical layer.

Case Study

Situation: Meta (Facebook) was facing a critical infrastructure wall. Their core business relies on Deep Learning Recommendation Models (DLRMs). These AI algorithms require enormous “embedding tables” to store user data and content relationships.

Challenge: The embedding tables were growing by terabytes every year, wildly outpacing the memory capacity of the GPUs and CPUs in Meta’s massive data centers. Buying entirely new servers just to get more memory slots was economically unsustainable and resulted in massive stranded compute resources.

Solution (The Protocol Adoption): Meta became a founding and highly vocal leader in the Compute Express Link (CXL) Consortium. They redesigned their server architectures to incorporate CXL memory expansion modules. Instead of tying compute and memory upgrades together, they decoupled them.

Outcome: By plugging CXL-enabled memory modules directly into the PCIe slots of their existing servers, Meta successfully expanded the memory capacity of their AI nodes independently of the compute cores. This allowed their massive DLRM algorithms to access terabytes of coherent cache seamlessly.

Lessons Learned: The Meta deployment proved that memory capacity could be scaled as a modular, plug-and-play resource. By championing the open standard, Meta forced the entire semiconductor supply chain (Intel, AMD, Samsung) to converge on a single protocol, avoiding proprietary vendor lock-in and proving that disaggregation is the only viable path to hyperscale AI profitability.

Future Outlook

Next 12–24 Months

The enterprise market will experience the mass commercialization of CXL 2.0. Memory expansion modules utilizing the E3.S form factor will become standard additions in enterprise AI servers. Hyperscalers will focus intensely on optimizing their software hypervisors (like KVM) to efficiently recognize and manage this new “Tier 2” CXL memory without dropping application performance.

Next 3–5 Years

The transition to CXL 3.1 over PCIe Gen6 will fundamentally alter the server rack. PCIe Gen6 doubles the physical bandwidth, allowing CXL to move data at blistering speeds. This era will see the widespread deployment of top-of-rack CXL Fabric Switches. Data centers will move to true “Memory Pooling,” where a single, massive tray of DDR5 RAM is dynamically carved up and shared across dozens of independent host CPUs in real time.

Next 10 Years

CXL will merge with Silicon Photonics to create the ultimate Optical Data Center Mesh. As copper hits its absolute physical limit for high-frequency signal integrity, CXL signals will be transmitted via light (Optical I/O). This will allow processors to pull memory from a completely different aisle in the data center with the exact same latency as if the memory was soldered to the motherboard, achieving the holy grail of total, friction-free data center disaggregation.

Most Likely Scenario

Compute Express Link is the most critical infrastructure protocol of the 2020s. It will not replace localized, ultra-fast memory like HBM or L1/L2 caches, but it will permanently solve the capacity bottleneck. By introducing cache-coherent elasticity to hardware, CXL ensures that the exponential growth of artificial intelligence is not derailed by the physical limitations of a six-inch green motherboard.

Key Takeaways

Compute Express Link (CXL) is an open-standard interconnect protocol that allows processors and memory banks to share data with absolute cache coherency.
CXL solves the “Memory Wall” by allowing data centers to expand memory capacity independently from compute power, adding terabytes of RAM via standard PCIe slots.
By enforcing cache coherency, CXL eliminates the need for software to manually copy and transfer data between chips, slashing latency and power consumption.
CXL rides on top of the physical wires of standard PCIe (Gen5 and Gen6), multiplexing three sub-protocols (io, cache, and mem) into data packets called “Flits.”
Advanced CXL (versions 3.0 and 3.1) enables “Memory Pooling,” where centralized arrays of RAM are dynamically assigned to different processors via CXL fabric switches, eradicating stranded memory.
The widespread adoption of CXL fundamentally shifts data center architecture from rigid, monolithic servers to flexible, “composable” and disaggregated infrastructure.

Glossary

Cache Coherency: A protocol ensuring that multiple processors or devices sharing a common memory resource always see the most updated, synchronized version of the data.

Composable Infrastructure: A data center architecture where compute, storage, and memory are separated into distinct physical pools and dynamically combined via software to create custom virtual servers on the fly.

Flit (Flow Control Unit): The basic unit of data transfer used by the CXL protocol. CXL packs multiple sub-protocols (like cache updates and standard I/O) into a single flit for highly efficient transmission over PCIe wires.

High Bandwidth Memory (HBM): A specialized, ultra-fast type of computer memory that is physically stacked in 3D and placed directly next to an AI processor on the same silicon package.

Memory Pooling: The practice of placing massive amounts of RAM in a centralized hardware chassis and dynamically assigning portions of it to different servers across a network as needed.

PCIe (Peripheral Component Interconnect Express): The universal, physical hardware interface standard used in almost all computers to connect motherboards to external devices like graphics cards and hard drives.

Stranded Memory: RAM that is physically installed inside a server but goes unused because the specific workload running on that server does not need it, resulting in wasted capital expenditure.

Frequently Asked Questions

Does CXL replace PCIe?

No. CXL uses PCIe. PCIe provides the physical lanes, pins, and electrical signals. CXL is the advanced software/protocol layer that rides on top of those physical wires to add cache coherency and memory pooling features.

Is CXL memory slower than regular memory?

Yes. Because the data has to travel across a physical PCIe bus and through an external controller, CXL memory has slightly higher latency (delay) than the DDR5 memory slotted directly next to the CPU. Therefore, CXL is typically used as “Tier 2” memory for massive capacity, while local memory is used for the most speed-critical tasks.

Do I need a new motherboard to use CXL?

You need a motherboard with a modern CPU (like Intel Sapphire Rapids, AMD Genoa, or newer) that natively supports the CXL protocol, and you need physical PCIe Gen5 (or newer) slots to handle the bandwidth requirements.

Why are hyperscalers like AWS and Google so obsessed with CXL?

Cloud providers waste billions of dollars every year on stranded memory. When they rent out virtual machines to customers, they often have unused RAM trapped inside physical servers. CXL memory pooling allows them to dynamically move that trapped memory to paying customers, drastically improving their hardware profit margins.

Who owns the CXL standard?

The CXL Consortium, an open industry standard group. It was originally pioneered by Intel but quickly gained universal support. Today, essentially every major tech company (NVIDIA, AMD, ARM, Google, Meta, Microsoft, Samsung) sits on the board of directors, ensuring it is a universally interoperable standard.

Sources

Compute Express Link (CXL) Consortium: Official CXL 3.1 Specification and Architectural Overviews (2025/2026 updates)
Samsung Semiconductor: CXL Memory Module (CMM) and Disaggregated Architecture Roadmaps
Astera Labs: Overcoming the AI Memory Wall with CXL and PCIe connectivity
Meta Engineering: Deep Learning Recommendation Models (DLRM) and the necessity of Memory Disaggregation