What if a PCIe card could be memory, not just a device?

The simple guide, in five ideas

1. CXL is PCIe plus two protocols. CXL.io is PCIe itself (configuration, BARs, DMA, interrupts); every CXL device has it. CXL.mem lets the CPU load and store to the device's memory. CXL.cache lets the device cache the host's memory.
2. The link decides at power-on. A CXL card in a CXL slot trains as CXL; in a plain PCIe slot it trains as PCIe, and its CXL abilities stay off.
3. Three device types. Type 1: an accelerator with a cache (io + cache). Type 2: an accelerator with its own memory too (io + cache + mem). Type 3: a memory expander (io + mem), by far the most common today.
4. The host keeps everything coherent. Every copy of a 64-byte line, in the CPU or in a device, is tracked by the host. A device never has to understand the CPU's own coherence protocol.
5. To software, CXL memory is memory. Linux shows a memory card as a NUMA node with no CPUs: slower than local DRAM, much faster than a disk, used by programs with no changes.

Protocols and device types: pick a device

Choose a device and whether the slot's root port speaks CXL. The link offers, answers and settles in its training sets, then the protocols share the link in 68-byte flits, each tagged with a two-byte protocol ID: CXL.cache and CXL.mem share one ID because they share one link layer.

On real hardware lspci shows the result in the Flex Bus port DVSEC. From the QEMU memory card on this page: FBCap: Cache- IO+ Mem+ 68BFlit+ (what it can do) and FBSta: Cache- IO+ Mem+ 68BFlit+ (what the link negotiated).

What CXL is for, and what it costs

• More memory, and more bandwidth, per CPU. DIMM slots run out, and each DDR channel costs hundreds of processor pins. A CXL x8 link carries a memory card's traffic over far fewer pins; a PCIe 5.0 x16 link carries about 64 GB/s each way. Shipping Type 3 cards (Micron CZ120, Samsung CMM-D, SK hynix CMM-DDR5) put 96 to 256 GB of DDR on an x8 link.
• Less stranded memory. Microsoft's Pond study found up to 25% of a cloud server's DRAM sits unused because CPUs ran out first; pooling memory across 16 sockets cut DRAM cost by about 7% with most workloads within 1-5% of local performance.
• Tiering. Hot pages in local DRAM, cold ones on CXL: Meta's TPP, partly merged in Linux 5.18, kept its workloads within 1% of all-local memory on early CXL hardware.
• Accelerators that share data. With CXL.cache a device works on host memory in its own cache, without copying buffers back and forth.

The price is latency. Estimates from the specification's authors and from Pond, not measurements:

Measured hardware varies more than the link does. On real Intel Sapphire Rapids systems, Sun et al. found the best of three CXL cards only 35% slower than memory on the other socket, and the others about two and three times slower: the memory controller on the card matters more than CXL itself. Measure your device; do not assume a remote NUMA node behaves like it.

Versions

1. 1.0, March 2019Intel's donated specification, on PCIe 5.0 (32 GT/s); the three protocols, 68-byte flits. 1.1 (mid-2019) added compliance testing.
2. 2.0, 10 November 2020Switches (one level), memory pooling (a multi-logical device splits into up to 16), persistent memory, hot-plug, link encryption (IDE).
3. 3.0, 2 August 202264 GT/s on PCIe 6.0 with 256-byte flits, multi-level switches and fabrics, memory sharing between hosts, back-invalidate coherence, peer-to-peer.
4. 3.1, 14 November 2023Fabric routing details, host-to-host memory (GIM), TEE Security Protocol (TSP) for confidential computing.
5. 3.2, 3 December 2024A hot-page monitoring unit for tiering, online firmware activation, more security and RAS.
6. 4.0, 18 November 2025128 GT/s on PCIe 7.0, native x2 links, bundled ports (several links as one), up to four retimers.

Each version keeps working with devices of the earlier ones.

For professionals: flits and channels

Protocol IDs in a 68-byte flit: FFFF CXL.io, 5555 CXL.cache and CXL.mem, CCCC an ARB/MUX link-management packet, 9999 an idle (NULL) flit. The ARB/MUX layer decides which link layer gets the next flit. A 256-byte flit has no protocol ID: the 2-bit flit type in its header says CXL.io, cache and mem, ARB/MUX, or idle. The physical layer chapter's flit lab shows PCIe's own 256-byte layout; its CRC and FEC are the ones CXL keeps (CXL 3.0 section 6.2.3.1).

• Who is in charge of coherence. For CXL.cache the host is the home agent and the device a cache it tracks (MESI, 64-byte lines, host physical addresses, so the device translates first with ATS). For CXL.mem the host is master, the device subordinate.
• Device memory and its coherence. HDM (host-managed device memory) comes in three kinds: HDM-H, which only the host caches (Type 3); HDM-D, where a Type 2 device switches lines between host bias and device bias over CXL.cache (CXL 1.x and 2.0); and HDM-DB, where the device sends the host back-invalidate snoops (BISnp) to take lines back (CXL 3.0, an alternative to bias).
• When CXL turns on. CXL.cache and CXL.mem are enabled only once the link reaches L0 at 8 GT/s or more; a CXL link that cannot may fail to come up at all, rather than run slowly.

Interleaving: which card holds an address?

A region spread over several cards is striped like RAID 0: the first granularity bytes on device 0, the next on device 1, and so on. Legal settings are 1, 2, 3, 4, 6, 8, 12 or 16 ways and 256 bytes to 16 KiB. Linux computes the position as (offset ÷ granularity) mod ways (cxl_calculate_position()). Try offset 1000 with 2 ways of 4 KiB: device 1, offset 0 into its part of the region, exactly where the QEMU machine put the value written at 490001000. That offset is the device address (DPA) only when the device's part starts at DPA 0, as on that machine; a device with memory set aside before the region adds that start.

Every decoder on the path does its share: the CEDT's window can interleave across host bridges, the host bridge across its root ports, a switch across its ports; each card's decoder then removes the position bits to get an offset into its part of the region, and adds where that part starts to get its own address (DPA, device physical address). 3, 6 and 12 ways need hardware that reports them.

Linux, step by step

1. Firmware describes the platform in ACPI's CEDT (CXL Early Discovery Table): a CHBS entry for each CXL host bridge (where its registers are) and a CFMWS entry for each fixed memory window (base, size, interleave across host bridges, what kinds of memory it may hold). In the QEMU boot: ACPI: CEDT … and pci0000:0c: host supports CXL.
2. cxl_acpi turns each window into a root decoder (decoder0.0: 490000000, 4 GiB) under root0.
3. cxl_pci binds each card by its class (0502, prog-if 10), finds its registers through the Register Locator DVSEC and makes mem0, mem1. cxl_mem and cxl_port link each card into the port tree: port1 (the host bridge) and endpoint2, endpoint3, each with decoders. The numbers follow the order the cards finished probing, which changes from boot to boot (in the captured boot mem1 is the card at 0d:00.0), so scripts pick cards by PCI address.
4. A region is created on a root decoder, given ways, granularity, size and its target cards, and committed: Linux programs every decoder on the path.
5. dax_cxl and kmem turn the region into a DAX device and the DAX device into memory blocks on a new NUMA node.

The QEMU machine (QEMU 8.2; a kernel with the CXL modules and an initramfs added):

qemu-system-x86_64 -M q35,cxl=on -m 2G,maxmem=8G,slots=8 -smp 2 -cpu max,-hypervisor \
 -object memory-backend-file,id=vmem0,share=on,mem-path=vmem0.bin,size=256M \
 -object memory-backend-file,id=vmem1,share=on,mem-path=vmem1.bin,size=256M \
 -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
 -device cxl-rp,port=0,bus=cxl.1,id=rp13,chassis=0,slot=2 \
 -device cxl-rp,port=1,bus=cxl.1,id=rp14,chassis=0,slot=3 \
 -device cxl-type3,bus=rp13,volatile-memdev=vmem0,id=cxl-vmem0 \
 -device cxl-type3,bus=rp14,volatile-memdev=vmem1,id=cxl-vmem1 \
 -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k

Then, with cxl-cli and daxctl from ndctl (the boot on this page wrote the same values to sysfs by hand; the order of the cards sets their positions):

cxl list -M
cxl create-region -m -t ram -d decoder0.0 -w 2 -g 4096 mem1 mem0
daxctl reconfigure-device --mode=system-ram dax0.0
numactl -H

What came out, from sysfs in the guest:

region0: resource 0x490000000 size 0x20000000 ways 2 gran 4096 target0 decoder2.0 target1 decoder3.0
decoder1.0: start 0x490000000 size 0x20000000 ways 2 gran 4096 targets 0,1
dax0.0 size 536870912 target_node 1
node1: cpus [] MemTotal: 524288 kB

The test that matters: six values written 4 KiB apart from 490000000 landed alternately in the two cards' backing files, offsets 0, 1000, 2000 in each. The card's own view, from lspci (press a line):

• On x86, Linux 6.8 will not commit a region when the CPU reports a hypervisor: it must invalidate CPU caches before memory appears at a new address and does not trust a guest to (cxl region0: Failed to synchronize CPU cache state; a kernel built with CONFIG_CXL_REGION_INVALIDATION_TEST skips the check, for testing only). That is why the QEMU command has -cpu max,-hypervisor; the fourth experiment shows the failure.
• With /sys/kernel/mm/numa/demotion_enabled set, reclaim moves cold pages to the CXL node instead of swapping them out.
• The enumeration chapter covers how the cards were found as PCI devices before any of this.

Switches, pools, sharing and security

• Pooling (2.0): a switch, or a card with several host ports, gives each host its own share. A multi-logical device (MLD) splits into up to 16 logical devices; a fabric manager (software in a host, the switch or a BMC) decides who gets which.
• Sharing (3.0): the same memory visible to several hosts at once, kept coherent by back-invalidate. Pooling moves memory between owners; sharing has many.
• Fabrics (3.0): several switch levels and non-tree topologies with port-based routing, up to 4096 nodes. Dynamic capacity devices hand out memory in extents at run time.
• Security: IDE (2.0) encrypts and authenticates the link with AES-GCM and a 256-bit key; TSP (3.1) lets a confidential VM trust memory on a CXL card. For how the IOMMU fits around devices, see the secure DMA chapter.