ASAravind Sivakumaran
Theme

Theme and appearance

Appearance
Palette

Pick any colour. The displayed accent adjusts for readable contrast in light or dark mode.

← IOMMU star map

VIRTUALIZATION / IOMMU GROUPS AND ACS

Why can't the VM have just this one device?

Before a VM can own a device, the host must be sure that nothing else can reach it, or be reached by it, without the IOMMU checking. That is not always true of one device on its own: two devices may share a requester ID, or send each other DMA that turns around in a switch before the IOMMU sees it. Linux therefore hands out devices in IOMMU groups, and VFIO, the driver that gives devices to VMs, takes a whole group or nothing. When passthrough fails, this is usually why.

What claims does this page make? 3 points
A network card writes to an NVMe drive below the same PCIe switch. Without ACS the write turns around inside the switch and never reaches the IOMMU. With ACS P2P Request Redirect it goes up to the root complex, where the IOMMU checks it. Root complex the IOMMU checks here PCIe switch Network card 03:00.0 NVMe drive 04:00.0 no ACS: turns around in the switch with ACS RR: up and back

Lab: build the groups

Choose a machine, then switch ACS controls on and off at its ports. The groups are computed the way Linux 7.3 computes them; the peer-to-peer writes follow the ports' ACS controls.

The devices, as Linux finds them

SV, RR, CR and UF are the four ACS controls Linux requires (the table below explains each). With an IOMMU present Linux turns on every one a port implements, so a ticked capability starts with all four on.

The groups Linux makes

    A peer-to-peer write

    Writes that never reach the IOMMU

    Give one device to a VM

    What an IOMMU group is

    The IOMMU finds a device's tables by its requester ID (bus, device and function, such as 03:00.0; see request identity). It can only keep two devices apart if their DMA arrives with different requester IDs and actually arrives. Linux's VFIO documentation puts it this way: a group is "a set of devices which is isolatable from all other devices in the system", and groups are "the unit of ownership used by VFIO". A VM can get a group; it cannot get half of one.

    One requester, or two

    Several things make two devices look like one requester:

    SR-IOV goes the other way: each virtual function (VF) has its own requester ID (VT-d §3.12.6), and Linux treats a VF as a single-function device (iov.c sets multifunction = 0). So with ACS above the card, each VF gets its own group and can go to a different VM.

    Peer-to-peer DMA, and what ACS changes

    A device's DMA does not have to go up to the root complex. A PCIe switch routes by address: a write from one port to a memory range below another port can turn around inside the switch. Two functions of one card may talk inside the card, and devices on a conventional PCI bus all share one bus. In each case the IOMMU never sees the write.

    Access Control Services (ACS) is the PCIe capability that makes ports send such traffic upstream instead. The controls, with their bits in the ACS Control register (pci_regs.h):

    ControlBitWhat it is forLinux's grouping
    Source Validation (SV)0x0001Checks the requester ID of requests from below, so that, in Linux's words, "devices cannot spoof their requester ID" (VT-d §3.12.3 recommends ACS on root ports for this)Required
    Translation Blocking (TB)0x0002Blocks memory requests whose address type is not "untranslated" (VT-d §4.2.2)Not required; Linux turns it on for external-facing or untrusted ports, or with ATS off
    P2P Request Redirect (RR)0x0004Sends peer-to-peer requests upstream instead of acrossRequired
    P2P Completion Redirect (CR)0x0008The same for completions (the answers to reads)Required
    Upstream Forwarding (UF)0x0010Linux: "all transactions are forwarded upstream, even as it passes through a bridge where the target device is downstream"Required
    P2P Egress Control (EC)0x0020Controls peer traffic port by port, through an Egress Control Vector; Linux counts it, with RR and CR, as a redirect controlNot required
    Direct Translated P2P (DT)0x0040Concerns peer requests that carry a translated addressNot required; Linux does not turn it on by default

    Linux's own peer-to-peer DMA code agrees on the effect: if RR, CR or EC is set on a port on the path, "traffic between the devices will go through the host bridge" (p2pdma.c). The lab models a write: RR decides at the port where it enters the switch, and UF at a port that would send it straight back down to another function of the same card.

    How Linux builds groups

    pci_device_group() in drivers/iommu/iommu.c runs for every PCI device, bridges first. For one device it:

    1. Follows its DMA aliases up the tree (pci_for_each_dma_alias). Behind a PCIe-to-PCI bridge it continues from the bridge, and joins the bridge's group.
    2. Walks up, one bridge at a time, until the whole path from that bridge to the root has ACS (pci_acs_path_enabled with REQ_ACS_FLAGS, which is SV, RR, CR and UF). Every bridge it passes without ACS pulls the device into that bridge's group.
    3. Joins the group of any device it is a DMA alias of (get_pci_alias_group), then of any other function in the same slot when neither has ACS (get_pci_function_alias_group).
    4. Otherwise it gets a new group.

    Which devices "have ACS" follows the PCIe type (pci_acs_enabled): root ports and switch downstream ports need the capability; a single-function endpoint passes without one, since it has no peer inside it to reach; a function of a multifunction device needs it; conventional PCI devices and PCIe-to-PCI bridges never pass. A control a port does not implement counts as hard-wired on. Some chipsets isolate without the standard capability, and quirks (pci_dev_acs_enabled[] in quirks.c) answer for them.

    A known gap. The walk checks the path above each device, not its neighbours. Choose the switch machine in the lab: without ACS on the downstream ports, Linux 7.3 puts the network card and the NVMe drive in different groups, although a write between them turns around inside the switch. Jason Gunthorpe's patch series (September 2025) reports exactly this topology and proposes a rework; the grouping code in Linux 7.3 is still the version described here. Until then, treat a switch without ACS as one group, whatever sysfs says.

    What VFIO needs, and why the ACS override is unsafe

    VFIO takes ownership of the whole group (iommu_group_claim_dma_owner). It refuses while any device in the group is bound to a driver that does its own DMA through the kernel. Every device must have no driver, or one marked driver_managed_dma: vfio-pci, pci-stub, or pcieport on bridges. In the VFIO documentation's example, assigning 06:0d.0 means also binding 06:0d.1, behind the same PCIe-to-PCI bridge. A graphics card's HDMI audio function is the everyday case.

    The "ACS override" is a kernel patch proposed in 2013 that adds a pcie_acs_override= option to assume ACS where the hardware has none. It was never merged, but patched kernels that carry it are still passed around for passthrough. It splits the groups and VFIO accepts them, but the switch still turns the write around: the VM's device can reach a device the host or another VM still uses, and the IOMMU never checks it. On the list, Bjorn Helgaas worried it was "an easy way to pretend isolation exists". Mainline's pci=config_acs= changes real ACS controls on ports that implement them; the kernel documentation warns that turning them off "may put more devices in an IOMMU group".

    On a real machine

    Not run by this site. Read-only commands; check their output against your own kernel's version. 

    ls /sys/kernel/iommu_groups/
ls /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices
sudo lspci -vvv -s 00:01.0 | grep -A2 'Access Control'

    The first lists the groups, the second the devices sharing 03:00.0's group, and the third the root port's ACS capability (ACSCap) and controls (ACSCtl), if it has one.

    Where this fits

    Request identity and the PCIe setup come first: request identity covers requester IDs and aliases as test cases, and PCIe setup what a device needs before it may send DMA at all. Next, shared page tables and the virtual IOMMU assume a device already assigned to a VM, which is the group this page gets you.

    Sources