Which switches make a device an IOMMU client?

What a request carries

When the card reads or writes memory it sends a packet, a TLP. The IOMMU sees only what is written on it:

The requester ID is the flat number on the return address; a PASID adds which resident of the flat wrote it. Without PASID, everything the card does shares one I/O address space. With it, the card can work for several processes or VMs at once, each with its own table: the idea behind shared page tables. The AT field says whether the address still needs translating; ATS is how it can say "already translated".

PASID: PASIDCap and PASIDCtl

Capabilities: [160 v1] Process Address Space ID (PASID)
  PASIDCap: Exec+ Priv+, Max PASID Width: 0e
  PASIDCtl: Enable- Exec- Priv-

Read lspci's pairs as Cap = what the device can do (fixed, read-only) and Ctl = what software turned on. Max PASID Width is printed in hex: 0e is 14 bits, so PASIDs 0 to 16383. Exec and Priv let a request also say "this is an instruction fetch" or "this is privileged (kernel) access". Flip them, including the read-only one:

Write ffff reads back 0007: only three bits of PASIDCtl exist. Linux enables PASID in pci_enable_pasid(), and refuses unless the device can carry the PASID label (End-End TLP Prefix support) and every switch above it forwards requests up to the IOMMU instead of straight to a neighbour (ACS Request Redirect and Upstream Forwarding, the subject of IOMMU groups; Bridges and AtomicOps follows both checks through a switch). Virtual functions have no PASID capability of their own; they share their parent's.

ATS, the ATC and PRI

Capabilities: [148 v1] Address Translation Service (ATS)
  ATSCap:  Invalidate Queue Depth: 00
  ATSCtl:  Enable-, Smallest Translation Unit: 00
Capabilities: [150 v1] Page Request Interface (PRI)
  PRICtl: Enable- Reset-
  PRISta: RF- UPRGI- Stopped+ PASID+
  Page Request Capacity: 00000020, Page Request Allocation: 00000000

Normally every request goes to the IOMMU untranslated and the IOMMU looks the address up, using its own cache, the IOTLB. With ATS the device asks first ("what is IOVA 0x1000?"), keeps the answer in its own ATC (address translation cache), and then sends requests marked translated. It saves the IOMMU a lookup on every request; the price is that when a mapping changes, the IOMMU must tell the device to forget it (an ATS invalidation) and wait for it to confirm.

• Smallest Translation Unit: 00: the smallest page the IOMMU will hand back is 2^12+STU bytes, so 00 is 4 KiB. Linux writes it from the page shift in pci_enable_ats().
• Invalidate Queue Depth: 00: how many invalidations the device can hold at once; 0 means 32.
• PRI goes one step further: when a translation comes back "not present", the device asks the OS to bring the page in (a page request) instead of failing. That is how a device can use ordinary, pageable process memory (shared virtual addressing, SVA). Page Request Capacity: 00000020 means it may have 32 page requests outstanding; Linux writes the allocation it grants and enables PRI only while PRISta says Stopped+.

Turn on the full SVA set, one switch at a time, and watch what the card may send:

The device's switch is half the story. The IOMMU must also agree to answer translation requests for this device; on AMD that is the I bit of its device table entry; with it clear, the IOMMU does not service the device's translation requests. On a real system the IOMMU driver sets both sides, and in a fixed order: PASID first, then ATS, then PRI, and PRI only once ATS is on. Intel's driver explains why: a device's behaviour is undefined if PASID is enabled after ATS. That is one more reason not to flip these by hand. The Translation lab lets you send ATS requests through the emulated AMD IOMMU, and IOTLB & cache state shows what a stale cached translation does.

SR-IOV: one card, many functions

Capabilities: [168 v1] Single Root I/O Virtualization (SR-IOV)
  IOVCtl:  Enable- Migration- Interrupt- MSE- ARIHierarchy- 10BitTagReq-
  Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
  VF offset: 128, stride: 2, Device ID: 0002

SR-IOV lets one physical function (PF) put up to Total VFs lightweight virtual functions on the bus, each with its own requester ID and BAR window. The IOMMU tells them apart by requester ID, so each VF can be given its own IOMMU domain: a VM can own one VF while the host keeps the PF. Where each VF appears is arithmetic: VF n is at the PF's requester ID + offset + n × stride. Set Number of VFs, then enable:

Virtual functions on the bus:

Try changing Number of VFs while VF Enable is on: this emulator ignores the write. Linux never tries: it sets the count only while the VFs are off, so through sysfs you write the count to sriov_numvfs, and to change it, write 0 first. With offset 128 and stride 2, the VFs start at 03:10.0; a card with 65 or more VFs would run past device 1f onto bus 04, a carry the Passthrough chapter computes. In the real world use sysfs, never setpci, for SR-IOV: the kernel must create a pci_dev for each VF, and a bare register write does not.

ASPM: letting the link nap

Active State Power Management lets an idle link drop into a low-power state on its own. L0s is a short doze in one direction, quick to leave; L1 is deeper sleep for both directions, saving more and waking slower. LnkCap says which the device supports and how long it takes to wake; LnkCtl says which are on. Both ends of the link must agree, and Linux sets the two together.

The trade is latency for power: a laptop wants L1, a latency-sensitive NIC may not, and some devices misbehave with ASPM on, showing up as link errors or devices that vanish. When that happens, turn ASPM down through the OS (next section) rather than with setpci: an ASPM state enabled on one end only, or behind the kernel's back, is not something the kernel tracks.

The same switches in Linux and Windows

On a real machine, read with lspci (or the OS's own view) and change things through the OS, which keeps the driver and the hardware in step.

Cheat sheet: lspci field to setpci register

Every switch on these two pages, with where it lives. Values are hexadecimal; value:mask changes only the mask's bits.

Next in the guide: the SR-IOV lab gives two VFs to two VMs and breaks it three ways; Bridges and AtomicOps follow one of these requests through a switch; IOMMU foundations follow it into the IOMMU.