ASAravind Sivakumaran
Theme

Theme and appearance

Appearance
Palette

Pick any colour. The displayed accent adjusts for readable contrast in light or dark mode.

← Systems Observatory

JOURNEY / PCIe, IOMMU, POWER

One NIC, three stops

The other pages each take one topic apart. This one follows a single network card, the one at 03:00.0, through all three, so you can see where each topic hands over to the next. Think of the card as a courier. PCIe decides whether the courier may leave the depot at all. The IOMMU decides which door the courier may deliver to. Power is the courier's night's sleep: after it, nobody should trust that the courier still remembers either rule until each has been checked again.

  1. PCIe: read the card, then turn on BusMaster, the switch that lets it start a write. Reading lspci, Before DMA.
  2. IOMMU: map one page for requester 03:00.0 and watch 16 bytes land in it, while an unmapped address is refused. 16 bytes, IOMMU foundations.
  3. Power: close the gate, sleep, wake and restore, in order. Then repeat the write. Device sleep & wake, System sleep.
What claims does this page make? 3 points
  • Linux behaviour: pci_set_master() sets the BusMaster bit, and pci_restore_state() writes a device's saved configuration back; whether a card lost it in D3hot is decided by PCI_PM_CTRL_NO_SOFT_RESET, which the PCI power-up code checks. AMD's driver disables the IOMMU on suspend and reloads it on resume (amd_iommu_suspend(), amd_iommu_resume()); all read in Linux v6.12.
  • Public specification: the AMD IOMMU picks a device's page tables by its requester ID and caches translations, as dmakit models it (AMD 48882 rev 3.11).
  • Teaching model: the order of the sleep and wake steps is the power lab's declared lifecycle, and which state the card loses in sleep is that fixture's choice, not a measurement. The card and its addresses are invented.
The CPU and OS reach the card through the root port. The card's requests go through the root port to the IOMMU, then to memory. CPU + OSconfig reads, MMIO IOMMUin the root complex MemoryRAM Root port00:01.1, the link Card 03:00.0config, BARs, ATC

Experiments

One platform, chosen for teaching: a PC whose IOMMU sits in the root complex, as on AMD and Intel systems.

Experiment

    Registers, as lspci prints them

      Commands and reads

        Notes

          What each stop handed to the next

          1. PCIe to IOMMU: the card's address, 03:00.0, became its requester ID. Without BusMaster the IOMMU never saw a request; with it, the requester ID picked the context, and that context's mappings and permissions decided whether this request could reach its target.
          2. IOMMU to power: the mapping lives in page tables in memory and survives the card's sleep, but the IOMMU's cached copies of it do not deserve trust. Skip the reload and a stale entry writes into a page that now belongs to someone else (try the "What if" before the reload).
          3. Power back to PCIe: a woken card may have lost its configuration, BusMaster included. The write after waking is only allowed because each earlier stop was repeated in order.

          This is the validation habit in miniature: after any transition, do not trust the success message, restore what was lost, then send a fresh allowed request and a fresh forbidden one and check both. Investigate a failure practises it on a fault record.

          Sources