ASAravind Sivakumaran
Theme

Theme and appearance

Appearance
Palette

Pick any colour. The displayed accent adjusts for readable contrast in light or dark mode.

← PCIe guide

PCIe / SR-IOV LAB

One card, two tenants: what must work after the VF appears?

The previous chapter turned one card into many: set a count, and virtual functions (VFs) appear on the bus, each with its own requester ID. That is where the hard part starts. Think of a building with one front door and two flats: a flat existing is not the same as a tenant living in it, each tenant needs a key that opens only their door, and the post room must not deliver one tenant's letters to the other just because the door number matches.

Watch an SR-IOV card come up, one scroll at a time → (animated showcase)

The card here is LabNIC 1, a made-up network card with two VFs: VF0 for VM A and VF1 for VM B. Eight labs, each a question, a working path, named broken paths ("What if") and the repair:

  1. A VF can exist without any guest being able to use it.
  2. The same DMA address reaches different memory, and taking a mapping away safely takes proof.
  3. A successful DMA does not mean the packet left the card.
  4. Separate VFs still share one port.
  5. The card may show eight VFs and afford only six.
  6. A finished reset is not a recovery until both tenants prove it.
  7. A completion in memory is not an interrupt in the guest.
  8. Moving a VF to another VM is safe only after every barrier.
What claims does this page make? 4 points
  • Public specification: the AMD IOMMU picks a device's page tables by its requester ID, caches translations in its IOTLB, and confirms commands with COMPLETION_WAIT (AMD 48882 rev 3.11), all as dmakit models them. SR-IOV itself is in the PCI Express Base Specification, which is members only and not cited here.
  • Linux behaviour: sriov_numvfs and sriov_drivers_autoprobe (PCI IOV howto), VFIO handing over whole IOMMU groups (VFIO), and an MSI being a write of its own that cannot pass the data writes before it, with MSI-X vectors masked one by one (MSI howto).
  • Teaching model: LabNIC 1, its addresses, its lab-vf driver, its 8-byte completion record, its packet policy, its port scheduler, queue budget, reset profiles, interrupt route, ownership epoch and the order of its lifecycle steps are invented for this page.
  • Prepared evidence: the Linux-style output in the lab is synthetic, generated from the model. It was never measured on hardware.
Control runs from the host administrator through the host PCI layer to the assignment stack and the guests; data runs from a guest through the card and the IOMMU to memory. Host admin, PF41:00.0, policy Host PCIVF BDFs, binding AssignmentVFIO, group, VM Guest AVF0's driver, queue LabNIC 1VF engines, policy, port Guest BVF1's driver, queue IOMMURID → domain → page MemoryA, B and a guard page - - - control—— data

The lab

Each lab starts from its own fixture; a "What if" runs on a copy and never leaks into the main run. Predict each result before you take the step.

Lab

    Result of the last operation

      Bytes

        Setup, VF by VF

          Translation, invalidations and counters
            Prepared Linux evidence (synthetic, read-only)

            What you would read on a real host or guest, in the shape Linux prints it, generated from this model. lab-vf and LabNIC are not real drivers or devices, and real paths and output vary.

              Who owns what

              1. Host administrator and PF driver: create the VFs and set each VF's policy (its MAC and VLAN here). Managing the PF does not mean the PF driver handles each tenant's packets.
              2. Host PCI layer: each VF's address (41:02.0, 41:02.1) and which host driver it is bound to: lab-vf, vfio-pci or none.
              3. Assignment: who owns each IOMMU group for DMA, and which VM a VF is attached to. Bound to vfio-pci is not owned; owned is not attached.
              4. Guest A and guest B: the guest's own PCI address for the VF (00:05.0, a label only), its driver and its queues.
              5. The card: the VF engines and the packet policy, in front of one shared link that a scheduler divides between them (lab 04).
              6. Memory: the IOMMU takes the requester ID (the host's 0x4110 or 0x4111, whatever the guest calls the device), picks that VF's domain, checks the page and its permissions, and only then do bytes move.

              An IOMMU group is one ownership unit: devices in it may not be isolatable from each other, so they come as one group and VFIO hands them to one owner. Without ACS above them, devices might reach each other without passing the IOMMU, and group numbers alone do not prove otherwise (IOMMU groups & ACS). A stale cached translation, the danger lab 02 guards against, has its own chapter: cache stories.

              On a real card

              Real per-VF policy and its counters differ by device and driver, and none of them match LabNIC's codes. Intel documents per-VF MAC, VLAN and anti-spoofing controls for its ice and ixgbe drivers, and Linux's igb driver for the 82576 logs Spoof event(s) detected on VF %d when its check drops frames. In each case the IOMMU's fault log stays empty, because no address was wrong.

              Real invalidation also differs: on AMD, Linux sends the invalidation and its COMPLETION_WAIT from inside the kernel's unmap path, and VFIO users do not send them themselves. The lab shows them as separate steps so that each proof can be seen.

              Budgets, resets and interrupts differ by card too. With SR-IOV on, ixgbe groups its descriptor queue pairs into pools, one for the PF and one for each VF, and how many pairs a pool holds depends on the configuration. The iavf documentation lists the minimum each VF gets in its base mode (4 queue pairs, 5 MSI-X vectors), and asks the guest to support MSI-X. ice names three reset types: a PF reset touches only that PF, a CORE reset the whole adapter, and a GLOBAL reset the adapter plus its MAC and PHY. None of these is LabNIC's isolated-vf or shared-pf profile, and no single rule says a VF reset leaves its neighbours alone. The lab's order for moving a VF is one careful choice, not a procedure for real hardware.

              Sources