An IOMMU checks the addresses a device uses when its traffic is routed through it (new to that? start
with IOMMU basics). When a VM owns a
physical NIC (the first path below) there are two checks. The guest's
table turns the device's address (its IOVA) into a guest-physical
address (GPA); the host's table, called stage 2, turns that into a real
system-physical address (SPA, often called HPA). Arm says IPA and PA.
Send one 16-byte write, then change one decision at a time and watch
the bytes. An address is a page base plus an offset: in 0x10001230 the
page is 0x10001000 and the offset 0x230.
What claims does this page make? · 3 points
Spec rule: AMD IOMMU specification 48882 rev 3.11 (§2.2.6.3, §2.2.7, §2.10) and Arm SMMUv3 IHI 0070 H.a define the two-stage walk, GCR3TRPMode and the commands a guest may send.
Platform assumption: the nested attach, the VMM's trapping of guest commands and the Linux iommufd behaviour (v7.3-rc5, checked 2 October 2026) depend on the kernel and VMM, and are not in either specification.
Teaching model: the experiments run on dmakit with synthetic addresses and no timing, and do not touch real hardware.
Experiments
Choose an experiment, predict the next result, then take one step.
Watch the highlighted route and the RAM it ends in. The tables show
what is written in memory; kept translations show what the next
request may still use. Start again resets the whole example.
A link below opens its experiment at step zero.
What the NIC puts on the bus (here 0x10001230). Linux calls it an IOVA or DVA.
GPA, SPA
Guest-physical address (the VM's idea of RAM) and system-physical address (RAM in the usual case, a peer device's memory on a peer-to-peer path).
Guest table, stage 2
The guest's table maps device addresses to GPAs; the host's table, stage 2, maps GPAs to SPAs.
PASID
A tag on a request that picks which guest table to use.
VMM
The host program that runs the VM, such as QEMU. It sees every command the guest gives its virtual IOMMU.
DTE, DomainID
The device table entry says how to translate one device. Devices with different tables must use different DomainIDs; devices sharing tables may share one (not with GIoV=1).
Kept translation
A result the IOMMU (its IOTLB) or the device (its ATC, filled over PCIe ATS) keeps so it need not walk again.
Invalidate, flush
Tell a cache to drop kept translations; a flush here is an invalidation plus a wait (COMPLETION_WAIT) for it to finish.
MSI, IRTE
A device's interrupt is a write (MSI); the interrupt remapping table entry (IRTE) says where it goes.
Experiment
The three lower arrows are separate shortcuts, not one: a request without a PASID (host table only), a kept IOMMU translation (no table read) and the device's own ATC (no IOMMU lookup). Solid arrows are the request; dotted lines show which region owns each stage (G guest VM, H host VMM, P physical platform). Each region keeps its letter and border style, not just a colour.
The picture shows the last request the experiment sent.
Every cache entry, before and after this step (typed fields; a "What if" step compares with a fresh start)
Owner
Full key
Before
After
Change
Inspect this step
Three kinds of claim: an architecture rule (the source line names its specification revision), platform behaviour (Linux v7.3-rc5 or QEMU v11.1.0, where the source line names them) and a teaching-model choice (synchronous host commands, one fixed fixture, retained caches, no demand paging, no vCPU delivery).
What is virtual here? Two separate questions
Three independent choices. Pick one on each axis; a combination this page does not run is shown dashed and never changes the experiment above.
Linux iommufd objects (conceptual, not run here)
Who issues the access
How it is translated
This page
Emulated device
Software vIOMMU, then the VMM writes guest RAM
runs
Assigned device
Shadow mappings: the VMM copies guest mappings into host tables
explanation only
Assigned device
Nested: the physical IOMMU walks the guest's tables inside its own
runs
Separately: who handles the guest's commands? Here the VMM traps and forwards them (modelled). AMD §2.10 hardware can read the guest's command buffer itself (not modelled in this lab; dmakit's viommu_hw, viommu_hw_edges and viommu_hw_callbacks examples model it). Neither choice changes the other.
Real-system map (not run here): in Linux iommufd, an IOAS (an address space) backs a nesting-parent HWPT_PAGING; a HWPT_NESTED holds the guest's tables and attaches to a device. A VIOMMU object, one slice of one physical IOMMU, sits beside the parent, and a VDEVICE ties a physical device to its virtual ID.
Expand the table fetches (a conceptual picture, not a trace)
Table read: the guest's root table address is a GPA, so stage 2 translates it before the entry can be read (this fixture sets GCR3TRPMode).
Table read: each next guest table page is also a GPA and also goes through stage 2 first.
Refused here: in this experiment the guest's L3 table page is unmapped at stage 2, so that fetch fails and the walk stops.
Not reached: the data page's GPA would go through stage 2 only after a guest entry for it was read. No data address exists, so RAM A is unchanged.
After repair: with the table page mapped again the walk can finish and the write reaches RAM A.
The model reports the failing table page, not every table read; rows 1, 2 and 4 are explanation.
The handoff as a checklist
VM A owns the NIC. Its ATC holds a translation, usable.
Host blocks the device and flushes the domain, then waits. The ATC entry stays, but the IOMMU refuses the stale write (DTE I=0; its own step in the lesson). In the model, unassigning allows the old memory to be reused; the device is not ready for a new owner.
Reset. The ATC is empty. The lesson clears it by hand as a stand-in for a device reset, not a full FLR model.
Attach to VM B, then write. The ATC is empty, so this lesson's write is walked through B's tables. It does not refill the ATC with an ATS request.
The "What if" step skips steps 2 and 3: the kept entry still points into VM A's RAM. It runs as its own separate run.
Two lanes, two results
Data lane: device write → IOMMU translation (tables and kept entries) → RAM A. A data page flush reaches this lane only. Result: payload arrived or not.
Interrupt lane: MSI 0 → interrupt remapping entry (IRTE, with its own kept copy) → interrupt controller. INVALIDATE_INTERRUPT_TABLE reaches this lane only. Result: MSI delivered or refused.
Steps that write and then signal use both lanes. Beyond the controller, getting the interrupt to a vCPU (for example KVM routing and irqfd) is not modelled here. The picture above is the data lane.
The return tracks (a conceptual picture)
Completion: host command processing ends → the guest's wait finishes. It says nothing about the device's answer.
Event: host timeout event → the VMM rewrites IDs and addresses into the guest's terms → the guest's event log. With the log off, the record is lost.
Notification: guest log state → the guest's event interrupt, if on. With it off, the record is still logged.
Wait finished, device answered, device cache entry kept, host event recorded, guest event recorded and guest notified are six separate facts (the list under the picture, and the recorded tracks below it). A finished wait never makes memory safe to reuse. Only the forwarded ATC-invalidation timeout takes this route in the model; a DMA fault may stop elsewhere.
One command, four stages (a conceptual picture)
Look up and check: the VMM finds the physical device assigned to the guest's, and checks guest DTE writes: a guest DTE with Mode 4 is refused and the host's own DTE is kept (a configuration refusal, not a DMA fault). A command for an emulated-only device ends here too: logged as not forwarded, nothing is issued.
Rewrite IDs: the guest's DeviceID and DomainID become the host's, using this VM's own maps.
Execute: the host command goes to the physical IOMMU and device. Host-only DTE fields are not copied from the guest's.
Report completion: the guest's wait finishes.
What each side holds after that step
This step ran on a separate, fresh VM; the panels show that run.
RAM A · SPA 0x120001230
RAM B · SPA 0x130001230
Guest table · the VM owns it
Host stage 2 · the host owns it
Kept translations · IOMMU and device
What the VMM did
Not modelled in this lab: timing, and AMD's hardware vIOMMU (§2.10), in
which the IOMMU reads the guest's command buffer itself (dmakit's
viommu_hw examples model it); here the
VMM traps every guest command (what that
changes). One guest table that the CPU and the
device share, with three separate caches, has its own page:
shared guest tables.
Bytes a step wrote, and lines it added, are marked. RAM A starts as
aa and RAM B as bb; before a step that writes again the lesson puts
them back, so each write shows on its own. The payload is
VIOMMU-DEMO-0001.
The same question on Arm SMMUv3
Arm's name for it is a virtual SMMU: the guest owns stage 1, the host
stage 2. This runs on the site's Arm teaching model, which stores
table leaves rather than walking table memory.
What a virtual IOMMU is
A VM can have a virtual IOMMU, just as it has a virtual network card.
The guest programs it to limit where its devices may write. The host
still decides which real memory belongs to the VM, so a guest mapping
never grants access to arbitrary host memory. With ATS on, the host
also trusts the device's own cached translations.
When people say vIOMMU they may mean
What it is
On this page
The guest's IOMMU
A virtual device the guest programs: virtual AMD-Vi, a virtual SMMUv3 or virtio-iommu
The guest table, device table and commands
AMD's hardware vIOMMU
AMD 48882 §2.10: the IOMMU reads guest command buffers and writes guest PPR logs itself; host events reach the guest through INSERT_GUEST_EVENT
An iommufd object: the host's slice of one physical IOMMU for one VM
The host side, with its DomainID map
The guest-visible IOMMU and the way it runs are separate choices. For
an emulated device, the VMM runs the virtual IOMMU in software and
writes guest RAM itself. For an assigned device, the host either has
the physical IOMMU walk the guest's tables nested in its own (this
page) or copies them into its own tables (shadowing). A
bounce buffer (SWIOTLB) is neither: it copies bytes; it does not
translate or authorize a device.
How Linux does it
The VMM gives the VM its RAM. The host maps it into a stage-2 table,
the nest parent, that only the host edits.
The guest writes a device table entry in its own memory and sends
INVALIDATE_DEVTAB_ENTRY. The VMM traps it and reads the
entry.
The host checks it (Mode must be 0; guest translation needs a GCR3
root) and writes the real entry: the nest parent, a host DomainID
for the guest's, and the guest's GCR3 root, still a guest-physical
address (GCR3TRPMode).
The device's requests now walk the guest's tables, and each table
address and the final address go through the host's.
When the guest invalidates, the VMM passes it on with the host
DomainID. When the host changes stage 2, it invalidates every host
DomainID of that VM.
Steps 1, 3 and 4 and the stage-2 flush follow Linux v7.3-rc5
drivers/iommu/amd/nested.c
(validate_gdte_nested, set_dte_nested) and
iommu.c (iommu_flush_pages_v1_hdom_ids).
Step 2 is the VMM's job. Linux v7.3-rc5 has no interface yet for passing
a guest's invalidations on, so step 5 follows the commands AMD
§2.10.7 lets a guest send.
The same steps run in C++: choose viommu_nested.cpp in
the dmakit code lab.
What AMD's hardware vIOMMU would change
On this page the VMM does every guest-facing duty in software. AMD's
§2.10 hardware takes some of them over. It is supported when
0030h[vIommuSup]=1, turned on by
0018h[vIommuEn], then per device by
DTE[vImuEn], with DTE[GuestID] naming the
VM. The duties stay the same; only who performs them moves.
The experiments One command, two sets of IDs and The
host's event, retold for the guest run them.
Duty
Here: the VMM
With §2.10: the IOMMU
Read the guest's commands
Traps each one
Fetches the guest's command buffer; runs the five commands of §2.10.7
Guest IDs to host IDs
Its own maps
Guest Device Map and Guest Domain Map, which the host programs (§3.5)
Tell the guest about a host event
Builds a guest-safe record in the virtual IOMMU's log
The host still builds it, then sends INSERT_GUEST_EVENT with it (§2.4.9); the IOMMU writes it unchecked
Guest page requests
Not modelled in this lab (dmakit's viommu_hw_edges does)
Logged in the guest's PPR log when DTE[vImuEn]=1 (§2.10.6)
Interrupt the guest's driver
The virtual IOMMU's own interrupt
Posted to a vCPU through a two-level table on the IOMMU's own DTE (§2.10.4); needs GAEn (§2.10)
This page runs the duties, not the registers.
Trying it on real software
Not run by this site. These are starting points
checked against the QEMU manual, not results. Record your QEMU
version, kernel, full command line and guest log before trusting
anything you see.
The emulated AMD IOMMU runs on any x86 host, an
Intel N100 included, because QEMU does the translating in software
(the emulated-device path above). It may not offer everything this
page uses, such as PASIDs and GCR3 tables: the guest's
dmesg lists what its driver found. First ask your binary
what it has:
In the guest, dmesg | grep AMD-Vi and
ls /sys/kernel/iommu_groups show whether its driver
found the IOMMU (what the groups mean). dma-remap=on lets a
vfio-pci device work under guest mappings (AMD v1
paging domains); QEMU then shadows the guest's tables in the host's,
which is neither of this page's two paths and does not turn on this
page's PASID and GCR3 nested path.
The nested path (the assigned NIC above) needs an
AMD host whose IOMMU reports guest translation, GIoSup
and GCR3TRPModeSup (Linux checks all three before it
offers a nest parent), Linux v7.3-rc5's iommufd nested support and a VMM
that uses it. An N100 is an Intel part
(VT-d), so it cannot run this path. AMD's patch series for hardware
vIOMMU (§2.10), v5 of 14 September 2026,
is partial: it defers hardware-queue and extended interrupt
remapping support to later series. A series under review is not
proof of upstream support or of a working QEMU setup.
Linux v7.3-rc5 also refuses the nested attach while the host has PASID
enabled on the device, so check how your VMM exposes PASID before
expecting the PASID experiments to carry over.
Where each panel's real counterpart lives. Host
events: dmesg | grep 'AMD-Vi: Event logged'. With
CONFIG_AMD_IOMMU_DEBUGFS, under
/sys/kernel/debug/iommu/amd/: write a device ID to
devid, then read devtbl (its DTE) and
irqtbl (its IRTEs); iommu00/cmdbuf is the
command buffer.
Never unbind a device the host needs, such as its own network or
storage controller, to try passthrough: use a spare device with a
known way back.
QEMU v11.1.0 options
(commit 84f0721; the online master manual moves on): -device amd-iommu and its
dma-remap, intremap and
xtsup options.
Linux IOMMUFD
and the AMD IOMMU driver (nested.c,
iommufd.c),
all at Linux v7.3-rc5 (commit 72d3fcf, checked 2 October 2026;
the final v7.3 was not out).