ASAravind Sivakumaran
Theme

Theme and appearance

Appearance
Palette

Pick any colour. The displayed accent adjusts for readable contrast in light or dark mode.

← IOMMU star map

IOMMU / COMMANDS, LOGS & INTERRUPTS

Who writes the next entry?

Ownership, ordering and completion make a buffer useful.

What claims does this page make? 3 points

Four rings, two directions

Structure Producer → consumer Entry Purpose
Command buffer Software → IOMMU 16 bytes Invalidate cached state, wait for completion, or return a page response.
Event log IOMMU → software 16 bytes Report events such as IO_PAGE_FAULT. Not a queue of successful DMA completions.
PPR log IOMMU → software 16 bytes Deliver an incoming peripheral page request for software to service.
Guest virtual APIC (GA) log IOMMU → hypervisor 8 bytes Notify software about an interrupt targeting a non-running guest vCPU.

The producer publishes Tail after writing an entry. The consumer advances Head after taking it. Head = Tail means empty in these examples; one slot is reserved so full can be distinguished from empty.

Each illustration uses eight slots to make wraparound visible. Real allocation sizes, alignment, enable/status registers, memory ordering and overflow recovery must follow the relevant hardware specification.

Faults & recovery: every command and buffer case → · All ten commands, capability gates, illegal commands, stop/restart, overflow and pointer misuse. Browse the error-scenario catalogue.

Invalidation sequence · fetch is not completion

Start with a changed mapping and stale caches. Queue the three-command sequence, fetch it, then complete each command. Try the DMA gate between those actions.

IOMMU cache
NIC ATC
DMA gate

Mapping changed. DMA waits for invalidation completion.

What does this completion prove?

You observedIt provesIt does not prove
The IOMMU fetched the command (Head moved)The entry left the ring.That it ran. Fetch is not completion.
The command retiredThat command's handler ran in the model.That later commands ran, or that the device's own cache obeyed.
COMPLETION_WAIT retired (store or interrupt)The commands queued before it have retired: a synchronization point.That DMA already in flight has drained. Stop and drain the device first, as in the SR-IOV labs.
A completion-queue record or interrupt from the deviceThe device says it finished its own work.That the IOMMU's invalidation finished, or the reverse. They are separate paths.

Step it one command at a time in dmakit's deferred_commands example (README): queued, applied, observed.

Experiment with individual commands and wraparound

Queue up to seven entries, then fetch to free slots. Consuming slots does not finish the commands. This teaching processor enforces its chosen safe invalidation order. COMPLETION_WAIT can finish without the needed invalidations, but the separate DMA safety gate stays closed. Reset restores a known starting state.

Command catalogue and expected effects
Which invalidate command do I need?
  • INVALIDATE_IOMMU_PAGES targets the IOMMU’s cached address-translation state.
  • INVALIDATE_IOTLB_PAGES requests invalidation in the peripheral’s translation cache. Device acknowledgment matters.
  • COMPLETION_WAIT gives software a synchronization point after earlier commands. An empty command ring alone is insufficient.
  • INVALIDATE_DEVTAB_ENTRY applies when cached device-table information changes. This lab keeps the DTE fixed.
  • INVALIDATE_INTERRUPT_TABLE applies to cached interrupt-remapping information after its table changes. It is separate from data-page invalidation.

This model invalidates the entire small session cache. Real commands select scopes such as domain, device, PASID and address range. Submission may be FIFO while execution overlaps; here completion is serial so dependencies remain visible.

Events and faults · diagnose before clearing

Choose a public event type to inspect its cause and recovery expectation. Injecting it below writes a semantic record to this page’s independent event log; it does not alter the Translation lab.

Completion, timeout and restart expectations

Do not put COMPLETION_WAIT between every command. Queue the ordered invalidate sequence, then place a wait where software needs a synchronization boundary. Software must observe the requested completion value or notification; command Head reaching Tail is not proof of completion. A wait does not fix an omitted invalidation.

This fixture withholds the acknowledgment and declares no InvTimeOut timer. If a device does not acknowledge invalidation, keep affected DMA/page reuse blocked. A software polling deadline must report failure rather than assume success. Command hardware errors and illegal commands stop command processing; clearing EventLogInt alone does not restart it.

Preserve the offending event and command context. Wait for the command engine to stop, disable CmdBufEn, repair the cause and buffer configuration while stopped, then re-enable according to §2.4. Hardware may have fetched beyond the bad command, so do not infer the culprit solely from the current Head.

Log overflow is different from a malformed record. Preserve unread information, drain or resize as appropriate, observe the log’s stopped state, configure while disabled, clear its overflow W1C flag and re-enable logging. Event, PPR and GA restart rules differ; a generic “clear everything” write is not a recovery procedure.

This lab models selected diagnostics and a blocked/recovered state. Fault-specific raw flags, hardware retries, feature-dependent events, overflow races and real MMIO accesses are outside the executable model. Consult the linked manual sections for platform implementation.

Log lab · hardware produces, software consumes

Select a log and generate a record. Watch Tail advance; consume the oldest record to move Head. Change logs to start a separate example.

Record labels are semantic examples. “Full” stops this demonstration without overwriting unread entries; hardware-specific overflow handling, dual buffers and auto-response options are not emulated.

PRI lifecycle · request, service, response

  1. Page out: set host PR=0 (or guest P=0), stop new affected DMA, publish the table change, invalidate translations and wait. Do not reuse backing memory while old accesses may still be in flight.
  2. Attempt access: ATS finds no usable page. No AT=10 memory request copies data from it. A PRI-capable NIC submits a page request; the IOMMU appends one PPR log record.
  3. Consume: software copies the request identity into a work item and advances the PPR Head. Pending log entries drop from one to zero; the work still needs a response.
  4. Page in: restore backing and PR/P=1, publish the update, invalidate IOMMU and device translations, then observe COMPLETION_WAIT.
  5. Respond: software submits COMPLETE_PPR_REQUEST using the originating device, PASID where applicable, and page-request group. The device receives the response code software chose (Success, Invalid Request or Response Failure); this response is not another incoming PPR entry.
  6. Retry: after Success, obtain a fresh ATS translation and issue DMA. A page response itself carries no packet payload.
Run the page-out / page-in example →

An ATS page-miss response is different from a failed AT=00 memory request and its IO_PAGE_FAULT event. PRI recovery requires the device and platform to support and enable the relevant facilities.

APIC and AVIC are not extra log rings

The local APIC tracks interrupt state, including pending and in-service vectors. AMD AVIC accelerates virtual interrupt handling using virtual APIC backing state and supporting tables. An interrupt-remapping table maps device interrupt requests; it is an indexed table, not a producer/consumer ring.

The IOMMU’s GA log is the related ring: when enabled and a guest target is not running, it can record a DeviceID and GATag so the hypervisor can act. A GA notification and the pending interrupt in the guest’s backing state are distinct.

Fixed mock vector 0x51, DeviceID 0x0200 and GATag 0x1234. This example tracks a pending vector and log notification; it does not emulate AVIC tables, vCPU scheduling, interrupt priority or full APIC registers.

Software setup · physical memory and MMIO registers

MMIO is the public programming interface here. Discover the platform’s IOMMU register window through firmware/PCI capability information. The register’s physical address is the discovered MMIO base plus its offset.

From a software allocation to hardware ownership
  1. Discover and check: obtain the IOMMU MMIO window and supported features. Keep the CPU mapping of that window separate from the physical address of a buffer in RAM.
  2. Allocate backing: use the platform’s DMA/coherent allocation mechanisms to obtain stable, contiguous physical backing with the required alignment and address width. A C++ pointer alone is not a hardware buffer address. Keep the allocation alive while hardware can reference it.
  3. Stop before reprogramming: quiesce affected activity and follow the buffer-specific disable/Run=0 procedure. Do not change a running log’s base, length or hardware-owned Tail.
  4. Initialize: initialize the storage and Head/Tail; program its SPA and length, not its CPU linear address. The command buffer uses 4 KiB alignment and ComLen=8 for 256 × 16-byte entries. The eight-slot illustrations below are intentionally smaller teaching views.
  5. Enable: configure IommuEn, CmdBufEn and the logs in use. PPR additionally needs PPREn, PPRLogEn and enabled device facilities. Configure interrupt enables if interrupts are used; polling remains a distinct notification strategy.
  6. Publish a command: reserve a free slot, write its complete 16 bytes, use the platform’s required memory/DMA ordering, then publish the new Tail. Hardware must never see Tail pointing past a partially written record.
  7. Fetch and complete: hardware fetches at Head and advances Head. Execution can overlap; use a completion token or interrupt when software needs proof that older work finished.
  8. Drain a log: observe Tail with the required ordering, copy/read complete records up to Tail and publish the consumed Head. Recheck for arrivals. Acknowledging status does not consume unread records.
Pointer arithmetic and addresses

Head and Tail are byte offsets within the buffer. Entry address = buffer SPA + offset. For 16-byte records, the next offset is (offset + 16) mod buffer_bytes; the GA log uses 8-byte records. For a 4 KiB command ring, offset 0xFF0 advances to 0x000. Keep one slot free to distinguish full from empty.

The CPU can write through a mapped linear pointer while the IOMMU reads the same backing through its SPA. The links between allocations and registers are separate from the actual DMA data mappings. The GA tail-address register at MMIO +0x00E8 points to a memory location for a hardware-written tail copy; it is not the tail offset itself.

Global controls and acknowledgment

MMIO +0x0018 is the Control register; +0x0030 reports Extended Features; +0x2020 is Status. Run bits are read-only state. Notification and overflow bits include write-one-to-clear (W1C) fields: EventLogInt bit 1, ComWaitInt bit 2, PprInt bit 6 and GAInt bit 10.

Write a deliberate mask of the W1C bits you intend to acknowledge. Writing zero does not clear them; blindly writing back an entire status value can acknowledge unrelated conditions. Draining entries, clearing notification status, clearing overflow, restarting logging and sending APIC EOI are separate operations. Use a drain/ack/recheck loop so racing arrivals are not missed.

AMD 48882 / §2.7

Decode a GA log entry

A guest virtual APIC notification is one 8-byte record. Revision 3.11 defines GA_GUEST_NR (opcode 1): an interrupt targeted a guest that was not running. This is a GA log type, not an event-log fault code or an interrupt-remapping entry.

Hexadecimal uses 0x. The displayed word is most-significant digit first; bytes below follow increasing memory addresses (little-endian).


        

GATag is a software cookie. It comes from the IRTE; it is neither a page address, a PASID, nor the pending vector. The vector is recorded in the guest APIC backing state. Reading this log does not clear that pending state or complete EOI.

Full GA field, register and restart reference · Why Arm has no matching GA log

Scope and public references

All controls modify browser memory only. Command and log words are semantic records, not an MMIO driver or a binary-complete emulator. Register calculations are synthetic examples and must not be used as discovered platform addresses. Hardware processing, PCIe acknowledgments and software polling are represented as explicit steps; timing, cache-line coherency and error recovery require platform-specific handling.

Back to the IOMMU star map · Inspect the current DMA configuration in C++