What is this card, and what may it do?

One line of lspci

Plain lspci -nn prints one line per device. Press each part.

The three numbers bus:device.function are also the device's requester ID: the return address on every request it sends. The IOMMU uses exactly this number to decide whose page table applies.

The nameplate: configuration space

Every function has 4096 bytes of configuration space. The first 64 are a fixed header, laid out the same on every endpoint (bridges use a second layout). Here they are, as lspci -x prints them:

00: e7 1e 01 00 06 04 10 00 01 00 00 02 10 00 00 00
10: 00 00 80 fe 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 e7 1e 01 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 01 00 00

Bytes are little-endian: e7 1e at offset 00 is vendor ID 1ee7; 06 04 at 04 is the Command register, 0406; 00 00 80 fe at 10 is BAR0, the address where the card's own registers appear, fe800000; and 40 at 34 points to the first capability.

Everything else a device can do is described by capabilities: small structures chained by pointers. The classic list lives in bytes 0x40 to 0xff; PCI Express added an extended list from 0x100 to 0xfff, where ATS, PASID and SR-IOV live. lspci prints each as Capabilities: [offset] name. Follow the pointers as lspci does:

That walk is why lspci -vvv prints Capabilities: [40] Power Management, [50] MSI-X, [70] Express, then [100 v1] Advanced Error Reporting, [148 v1] Address Translation Service (ATS) and so on: the number in brackets is where the walk found it.

Run lspci -vvv as root. Linux lets other users read only the first 64 bytes, so without root the capabilities are missing.

The Command register: may it start requests?

The Control: line of lspci -vv is the Command register, decoded bit by bit. A + is a 1 and a - is a 0. Three bits matter today:

• Mem: the card answers when the CPU reads or writes its BARs. Off, its registers vanish.
• BusMaster: the card may start requests of its own. Every DMA read and write needs it, and so do MSI and MSI-X interrupts, because an MSI is a memory write. Linux drivers turn it on with pci_set_master().
• DisINTx: the old interrupt wire (INTx) is off. This does not enable MSI or MSI-X; check those capabilities' Enable flags separately.

Flip them. Each switch runs the setpci command you see in the log.

What lspci prints

Prerequisites controlled here

setpci

Compare the two writes: COMMAND=0 clears Mem too; =0:4 uses a mask and changes only bit 2.

With BusMaster off the card cannot even ask, so the IOMMU never sees a request. Turning it on is the first step of Before DMA; whether the request then reaches memory is the IOMMU's decision, which the IOMMU foundations follow.

Regions: where the card's memory appears

Each Region n: line is one BAR: a window of addresses the card answers. Here are the regions of a GPU a reader sent in:

Region 0: Memory at 6000000000 (64-bit, non-prefetchable) [size=16M]
Region 2: Memory at 4000000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at 3000 [size=64]
Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]

lspci builds each line from a BAR's bits. The lowest four bits describe the window; the rest are its address. Pick a region and watch the bits turn into the line:

Prefetchable or not

The one bit that changes how the CPU may touch a region is bit 3. Non-prefetchable memory may have side effects: reading a FIFO register can take the entry out, reading a status register can clear it. Every read must happen exactly as the driver wrote it, once, so the OS maps it uncached. Region 0 holds the GPU's control registers.

Prefetchable memory behaves like RAM: reading it changes nothing, so the CPU and bridges may read ahead and merge writes. The OS may map it write-combining, which is how a driver streams textures into Region 2, the GPU's video memory window. Linux offers /sys/bus/pci/devices/…/resourceN_wc only for prefetchable regions. The experiment Prefetchable or not above shows what goes wrong when a FIFO is read ahead.

So how is a non-prefetchable Region 0 at 6000000000, above 4 GiB? Behind a bridge it could not be: a bridge's non-prefetchable window has only 32-bit address registers, and Linux puts every non-prefetchable BAR there. But this GPU is built into the processor (00:02.0, an Intel N100), on bus 00 with no bridge above it, so it takes addresses straight from the windows firmware gives the host bridge, which can be anywhere.

The last two lines

• I/O ports at 3000 [size=64]: bit 0 of the BAR is 1, so this window is in the x86 I/O port space (the in and out instructions), not memory. PCIe keeps it for old software such as VGA; new devices should not need it.
• Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]: the option ROM, firmware code the card carries for booting. [virtual] means the address comes from the OS while the card's own ROM register reads 0: on x86, Linux points the boot display's ROM at the copy the firmware left at 0xC0000, 128 KiB long (pci_fixup_video). [disabled] means the ROM's enable bit is 0. Both are normal.

Generations and lanes: LnkCap and LnkSta

A link is a bundle of lanes, each a pair of wires in each direction, like the lanes of a road. The width is how many lanes (x1, x2, x4, x8, x16); the generation is the speed limit on each one. Each generation roughly doubles what a lane carries:

GT/s counts transfers on the wire, not data. Gen 1 and 2 spend 10 bits to send 8; Gen 3 to 5 spend 130 to send 128. Gen 6 doubles again by sending four voltage levels (PAM4), so each symbol carries two bits: 64 GT/s is 32 Gbaud. It packs traffic into fixed-size FLITs with error correction; Linux counts its rate as the full 64 Gb/s. These are the numbers Linux itself uses to report a link's bandwidth, before packet headers.

A link trains to what both ends support: the lower generation and the fewer lanes. lspci shows what the card could do in LnkCap and what it got in LnkSta, and marks the gap (downgraded). Put a Gen 4 x16 card in a slot wired as Gen 3 x4:

A slot can be x16 long and x4 wired, and slots that share lanes with an M.2 socket drop to fewer when both are used; the board manual says which. Some devices, GPUs especially, also slow their link when idle to save power, so read LnkSta while the device is busy.

Every line of lspci -vvv

The full output is long, but it has a pattern. Each capability is a block; inside it, registers come in threes: Cap says what the device can do, Ctl what software turned on, Sta what is happening now. So DevCap, DevCtl, DevSta for the device, LnkCap, LnkCtl, LnkSta for the link, and the same again with a 2 for features added later. A feature works only when Cap has it and Ctl turns it on.

Below is the real lspci 3.15 output for this page's card and for the bridges above it. Press any line to see what it means, flag by flag, and the setpci command that reads it. Or paste your own: it is explained in this page and sent nowhere.

The fourth example is real: the integrated GPU of an Intel N100, as a reader's machine printed it. Read its DevCap2 and its PASID lines together. The GPU has a PASID capability but says EETLPPrefix-, and Linux's pci_enable_pasid() refuses any device without prefix support (unless a quirk says otherwise). Intel's IOMMU driver asks for PASID first and then turns ATS on, so this GPU ends up with PASIDCtl: Enable- next to ATSCtl: Enable+.

Changing a bit with setpci

Every switch above ran a command shaped like this:

setpci -s 03:00.0 ECAP_PASID+6.w=1:1

• -s 03:00.0 picks the device, as lspci printed it.
• ECAP_PASID finds the extended capability with ID 001b, by walking the list like the walk above. CAP_EXP finds the PCI Express capability in the classic list; plain names such as COMMAND are fixed offsets. setpci --dumpregs lists them all.
• +6 is the offset inside that capability: PASID Control.
• .w is the width: .b 8 bits, .w 16, .l 32.
• =1:1 writes value 1 under mask 1. Only the mask's bits change; without a mask the whole register is overwritten. Numbers are always hexadecimal.
• Without =, setpci reads and prints the value.

Once you have a dump, everything after it is safe: lspci and setpci read the saved copy as if it were the machine, and never touch the hardware. Capturing it is the one read that carries the risk above:

# capture all 4096 bytes: reads the live device
sudo lspci -xxxx -s 03:00.0 > card.txt
# decode the copy: no root, no hardware
lspci -F card.txt -vvv
# read from the copy
setpci -A dump -O dump.name=card.txt -s 03:00.0 COMMAND

Writes to a dump are refused, which is what the console on this page is for. For more detail: PASID, ATS, PRI, SR-IOV and ASPM, the switches that make a device an IOMMU client.