Why does it work in one slot and not the other?

The tree: lspci -tv

lspci -tv draws the bridges. For the card behind a switch it prints one line. Press each part:

A switch is not one device but several bridges: one upstream port facing the root and a downstream port for each link below, joined by an internal bus. AMD's motherboard chipsets are switches, and so is the first thing above the GPU on recent AMD Radeon cards: lspci lists an Upstream Port and a Downstream Port of a PCI Express switch before the GPU itself. A card plugged straight into a CPU slot has only the root port above it; a device built into the processor (bus 00) has none.

A bridge in lspci

A bridge's configuration space has a different header (type 1). Instead of the card's six BARs it has the bus numbers and the address windows that decide what it passes down:

00:01.1 Class 0604: Device 1ee7:0100 (rev 01)
	Bus: primary=00, secondary=01, subordinate=03, sec-latency=0
	Memory behind bridge: fe800000-fe8fffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
	Capabilities: [40] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap2: … ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 4
			 AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS-
		DevCtl2: …
			 AtomicOpsCtl: ReqEn- EgressBlck-
	Capabilities: [100 v1] Access Control Services
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-

• Bus: configuration requests for buses 01 to 03 go down through this bridge.
• Memory behind bridge: CPU reads and writes in fe800000 to fe8fffff go down; it holds the card's BAR0. Anything the card sends to other addresses goes up, toward the IOMMU and RAM.
• Express … Root Port: the type the kernel's path checks look at. A switch says Upstream Port or Downstream Port.
• EETLPPrefix, AtomicOpsCap, AtomicOpsCtl, ACSCtl: the four lines this chapter is about.

The line explainer on the lspci page opens every line of this root port and of the switch's upstream port. Each feature in the table needs the right bits not just on the card but on every port above it:

The path lab

Change what the root port and the switch support. Both checks run as Linux runs them, from the same registers lspci prints:

What lspci prints

AtomicOps and PASID depend on different parts: turn off prefix support and AtomicOps still work; block AtomicOps and PASID still works. A missing ACS capability does more than stop PASID: without it the switch may pass requests between its own ports, past the IOMMU. Linux usually puts the devices below it in one IOMMU group, but its grouping looks at the path above each device, not at its neighbours, so separate group numbers alone do not prove they are isolated (the known gap).

A real root port: a reader's N100

A reader sent the lspci of an Intel N100 mini PC. It lists five PCI bridge entries, all root ports (00:1c.0, 1c.2, 1c.3, 1c.6, 1d.0), and five devices on buses 01 to 05: a USB controller, two Ethernet ports, Wi-Fi and an NVMe drive. No switch ports, so each card has exactly one bridge above it. The GPU is built in at 00:02.0 and has none. Root port 00:1c.0, whose bus 01 holds the Renesas USB controller, prints:

LnkCap:	Port #2, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
LnkSta:	Speed 5GT/s, Width x1
DevCap2: … ExtFmt- EETLPPrefix-
	 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-

Read it with the table above:

• AtomicOps: this root port completes none, so pci_enable_atomic_ops_to_root() fails for any card behind it. A GPU whose compute stack needs AtomicOps would be refused in this slot.
• PASID: EETLPPrefix- at the root port, so no card behind it gets PASID from Linux.
• ACS is on, as Linux sets it with an IOMMU, so the card below is isolated: the port reports its own IOMMU group (5).
• The link: the port could run Gen 3 x1 but runs at 5 GT/s, Gen 2. The uPD720201 USB controller below it is a Gen 2 part, and the port's Target Link Speed is 5GT/s too.

The whole output, and the same machine's GPU, are in the line explainer.

AtomicOps: FetchAdd, Swap and CAS

An ordinary update of memory is two requests: read the value, then write the new one. Between them, someone else can write, and one of the updates is lost (the Lost update experiment above shows it). An AtomicOp is one request: the completer, the root complex for RAM, reads, changes and writes the value with nothing in between, and sends the old value back.

The address must be a multiple of the operand size (natural alignment). These three are enough to build counters, queues and locks that a device and the CPU share, which is what GPU compute stacks use them for. Try them on 32 bytes of RAM; numbers are hexadecimal, as in setpci, and memory is little-endian:

Try a 64-bit FetchAdd at address 4: it is refused, because 4 is not a multiple of 8. A CAS whose compare value is wrong returns the current value and writes nothing; software reads that value and tries again.

Turning AtomicOps on, and checking your machine

Before a card may send AtomicOps to RAM, all of these must hold:

1. The card's driver knows how to use them. There is no "can request" bit; the driver asks.
2. The root port completes the sizes asked for: AtomicOpsCap: 32bit+ 64bit+ in its DevCap2. amdgpu asks for both.
3. Every switch port in between says Routing+, and no upstream port says EgressBlck+.
4. The card is a physical function: a VF uses its PF's setting.
5. The driver calls pci_enable_atomic_ops_to_root(), which checks 2 to 4 and then sets AtomicOpsCtl: ReqEn+ on the card.

On a real machine, read each part:

# which bridges sit above the card
lspci -tv
# the root port: Routing+ 32bit+ 64bit+ ?
sudo lspci -vvv -s 00:01.1 | grep AtomicOps
# the card: ReqEn+ once its driver asked
sudo lspci -vvv -s 03:00.0 | grep AtomicOpsCtl
# amdgpu: "PCIE atomic ops is not supported"
sudo dmesg | grep -i atomic

When the check fails, amdgpu says so, and ROCm's compute driver (KFD) skips GPUs that need AtomicOps, logging PCI rejects atomics. The usual cause is a root port or a riser's switch that does not support them, which is why the same GPU can work in one slot and fail in the next.

setpci -s 03:00.0 CAP_EXP+28.w=40:40 would set ReqEn by hand, but it skips the checks, not the hardware: AtomicOps the path cannot carry are still refused, as errors. PASID is the same: the PASID bench can turn PASIDCtl on, but behind a switch without prefix support, requests that carry one are rejected.