ASAravind Sivakumaran
Theme

Theme and appearance

Appearance
Palette

Pick any colour. The displayed accent adjusts for readable contrast in light or dark mode.

← All notes

FIELD NOTES / PCIe · ONE GUIDE, NINE CHAPTERS

One N100. Every link a request crosses.

Nine chapters on how a PCIe device is found, configured and allowed to write to memory, drawn from a reader's real lspci on an Intel N100 mini PC (the same chip serves this site). Each has a lab on an emulated card you can change and break.

Start with Reading lspci → ≈ 12 min, no setup

The N100's PCIe tree: memory above the IOMMU above the root complex. On the root bus sit the built-in graphics and five root ports, which are bridges; below each root port, alone on its own bus, is one card: the Renesas USB 3 controller on bus 01, two Ethernet ports on buses 02 and 03, Wi-Fi on 04 and an NVMe drive on 05. Memory (DRAM)where an allowed DMA write lands Intel IOMMU (VT-d)The IOMMU guide → Root complex, in the N100 chip: every device's DMA writes pass through it UHD Graphics00:02.0, no bridgeFeatures →Root port00:1c.0 → bus 01Bridges →Renesas USB 301:00.0Reading lspci →Root port00:1c.2 → bus 02Bridges →Etherneton bus 02Before DMA →Root port00:1c.3 → bus 03Bridges →Etherneton bus 03Before DMA →Root port00:1c.6 → bus 04Bridges →Wi-Fion bus 04Before DMA →Root port00:1d.0 → bus 05Bridges →NVMe driveon bus 05Before DMA →

A reader's N100, from its lspci -tv (on a narrow screen, swipe it sideways). A device's DMA write climbs this tree to memory; each box with an arrow opens the chapter that explains it, and Memory is only where the writes land. What each box is.

A short first route

Three chapters, about 40 minutes. One outcome each; the Next link at the end of each chapter follows this route.

  1. Reading lspci Read one card's address, class and BusMaster bit.
  2. Before DMA: PCIe setup See the one switch that lets it write to memory.
  3. Boot discovery Learn how it got its address at boot.

Jump to a topic

Already working with devices? Go straight to the job in front of you.

Decode a device
Reading lspci
Understand the link
Physical layer, Gen 1 to 7
Bring up or enumerate a host
Boot discovery, Enumeration
Set up PASID, ATS and SR-IOV
PASID, ATS & SR-IOV, SR-IOV: two tenants
Debug bridges and AtomicOps
Bridges & AtomicOps
Explore CXL
CXL
Check what DMA needs first
Before DMA: PCIe setup

The full course, in order

All nine chapters, front to back. This is longer than the first route; it covers the link, boot, SR-IOV, bridges and CXL too.

  1. Reading lspci ≈ 12 min
  2. Physical layer, Gen 1 to 7 ≈ 20 min
  3. Boot discovery ≈ 25 min
  4. Enumeration ≈ 20 min
  5. PASID, ATS & SR-IOV ≈ 12 min
  6. SR-IOV: two tenants ≈ 30 min
  7. Bridges & AtomicOps ≈ 12 min
  8. CXL ≈ 25 min
  9. Before DMA: PCIe setup ≈ 3 min

Where the request goes next: the IOMMU guide picks it up at the IOMMU. All three topics in one sitting: One NIC, three stops follows this card through PCIe, the IOMMU and a sleep and wake.

The N100's hardware

Model handoff: this tree is a real Intel N100's lspci, which has Intel's VT-d. The IOMMU chapters it links to run an AMD-style model with AMD's terms (DTE, event log). The same device identity gives continuity; it does not make the two translation architectures interchangeable.