Real buses: PCI, PCI Express and USB

The previous chapter worked out how a bus is timed and shared. This one looks at the real buses that connect devices to a PC: ISA, PCI, PCI Express and USB. It's also the story of one big change: from wide parallel buses shared by many devices to narrow serial links that each connect exactly two, and carry packets rather than signals.

ISA and PCI: the parallel era

The ISA bus of the IBM PC/AT ran at 8.33 MHz and moved 2 bytes per cycle: at most 16.7 MB/s. It was too slow for graphics by the early 1990s, and Intel designed its replacement, PCI (Peripheral Component Interconnect). The original PCI bus moved 32 bits per cycle at 33 MHz, 133 MB/s, and later versions doubled both the width and the clock, to about 530 MB/s.

PCI is a classic synchronous bus, and most of the previous chapter can be seen in it:

• Multiplexed address and data: the same 32 AD lines carry the address in the first cycle and the data afterward. A read needs a turnaround cycle in between, so that the master stops driving the lines before the target starts.
• The master starts a transaction by asserting FRAME#. It says it is ready with IRDY#; the target claims the address with DEVSEL# and says it is ready with TRDY#. Each side inserts wait states simply by holding its READY signal negated: a handshake on a clocked bus.
• Arbitration is centralized: every slot has its own REQ# and GNT# lines to an arbiter in the chipset.
• Every device has a 256-byte configuration space, read through special configuration cycles, which tells the system what the device is and which resources it needs. It made plug-and-play possible, and it's the part of PCI that survived.

PCI was long the main peripheral bus of PCs; PCI Express replaced it for everything fast in the 2000s. Today parallel PCI and its graphics offshoot AGP have disappeared from new computers, replaced by PCI Express.

PCI Express: lanes, not a bus

The problem with making a parallel bus faster is skew: the bits of a word, traveling on different wires, don't arrive at exactly the same time, and at high frequencies the difference becomes a large fraction of a clock cycle. The shared wires are also long and have many connectors along them, each reflecting part of the signal.

PCI Express (PCIe) solves both problems by giving up the shared bus:

• Each device has its own point-to-point link to a port of a switch or of the root complex (in the CPU or chipset). To attach more devices than there are ports, you add a switch, which has one upstream link and several downstream ones. The topology is a tree, like a switched Ethernet network.
• A link is made of lanes. Each lane has two differential pairs, one for each direction. A differential pair carries each bit as the difference between two wires driven in opposite directions, which cancels most of the noise picked up on the way. Both wires of the pair carry the signal; neither is a ground.
• A link is 1, 2, 4, 8 or 16 lanes wide (×1 to ×16; ×12 and ×32 were also defined but are practically unused). Consecutive bytes of a packet are spread across the lanes, and each lane is received and resynchronized on its own, so skew between lanes is corrected by the receiver instead of limiting the clock.
• There is no bus clock. The transmitter embeds its clock in the data stream and the receiver recovers it from the transitions. Most systems still distribute a common 100 MHz reference clock to the devices, from which both ends derive their bit rate.

Encoding and generations

For the receiver to recover the clock, the data must have enough transitions: a long run of identical bits is a flat line. 8b/10b encoding, used by PCIe 1.0 and 2.0, turns every byte into a 10-bit symbol chosen from a table that guarantees frequent transitions and as many 1s as 0s over time. It costs 20% of the raw rate: 2.5 GT/s (gigatransfers, raw bits per second per lane) carry 2 Gb/s of data. PCIe 3.0 switched to 128b/130b: 128 bits of scrambled data plus a 2-bit header, only 1.5% overhead. PCIe 6.0 went further, sending two bits per symbol with four voltage levels (PAM4) and grouping data into fixed-size flits protected by forward error correction.

The figures for 6.0 and 7.0 are before the few percent the flits spend on error checking. Every step since 3.0 has doubled the rate per lane. First-generation links ran at 2.5 GT/s; today's graphics cards and SSDs use 16 or 32 GT/s, and PCIe 6.0 is arriving in servers.

Packets instead of signals

On a PCI bus, a read is a pattern of control signals. On PCIe, it's an exchange of packets, built by a stack of layers inside each chip:

• the transaction layer creates transaction layer packets (TLPs): a 12- or 16-byte header (type, length, requester ID, address) and an optional payload of up to the link's maximum payload size (128 to 4096 bytes; 256 is common);
• the data link layer adds a sequence number and a 32-bit CRC. The receiver acknowledges good packets and asks for bad ones again (ACK/NAK), so errors on the wire are corrected by retransmission. It also sends flow-control credits: the sender may transmit only as many packets as the receiver has announced buffer space for;
• the physical layer adds framing and does the encoding described above.

A read is a split transaction: a memory read request TLP goes out with no data, and the device answers later with one or more completion TLPs carrying it. Meanwhile, the link is free for other traffic. There are also configuration requests, which reach the same configuration space that PCI had, and messages, which replace side-band signals. An interrupt from a PCIe device is not even a message: it is an ordinary memory write TLP to an address that the interrupt controller watches (MSI).

PCI Express on a real machine

The Mac Studio this chapter was written on has no expansion slots, but its I/O chips still hang off PCIe. macOS lists them in the I/O Registry:

$ ioreg -r -c IOPCIDevice -l | grep -E '"(IOName|model|compatible|IOPCIExpressLinkStatus|IOPCIExpressLinkCapabilities)"'
  |   "IOName" = "pci-bridge"
  |   "IOPCIExpressLinkCapabilities" = 7567380
  |   "compatible" = <"apcie-bridge">
  |   "IOPCIExpressLinkStatus" = 12306
    | |   "IOPCIExpressLinkCapabilities" = 4651026
    | |   "IOName" = "pci14e4,4434"
    | |   "IOPCIExpressLinkStatus" = 4114
    | |   "compatible" = <"wlan-pcie,bcm4387","wlan-pcie,bcm">
  …
      |   "IOName" = "ethernet"
      |   "IOPCIExpressLinkCapabilities" = 4680772
      |   "compatible" = <"pci106b,223","pci1d6a,4c0","pciclass,020000","lan-10gb">
      |   "IOPCIExpressLinkStatus" = 4116
      |   "model" = "Apple AQC113"
  …

Each device sits behind its own root port (apcie-bridge, Apple's PCIe bridge in the M2 Ultra): one link per device, no shared bus. The numbers are copies of the PCIe Link Capabilities and Link Status registers. Bits 3–0 give the speed (1 = 2.5 GT/s, 2 = 5, 3 = 8, 4 = 16) and bits 9–4 the width. Decoded:

The Ethernet chip could use four lanes, but the M2 Ultra's port gives it only one: at 16 GT/s, one lane carries 1.97 GB/s each way, more than the 1.25 GB/s of 10 Gb Ethernet. The last two status registers, and those of their root ports, read as all ones, the value a PCI read returns when nothing answers: those links were most likely powered down, with no SD card inserted and, as the USB listing below shows, nothing on the ASMedia controller.

A Linux virtual machine on the same Mac sees a different, virtual PCI bus, and Linux exposes each device's configuration space as a file:

$ cat /sys/bus/pci/devices/0000:00:01.0/{vendor,device,class}
0x1af4
0x1041
0x020000
$ head -c 64 /sys/bus/pci/devices/0000:00:01.0/config | od -A x -t x1
000000 f4 1a 41 10 06 00 10 00 01 00 00 02 00 40 00 00
000010 04 00 00 80 02 00 00 00 00 01 00 50 00 00 00 00
000020 00 00 00 00 00 00 00 00 00 00 00 00 f4 1a 41 00
000030 00 00 00 00 40 00 00 00 00 00 00 00 0e 01 00 00
000040

The first four bytes are the vendor ID 0x1af4 and device ID 0x1041, little-endian: a virtio network card. Bytes 0x09 to 0x0b, 00 00 02, are the class code 0x020000, again little-endian: network controller, Ethernet. Every PCI and PCIe device in the world starts its configuration space this way, which is how an operating system can find out what is plugged in before loading any driver. Bytes 0x10 to 0x1b are the device's base address registers, the programmable address decoders that the I/O chapter takes apart. (The lspci tool, which prints all this by name, isn't installed in the container; the files in /sys are what it reads.)

USB: one cable for everything else

PCI Express is for chips on a board. For external devices, the Universal Serial Bus replaced the serial, parallel, keyboard and mouse ports of the 1990s with one kind of cable. Its design choices still hold:

• the topology is a tree: a host controller with a root hub, then hubs and devices, up to 127 devices with 7-bit addresses assigned by the host when a device is plugged in;
• the host controls everything: a device never transmits unless the host asks it. Even "interrupt" transfers, used by keyboards and mice, are the host polling the device at a fixed interval;
• there are four transfer types: control (configuration), bulk (disks, printers; retried on error), interrupt (small, periodic, polled) and isochronous (audio, video: guaranteed bandwidth, never retried);
• the host schedules the traffic in frames of 1 ms at full speed, divided into eight microframes of 125 µs at high speed and above.

Each device describes itself with descriptors (vendor and product ID, USB version, packet sizes, power needs) read by the host at connection time, so that the operating system can pick a driver automatically.

The speeds have grown by a factor of more than 50,000:

USB 1.0, from 1996, already defined both Low Speed (1.5 Mb/s) and Full Speed (12 Mb/s); USB 1.1 followed in 1998. Many commonly quoted USB details belong to USB 1.x and 2.0: the 64-byte maximum data packet is the Full Speed limit (High Speed bulk packets carry 512 bytes and SuperSpeed ones 1024), and the four-wire cable is the USB 2 cable.

USB 1.x used two competing host controller interfaces, UHCI and OHCI, and USB 2.0 added EHCI. Since USB 3.0, a single standard interface, xHCI, handles every speed. This Mac's USB tree shows all of that:

$ system_profiler SPUSBHostDataType
    USB 3.1 Bus:
      Driver: AppleT8112USBXHCI
        Unnamed Device:
          Link Speed: 480 Mb/s
          USB Vendor ID: 0x0451
            RODE AI-1:
              Link Speed: 12 Mb/s
              Power Allocated: 2.5 W (500 mA)
            Unnamed Device:
              Link Speed: 480 Mb/s
                USB Controls:
                  Manufacturer: LG Electronics Inc.
                  Link Speed: 12 Mb/s
    …
    USB 3.1 Bus:
      Driver: AppleT8112USBXHCI
        USB Receiver:
          Manufacturer: Logitech
          Link Speed: 12 Mb/s
    USB 3.0 Bus:
      PCI Vendor ID: 0x1b21
      PCI Device ID: 0x2142
      Driver: AppleEmbeddedUSBXHCIASMedia3142

(Trimmed to the interesting lines. On this version of macOS, the older SPUSBDataType report prints nothing; the tree is under SPUSBHostDataType.) The "Unnamed Devices" at 480 Mb/s are hubs, with Texas Instruments' vendor ID 0x0451. An audio interface, a monitor's control channel and a Logitech wireless receiver all still talk at 12 Mb/s, Full Speed from 1996, plenty for audio samples, keystrokes and mouse movements. The 2.5 W allocated to the audio interface is the USB 2 limit of 500 mA at 5 V. And the last bus is the ASMedia controller from the PCIe table above: a PCIe device whose job is to be a USB host, with nothing plugged into it at the moment.

The Linux VM on the same machine has two virtual host controllers, one for each generation of signaling:

$ cat /sys/bus/usb/devices/usb1/{speed,version} /sys/bus/usb/devices/usb2/{speed,version}
480
 2.00
10000
 3.10

USB-C, USB4 and Thunderbolt

USB-C is a connector, not a protocol: a reversible 24-pin plug. Through it can travel USB 2 on its own pair of wires, USB 3.x or USB4 on up to four high-speed pairs, DisplayPort video in alternate mode, and Power Delivery, a negotiation on a separate configuration line that raises the bus voltage from 5 V up to 20 V (or 48 V in the extended range) for up to 240 W. Whether a given cable and port support 480 Mb/s or 80 Gb/s, 15 W or 240 W, is not visible from the plug.

USB4 is built on Thunderbolt 3, which Intel contributed to the USB Implementers Forum in 2019. Its main idea is tunneling: one USB4 link carries packets of several protocols at once: USB 3, DisplayPort and PCI Express. An external SSD or graphics enclosure on a Thunderbolt port is a PCIe device whose lanes happen to pass through a cable. Thunderbolt 4 (2020) is a certification of USB4 at 40 Gb/s with PCIe tunneling required, and Thunderbolt 5 (2023) runs at 80 Gb/s, up to 120 Gb/s in one direction for displays.

$ system_profiler SPThunderboltDataType
Thunderbolt/USB4:
    Thunderbolt/USB4 Bus 5:
      Vendor Name: Apple Inc.
      Device Name: Mac Studio
      Port:
          Status: No device connected
          Speed: Up to 40 Gb/s
          Receptacle: 6
    …

The M2 Ultra Mac Studio has six such buses, one per USB-C port (receptacles 1 to 6), each rated up to 40 Gb/s and able to tunnel PCIe to whatever is plugged in. The line between an internal bus and an external cable, so sharp in the days of ISA and PCI, has nearly disappeared.

Takeaways

• ISA and PCI were parallel, synchronous, shared buses: multiplexed address/data lines, FRAME#/IRDY#/TRDY# handshakes, per-slot REQ#/GNT# arbitration. PCI's configuration space is the part that survives.
• PCI Express replaces the bus with point-to-point lanes of differential pairs, switched like a network, with the clock embedded in the data. Encoding went from 8b/10b (20% overhead) to 128b/130b (1.5%) to PAM4 flits; each generation since 3.0 doubled the rate, up to 128 GT/s per lane for PCIe 7.0.
• PCIe moves packets: TLPs with a header and a payload, protected by a sequence number, a CRC and retransmission, paced by credits. Reads are split transactions, interrupts are memory writes.
• USB is a host-polled tree of hubs and devices, with control, bulk, interrupt and isochronous transfers, from 1.5 Mb/s in 1996 to 80 Gb/s with USB4 v2. On this Mac, mice and audio still run at 12 Mb/s.
• USB-C is a connector that can carry USB 2, USB 3.x, USB4, DisplayPort, Thunderbolt and up to 240 W. USB4 and Thunderbolt tunnel PCI Express through the cable.