I/O chips and address decoding

The previous chapters covered memory chips, CPU chips and the buses between them. The last piece is how the CPU reaches the outside world: I/O interface chips, and the address decoding that decides which chip answers when the CPU puts an address on the bus. A small embedded computer makes a clear example; the same principles run a modern PC, only hidden inside the chipset and the PCI Express configuration.

I/O chips are a few registers

To the CPU, an I/O chip looks like a handful of registers at fixed addresses. Most have three kinds:

• data registers, through which bytes go in and out;
• status registers, which say whether the device is ready, busy or in error;
• control registers, which set the device's mode: speed, direction, interrupts on or off.

Everything else (shifting bits onto a wire, sampling a sensor, waiting for a slow printer) happens inside the chip.

The UART (universal asynchronous receiver-transmitter) is the classic example. It takes a byte written to its data register and sends it bit by bit on a serial line: a start bit, 8 data bits and a stop bit, 10 bits per byte in the common "8N1" format. Old terminals and modems used speeds of 50 to 19,200 bits/s. The PC's 16550 UART, with its standard 1.8432 MHz crystal, went up to 1.8432 MHz / 16 = 115,200 baud, which is 11,520 bytes per second in 8N1; the USB-to-serial chips that replaced it run at several megabaud. UARTs are anything but dead: every microcontroller has one or more, and they are the first thing an engineer connects to when bringing up a new board. Linux still names its UART driver after the chip of the original IBM PC, and the Linux VM used for these chapters has it:

$ ls /sys/bus/platform/devices
40000000.pci
Fixed MDIO bus.0
alarmtimer.0.auto
gpio-keys
hypervisor
psci
psci-cpuidle
serial8250
timer
vhci_hcd.0

serial8250 is the driver for the 8250 UART and its descendants, including the 16550.

The PIO (parallel I/O) is the other classic interface chip, modeled on Intel's 8255A. It has three 8-bit ports, A, B and C, whose 24 pins can each be read or driven, and a control register that sets which ports are inputs and which are outputs. On the CPU side, it has 8 data pins, two address pins A0–A1 that select one of its four registers (ports A, B, C and control), a chip select, read and write strobes, and a reset. (A simplified model needs only 3 control bits, one direction bit per port. The real 8255A's control word has 8 bits, because it also selects among three operating modes and can split port C into two independent halves.) The original IBM PC used an 8255 at I/O addresses 0x60 to 0x63 to read the keyboard and the configuration switches and to drive the speaker, and to this day a PC's keyboard controller answers at port 0x60.

Memory-mapped or port-mapped

How does the CPU address those registers? There are two answers.

Port-mapped I/O gives devices a separate address space, reached with special instructions. x86 has 65,536 I/O ports, and the instructions in and out:

in   al, 0x60        ; e4 60      read a byte from port 0x60
out  0x80, al        ; e6 80      write a byte to port 0x80
mov  dx, 0x3f8       ; 66 ba f8 03
out  dx, al          ; ee         port number in dx: 0x3f8 is the first serial port
in   al, dx          ; ec

(Encodings from clang -target x86_64-linux-gnu.) On the bus, the CPU signals that the address is an I/O port rather than a memory address with a dedicated control line (M/IO# on x86 chips, IORQ# on the Z80), so memory chips ignore the cycle and I/O chips answer. Port 0x80 has an odd second life: firmware writes a progress code to it during boot, which a debug card plugged into the motherboard can display.

The simulator on this site doesn't implement in and out. That's faithful in its own way: a normal program can't execute them either. In user mode, they raise a general-protection fault, which Linux turns into a SIGSEGV, unless the kernel has explicitly granted the process access to those ports (the ioperm and iopl system calls). Touching hardware is exactly the kind of thing that has to go through the kernel, as the system calls chapter explains.

Memory-mapped I/O puts device registers in the ordinary address space. A load or a store to the right address reaches the device instead of RAM. ARM and RISC-V have nothing else, and on x86 nearly every modern device, anything on PCI Express, is memory-mapped too; the port space survives for legacy devices. The advantage is that any instruction and any pointer can reach a device. In C, the pointer must be declared volatile, so that the compiler performs every access exactly as written, with no skipping a read whose value it thinks it knows, no merging two writes:

#define PL061_BASE 0x20060000UL
#define GPIODIR       (*(volatile unsigned int *)(PL061_BASE + 0x400))
#define GPIODATA(m)   (*(volatile unsigned int *)(PL061_BASE + ((m) << 2)))

void led_on(void) {
    GPIODIR |= 1u << 3;          /* line 3 is an output */
    GPIODATA(1u << 3) = 0xff;    /* set line 3; the address masks the other 7 */
}

With clang -O2, this becomes plain loads and stores, the same instructions as for memory. For ARM64:

    mov  w8, #1024
    mov  w10, #255
    movk w8, #8198, lsl #16      ; x8 = 0x20060400, GPIODIR
    ldr  w9, [x8]
    sub  x11, x8, #992           ; x11 = 0x20060020, GPIODATA with mask 0b00001000
    orr  w9, w9, #0x8
    str  w9, [x8]
    str  w10, [x11]
    ret

And for x86-64, where one instruction can read, modify and write memory:

    or   dword ptr [537265152], 8
    mov  dword ptr [537264160], 255
    ret

537,265,152 is 0x20060400, the direction register, and 537,264,160 is 0x20060020. The hardware must cooperate too: device registers must not be cached, and accesses to them must not be reordered or merged by the CPU. Page tables mark such regions with special memory types (uncacheable on x86, Device memory on ARM) that guarantee it.

The PL061 used here is a real GPIO controller designed by ARM, and its data register shows a neat use of address lines. It occupies addresses 0x000 to 0x3FC, and bits 9–2 of the address act as a mask: a write changes only the lines whose mask bit is 1. Writing 0xff to offset 0x20 (mask 0b00001000) sets line 3 and leaves the seven others alone, with no need to read the register first.

Address decoding

Each chip on a bus must answer only to its own addresses. The circuit that decides is the address decoder, and it drives each chip's chip select (CS) input. Take a small computer with a 16-bit address bus (64 KiB) and four chips:

The two highest address bits pick the chip; the 14 lower bits go to every chip and select a byte inside it. The decoder is the 2-to-4 decoder with one more input, MREQ, which says that the CPU is actually doing a memory access right now:

Try all four combinations of A15 and A14: exactly one chip select lights up each time. Turn MREQ off and none does, whatever the address. The address bus may carry garbage between bus cycles, and no chip may respond to it.

Now see why MREQ matters. Set A15 = A14 = 0 (ROM selected), turn auto off, and flip A15 to 1 while MREQ stays on. For one gate delay, CS ROM and CS RAM1 are both on: the new chip select rises before the inverter has turned the old one off. If both chips drove the data bus in that instant, they would fight. That is why bus specifications require the address to be stable before MREQ is asserted (the address-setup time T_ML in the bus timing chapter), and why decoders are enabled by MREQ.

Real systems use exactly this kind of chip. The classic 74HC138 is a 3-to-8 decoder with three enable inputs; its outputs are active low, since chip selects usually are.

Full and partial decoding

The I/O chip has perhaps four registers, selected by A1 and A0. But the decoder gives it all 16 KiB from 0xC000 to 0xFFFF. Bits A13 to A2 are ignored, so the chip answers to every address whose low two bits select a register: its four registers appear 16,384 / 4 = 4096 times, at 0xC000, 0xC004, 0xC008 and so on. This is partial decoding. It's cheap and harmless as long as nothing else needs those addresses; it's a problem when the system has to grow. Full decoding would compare all 14 upper bits (A15–A2), so that the chip appears exactly once.

An even cheaper decoder makes the same point: if the only chip in the lower half of memory is the EPROM, its chip select can simply be wired to A15, and the EPROM answers to 32 KiB of addresses.

Programmable decoders: PCI base address registers

A PC can't fix its address map in wiring, because nobody knows in advance which cards will be installed. So each PCI or PCIe device contains its own address comparators, and the operating system, or the firmware, programs them. They are the base address registers (BARs) in the device's configuration space, at offsets 0x10 to 0x27. Here are the first ones of the virtio network card in the Linux VM of the PCI Express chapter:

$ head -c 64 /sys/bus/pci/devices/0000:00:01.0/config | od -A x -t x1
000000 f4 1a 41 10 06 00 10 00 01 00 00 02 00 40 00 00
000010 04 00 00 80 02 00 00 00 00 01 00 50 00 00 00 00
…
$ head -3 /sys/bus/pci/devices/0000:00:01.0/resource
0x0000000280000000 0x000000028000ffff 0x0000000000140204
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000050000100 0x000000005000013f 0x0000000000040200

Read as little-endian 32-bit words:

• BAR0 = 0x80000004. Bit 0 is 0: a memory BAR, not an I/O-port one. Bits 2–1 are 10: a 64-bit address, whose upper half is in BAR1 = 0x00000002. Together: the device decodes addresses from 0x2_8000_0000, and Linux's resource file shows the range, 64 KiB up to 0x2_8000_FFFF.
• BAR2 = 0x50000100: a 32-bit memory BAR at 0x5000_0100, 64 bytes long.
• The command register at offset 4 is 0x0006: bit 1 enables the device's memory decoders, and bit 2 allows it to become a bus master, for DMA.

The size isn't stored anywhere visible. The firmware finds it by writing all ones to a BAR and reading it back: the device keeps the low bits that its comparator ignores at 0, so the number of zero bits gives the size. Then it writes the chosen base address. The BAR is the address decoder of the previous section, made programmable, and it's how two cards from different makers can be given non-overlapping addresses automatically.

Devices built into a chip don't need BARs: their addresses are fixed by the chip's designers and described to the operating system by the firmware. The Linux VM has two such devices, named after their addresses:

$ ls /sys/bus/amba/devices
20050000.pl031
20060000.pl061
$ cat /sys/bus/amba/devices/20060000.pl061/resource /sys/bus/amba/devices/20060000.pl061/id
	0000000020060000	0000000020060fff	0000000000000200
00041061

A PL031 real-time clock at 0x2005_0000 and a PL061 GPIO controller (the one of the C example above) at 0x2006_0000, occupying 4 KiB. The id 0x00041061 is ARM's code (0x41) and the part number (0x061). The full map of the machine is in /proc/iomem, but inside the container every range reads 00000000-00000000: the kernel hides physical addresses from processes without administrator privileges.

GPIO on microcontrollers

On a microcontroller, general-purpose I/O (GPIO) pins are the main way the program touches the world: buttons, LEDs, relays, the door switch of a microwave oven. The ATmega168 of the CPU chapter controls each port with three registers:

• DDRx (data direction): 1 makes the pin an output;
• PORTx: the value driven on output pins (or, on inputs, whether a pull-up resistor is on);
• PINx: the current level of the pins, read-only.

On an Arduino Uno, the built-in LED is on pin 5 of port B. Turning it on takes two lines of C, with the registers at their data-memory addresses, 0x24 for DDRB and 0x25 for PORTB:

#define DDRB  (*(volatile unsigned char *)0x24)
#define PORTB (*(volatile unsigned char *)0x25)

void led_on(void) { DDRB |= 1 << 5; PORTB |= 1 << 5; }

Compiled with LLVM's AVR back end for the ATmega168, it becomes:

led_on:
    sbi 4, 5      ; set bit 5 of I/O register 4 (DDRB)
    sbi 5, 5      ; set bit 5 of I/O register 5 (PORTB)
    ret

The AVR has both mechanisms at once. Its first 64 I/O registers are memory-mapped at data addresses 0x20 to 0x5F, and are also reachable as I/O addresses 0x00 to 0x3F with the special instructions in and out, and, for the first 32 of them, with sbi and cbi, which set or clear a single bit. The compiler saw the memory addresses 0x24 and 0x25, recognized them as I/O registers 4 and 5, and used sbi, which sets a bit in one instruction. That also avoids a subtle bug: the read-modify-write of PORTB |= … done as three instructions could be interrupted in the middle by an interrupt handler that changes another bit of the same port, whose change would then be lost.

Serial buses between chips

Most chips on a board don't sit on the CPU's memory bus at all. They hang off simple serial buses driven by controller blocks inside the SoC or microcontroller:

SPI is how a PC reads its firmware from the NOR flash chip at power-on; I²C and its close relative SMBus are how it reads the SPD chip on every memory module and talks to temperature sensors and voltage regulators. Chip select, address decoding and handshakes are all still there, just spread out in time over two or four wires.

How a driver talks to a device

Putting it together, here is what an operating system's driver does with a PCI Express device:

1. Find it: at boot, the kernel reads the configuration space of every possible device address. A vendor ID of 0xFFFF means nothing is there; otherwise the vendor and device IDs, and the class code, pick the driver.
2. Give it addresses: the firmware or the kernel sizes and programs the BARs, then sets the memory-enable and bus-master bits in the command register.
3. Map the registers: the BAR's physical addresses are mapped into the kernel's virtual address space as uncacheable device memory (ioremap in Linux).
4. Talk: the driver reads and writes registers with accessors like Linux's readl and writel: volatile accesses with the memory barriers the architecture needs. Bulk data doesn't go through the registers: the driver gives the device the address of a buffer in RAM, and the device transfers it by DMA.
5. Listen: the device signals completion with an interrupt, a memory write through MSI, which runs the driver's handler.

User programs never see any of this. They call read and write on a file descriptor, and the kernel's drivers turn those calls into register accesses. That is the boundary drawn in the system calls chapter, and the subject of the chapter on files, devices and I/O.

Takeaways

• An I/O chip is a few data, status and control registers. UARTs (the 8250/16550 family) and PIOs (the 8255A) are the classics; UARTs are still on every microcontroller and most boards.
• Port-mapped I/O uses a separate address space and special instructions (x86 in/out, which user programs can't execute). Memory-mapped I/O uses ordinary loads and stores to volatile, uncached addresses; ARM, RISC-V and nearly all modern devices use it.
• An address decoder turns the high address bits into chip selects, enabled by MREQ so that nothing answers while the address is changing. Partial decoding makes a chip appear many times; full decoding compares every address bit.
• PCI's base address registers are programmable address decoders: sized by writing ones, set by the firmware or the OS. Built-in devices have fixed addresses, like the PL061 GPIO at 0x2006_0000 in the Linux VM.
• Microcontrollers drive GPIO through direction, output and input registers; the AVR reaches them both as memory and as I/O ports (sbi). Chips on a board talk over UART, SPI and I²C, and a driver ties it all together: configuration space, BARs, mapped registers, DMA and interrupts.