From keypress to pixel

You press H in a text editor and an H appears. The whole trip takes a few tens of milliseconds, and it passes through a microcontroller in the keyboard, a USB host controller, an interrupt, a kernel driver, a system process that owns the screen, your application's main loop, a font renderer, the GPU and a display panel that redraws itself dozens of times per second.

This chapter follows the keystroke all the way, one stage at a time, and ends with the time each stage can take. The same road, with small changes, carries every mouse click and every touch.

The whole path

Here is the same trip as an animation. Each step says which level of the machine is doing the work, and what the keystroke looks like by then: a closed contact, a few bytes, an event, a character, pixels, light.

1–2. Inside the keyboard

A keyboard has no wire per key. The switches sit at the crossings of a matrix of rows and columns, and a small microcontroller scans it continuously, one row at a time. Mechanical contacts bounce for a few milliseconds after a press, so the controller debounces: it believes a new state only once it has been stable for a while. The keyboards and displays chapter takes the matrix and the debouncer apart, with a demo.

On the PC keyboard of the IBM era, every press and every release caused an interrupt, and the handler reads the number of the key, from 1 to 102, from the keyboard controller. That's still the right picture of the information: the computer learns about physical keys going down and up, and all interpretation, including Shift and Ctrl combinations, is done by software. But today's USB and Bluetooth keyboards don't interrupt anybody.

3. USB: the computer asks

A USB keyboard is a HID device (human interface device); Bluetooth keyboards use the same HID reports over a radio link. A USB device never speaks first: the host controller, a piece of hardware in the computer, asks it for news at a fixed interval that the device requests when it's plugged in. The basic keyboard report is 8 bytes: a byte of modifier bits (Ctrl, Shift, Alt and ⌘ or ⊞, left and right), a reserved byte, and up to six keys currently held down, as usage IDs: 0x04 is A, 0x05 is B … 0x0B is H. Pressing Shift+H produces 02 00 0B 00 00 00 00 00. The report says which keys are down, not which key was pressed: the computer finds out about a press by comparing each report with the previous one.

How often does the host ask? The keyboard on this Mac is a Bluetooth one, but a Logitech wireless USB receiver is plugged in too, and it presents itself as a keyboard, among other things. ioreg, the macOS tool that lists hardware, shows the interval it requested:

"Product" = "USB Receiver"
"PrimaryUsage" = 6            ← 6 = keyboard
"ReportInterval" = 1000       ← microseconds: polled every 1 ms

A USB keyboard is sometimes said to be polled every 50 ms. In practice keyboards ask for much shorter intervals (1 ms here, and typically between 1 and 10 ms), and the polling is done by the host controller in hardware, not by the operating system. USB itself has moved on: USB 3.0 ran at 5 Gbit/s, and USB4 now runs at 40 or 80 Gbit/s over the same kind of cable. A keyboard needs none of it: most still use USB's slowest speeds.

4. Interrupt and driver

When a report arrives with something new, the host controller writes it into memory and raises an interrupt. On current machines that's a message-signaled interrupt, a special memory write, as described in the traps and interrupts chapter. The CPU stops whatever it was running, saves its state, and jumps into the kernel.

The kernel's HID driver compares the new report with the previous one and produces events: "usage 0x0B went down, with left Shift held". Those events go into a queue for the rest of the system to pick up, and the interrupted program resumes, none the wiser. If you hold the key, the keyboard just keeps reporting it as down; key repeat is generated in software, which decides how long to wait before the first repeat and how fast to repeat after.

On macOS, the system's HID event service passes it on to the window server. On Linux, the driver publishes the event on a device file under /dev/input/, and the display server reads it from there.

5. The window server routes the event

A key press doesn't know which program it's for. The window server decides: on macOS it's a process called WindowServer; on Linux it's the Wayland compositor (or the X server); on Windows, the window manager inside the system. It knows which window has the keyboard focus and delivers the event there.

Somewhere on the way, a physical key becomes a character. The keyboard's usage 0x0B is always "the key in the H position"; on an AZERTY or a Dvorak layout, that key produces a different letter. The keyboard layout maps keys plus modifiers to characters, and an input method can combine several keystrokes into one character, which is how dead keys produce é and how Chinese and Japanese text is typed.

6. The application's event loop

Every graphical application is built around an event loop. The main thread asks the system for the next event, handles it, and asks again; when there's nothing to do, it sleeps inside a system call, costing nothing:

for (;;) {
    Event e = wait_for_next_event();   /* sleeps until something happens */
    handle(e);                         /* e.g. insert a character */
    if (something_changed) redraw();
}

Each framework dresses it up differently (NSApplication's run loop on macOS, GetMessage and DispatchMessage on Windows, the main loop of GTK or Qt on Linux, the event loop that runs every JavaScript callback in a browser), but it's always this loop. The handler for a key event inserts the character into the document and marks the part of the window that changed as needing a redraw. It doesn't draw immediately: drawing happens once per frame, for all changes together.

This is also why an app "freezes". If a handler takes two seconds (a slow file read on the main thread, a long computation), the loop doesn't come back for the next event, and nothing on screen changes until it does.

7. Text rendering

Turning "H" into pixels is its own small pipeline, working on Unicode characters as described in the characters and text chapter:

1. Font selection: find a font that has a glyph for this character, falling back to other fonts for emoji or scripts the main font lacks.
2. Shaping: turn the sequence of characters into a sequence of positioned glyphs. For Latin text that's mostly advance widths and kerning ("AV" sits tighter than "AH"); for Arabic or Devanagari, a character's shape depends on its neighbors.
3. Rasterization: a glyph is stored as mathematical outlines (curves) and is converted into a small grid of pixels at the current size, with anti-aliasing: edge pixels get partial coverage, so curves look smooth.
4. Caching: rasterized glyphs are kept in a glyph cache, often a texture on the GPU, so the next H at the same size costs a copy.

8. Composition and the framebuffer

The application doesn't own the screen. It draws its window into its own buffer, and the window server's compositor combines all visible windows, with their shadows, transparency and animations, into the final image, usually on the GPU. That final image is the framebuffer: one color value per pixel, typically 4 bytes (red, green, blue and alpha or padding).

This Mac drives two displays of 5120 × 1440 pixels (the keyboards and displays chapter works through the same numbers). One frame for one of them is 5120 × 1440 × 4 = 29,491,200 bytes, about 29.5 MB, and at 60 frames per second the display controller reads 1.77 GB/s from memory just to keep one screen lit.

A classic 1080p example is much smaller and lives elsewhere: a 1920 × 1080 image at 3 bytes per pixel, 6.2 MB, stored in a video RAM on the display controller card. Separate graphics cards still have their own memory, but on Apple silicon, and on most laptops and phones, the framebuffer lives in the same main memory as everything else, shared by the CPU, the GPU and the display controller.

To avoid showing a half-drawn image, the system uses at least two buffers: the display shows one while the next frame is drawn into the other, and they're swapped between frames. That's double buffering, and the swap is timed to the display's refresh: vsync.

9. Scan-out and the display

The display doesn't show a frame all at once. The display controller sends it row by row, from top to bottom, at a fixed refresh rate; when the last row is sent, it starts over. system_profiler SPDisplaysDataType shows this Mac's displays running at 60.00 Hz:

LG HDR DQHD:
  Resolution: 5120 x 1440
  UI Looks like: 5120 x 1440 @ 60.00Hz

Screens were long repainted 60 times a second, with twisted-nematic LCDs and OLED as a promising newcomer. The 60 Hz baseline is still common, but phones, laptops and gaming monitors now run at 120 Hz or more, often with a variable refresh rate that drops when nothing moves to save power. And OLED panels are now standard in phones and common in laptops and monitors.

The latency budget

Here are the stages that have fixed, knowable costs on this machine:

Everything else varies too much to put a single number on it: the keyboard's scanning and debouncing, a wireless link's radio protocol, how busy the app's event loop is, how many frames the compositor buffers, and the panel's own response time. Measured end to end, from the key hitting bottom to light changing on the screen, typical computers take a few tens of milliseconds, and much more when something in the chain is slow. The display clock dominates the fixed part: at 60 Hz, simply waiting for the next frame and scanning it out can cost up to about 33 ms, which is why a faster refresh rate makes typing and scrolling feel better even when nothing else changes.

Takeaways

• A keystroke goes key matrix → keyboard microcontroller → USB → interrupt → HID driver → window server → event loop → text renderer → compositor → framebuffer → display.
• USB keyboards are polled by the host controller: an 8-byte HID report lists the modifier keys and up to six keys held down; this Mac's receiver asks to be polled every 1 ms, not every 50 ms.
• Keys become characters late: the keyboard layout and input method are software, in the window server's domain.
• Every GUI app runs an event loop; a slow handler freezes the app because the loop can't get to the next event.
• Text rendering is font selection, shaping, rasterization and caching; the result is composited into a framebuffer (29.5 MB per frame at 5120 × 1440) in main memory on Apple silicon, not in a separate video RAM.
• The display refreshes at a fixed rate (16.7 ms per frame at 60 Hz, 8.3 ms at 120 Hz), which sets a floor on how fast any keystroke can appear.