Signals on wires and fibers

Storing a bit is half the job; the other half is moving it. Inside a chip, bits move over the RC-limited wires of the first chapter. Between chips, across a board, down a cable or under an ocean, they travel as electromagnetic waves guided by copper or glass. This chapter covers the physics that decides how fast and how far: the speed of the wave, what happens at the end of a wire, why fast links use pairs of wires, how fiber keeps light inside, and the hard limit on how many bits per second any channel can carry.

How these links are organized into buses and protocols belongs to the levels above: Real buses: PCI, PCI Express and USB on the digital-logic level and From modems to Ethernet on the device level.

How fast a signal travels

A voltage change doesn't travel down a wire by electrons racing along it — they drift at fractions of a millimetre per second, as the first chapter computed. It travels as an electromagnetic wave in the space around and between the conductors, and its speed depends on the insulator that fills that space:

v = c / √εr

where c = 299,792,458 m/s is the speed of light in a vacuum (exact, by definition of the metre) and εr is the insulator's relative permittivity. In fiber, the same role is played by the glass's refractive index n: v = c/n.

c = 299792458
print(c * 1e-9 * 100, "cm per ns")                 # 29.98
for name, er in [("coax, polyethylene", 2.25), ("board microstrip (eff.)", 3.0),
                 ("board stripline, FR-4", 4.0)]:
    v = c / er**0.5
    print(name, v / c, v * 1e-9 * 100, "cm/ns", 1e-3 / v * 1e12, "ps/mm")
print("fiber", 1 / 1.468, c / 1.468 * 1e-9 * 100)  # n ≈ 1.468

Some consequences:

• In one nanosecond, light in a vacuum covers 30 cm — about a foot. During one cycle of a 5 GHz clock (200 ps) it covers 6 cm.
• A signal crossing a 30 cm motherboard trace on an inner layer takes about 2 ns: 8 cycles of a 4 GHz processor, before any logic has run.
• A request to a DRAM chip 10 cm away and back travels 20 cm, at least 1.3 ns at half the speed of light. Memory latency has a floor that no circuit trick removes.

Tanenbaum gives 20 cm/ns for signals "in copper wire or optical fiber". That's right for fiber and typical cables. On a circuit board, where the insulator is glass-epoxy laminate (FR-4), signals travel at 15–17 cm/ns. And on the chip itself, long wires are far slower than any of these because of their resistance.

Wires as transmission lines

At low speed, a wire is just a connection: the voltage is the same all along it. That stops being true once a signal changes faster than it can travel the wire's length. A rising edge that takes 100 ps to go from 0 to 1 is spread over 1.5 cm of board trace at half the speed of light:

print(100e-12 * 0.5 * c * 100, "cm")   # 1.5 cm

Any trace longer than a fraction of that — so essentially every high-speed trace on a board — holds different voltages at different points at the same instant. It behaves as a transmission line: a wave moving along a pair of conductors (the signal trace and the ground plane under it).

A transmission line has a characteristic impedance Z₀ = √(L/C), where L and C are its inductance and capacitance per unit length. Z₀ is set by the geometry — trace width, distance to the ground plane, insulator — and not by the length. It's the ratio of voltage to current in the travelling wave: a 50 Ω line driven with a 1 V step draws 20 mA while the edge travels down it, whatever is at the far end, until the edge gets there.

Reflections

When the wave reaches the end of the line, it meets a load impedance ZL. If ZL equals Z₀, the load absorbs the wave exactly as the line would have carried it, and nothing comes back. Otherwise part of the wave is reflected back toward the source, with a reflection coefficient:

Γ = (Z_L − Z₀) / (Z_L + Z₀)

G = lambda zl, z0=50: (zl - z0) / (zl + z0)
for zl in (50, 75, 100, 25, 0, float("inf")):
    print(zl, G(zl) if zl != float("inf") else 1.0)

A CMOS input is close to an open circuit, so an unterminated line reflects nearly the whole edge. The reflection travels back, reflects again at the driver, and the receiver sees the signal ring — overshoot, undershoot, false edges — for several round trips. The fix is termination: a resistor equal to Z₀ at the end of the line (or in series at the source) so that the wave is absorbed. Tanenbaum's description of SCSI, where the last device on the cable had to be terminated "to prevent reflections", is this effect on a 1980s cable. Today, DDR memory chips switch termination resistors on and off inside the chip (on-die termination), and high-speed serial receivers have them built in.

Bits in flight

At high enough speed, a wire holds many bits at once, like a pipeline. A PCIe 5.0 lane runs at 32 gigatransfers per second, so each bit lasts 31.25 ps:

ui = 1 / 32e9
t = 0.30 / (0.5 * c)             # 30 cm of board trace at c/2
print(ui * 1e12, t / ui)         # 31.25 ps per bit, 64 bits in flight
print(ui * 0.5 * c * 1000, "mm") # each bit occupies 4.7 mm of trace

On a 30 cm trace, 64 bits are on the wire at once, each a 4.7 mm stretch of travelling voltage.

At these frequencies, the wire also loses more and more of the signal. Current at high frequency flows only in a thin outer layer of the conductor, the skin depth, which shrinks as 1/√f:

import math
rho, mu0 = 1.68e-8, 4e-7 * math.pi
for f in (60, 1e6, 1e9, 10e9):
    print(f, math.sqrt(rho / (math.pi * f * mu0)) * 1e6, "µm")
# 60 Hz: 8,422 µm   1 MHz: 65 µm   1 GHz: 2.1 µm   10 GHz: 0.65 µm

At 60 Hz the skin depth is 8.4 mm, more than the radius of any ordinary wire, so current uses the whole conductor; at 10 GHz it uses only the outer 0.65 µm, and the effective resistance is much higher. The board's insulator also absorbs more energy at higher frequencies. The high-frequency parts of a fast signal are attenuated much more than the low ones, which smears each bit into its neighbours. Fast receivers undo this with equalizers that boost the high frequencies back, and long board paths need retimers that receive and regenerate the signal part way.

Clock skew

A synchronous circuit assumes that the clock edge reaches every flip-flop at the same moment. Any difference in arrival time is clock skew, and it eats directly into the cycle: data launched by an early clock and captured by a late one gets less time than the design assumed.

Inside a chip, the clock is distributed by a carefully balanced tree of wires and buffers (an H-tree is the textbook shape: every path from the root to a leaf has the same length), and designers budget the skew to a small fraction of the cycle — a hard target when an unbuffered millimetre of thin wire already takes a whole cycle. Gates have delay too; a chain of them is the classic way to measure it on a chip, with a ring oscillator, an odd number of inverting gates in a loop:

With EN on, the output toggles every three gate delays: the period is six delays, twice the number of gates in the loop. On silicon, a ring oscillator's frequency counter gives the gate delay directly, which is how chips measure the speed of their own transistors.

Between chips, skew is what killed wide parallel buses. Tanenbaum makes the point for PCI versus PCI Express: the 32 or 64 wires of a parallel bus never have exactly the same delay, and the faster the bus, the larger a fraction of the cycle that difference becomes. The answer was to go serial and send the clock with the data: a serial receiver recovers the timing from the transitions in the bit stream itself (clock and data recovery), so there's no separate clock wire to be skewed against. A multi-lane link still has skew between its lanes, but each lane is timed independently and the receiver realigns them in buffers. PATA became SATA, PCI became PCIe, parallel printer ports became USB.

Differential signaling

Nearly every fast serial link — PCIe, USB, SATA, Ethernet, HDMI, DisplayPort — carries each signal on a pair of wires, driven with opposite voltages. The receiver looks only at the difference between the two. This buys several things at once:

• Noise rejection. Interference from a neighbouring wire or a power supply hits both wires of a tightly coupled pair almost equally. The difference cancels it (common-mode rejection). Twisting the pair, as in Ethernet cable, keeps the two wires equally exposed.
• Small swings. Since noise mostly cancels, the signal can be small: the LVDS standard uses a difference of about 350 mV. Smaller swings mean faster edges and, since energy goes as CV², less power.
• Less emission. The two opposite currents create fields that mostly cancel a short distance away, so the pair radiates less interference of its own.
• Its own reference. The receiver doesn't compare the signal with a ground that may be at a slightly different voltage on the other chip or the other end of a cable.

A pair is a transmission line too, specified by its differential impedance: 85 Ω for PCIe, 90 Ω for USB, 100 Ω for Ethernet twisted pair. The cost is two wires per signal — easily repaid, since each pair runs many times faster than a single-ended wire could.

Fiber optics

Over long distances, copper loses too much. Optical fiber carries light instead, in a thread of extremely pure glass.

The light is kept inside by total internal reflection. A fiber has a core with a slightly higher refractive index than the cladding around it. At a boundary, Snell's law (n₁ sin θ₁ = n₂ sin θ₂) bends light away from the perpendicular when going into the lower index. Beyond the critical angle θc = arcsin(n₂/n₁), there's no transmitted ray at all: the light is entirely reflected. With a core index difference of about 0.36 %, typical of single-mode fiber:

n1 = 1.468; n2 = n1 * (1 - 0.0036)
print(math.degrees(math.asin(n2 / n1)))            # 85.1° from the perpendicular
NA = math.sqrt(n1**2 - n2**2)
print(NA, math.degrees(math.asin(NA)))             # 0.12, acceptance half-angle 7.1°

Light travelling within about 5° of the fiber's axis inside the glass is reflected at every contact with the cladding, without loss. The fiber accepts light entering within about 7° of its axis (its numerical aperture, about 0.12 here).

Two kinds of fiber are common:

• Multimode fiber has a wide core, 50 or 62.5 µm. Light takes many paths (modes) of slightly different lengths, which smears pulses over distance; it's used with cheap 850 nm sources over tens to hundreds of metres, inside buildings and data centres.
• Single-mode fiber has a core of about 9 µm, small enough that only one mode propagates. Used at 1310 and 1550 nm, it carries data over kilometres to thousands of kilometres.

Decibels and attenuation

Signal loss is measured in decibels (dB), a logarithmic ratio of powers: dB = 10 log₁₀(Pin/Pout). Losses in dB add up along a path, instead of multiplying:

Modern single-mode fiber loses about 0.2 dB per kilometre at 1550 nm, where glass is most transparent:

for km in (10, 80, 100):
    print(km, km * 0.2, "dB", 10 ** (-km * 0.2 / 10))
# 10 km: 2 dB (63 %)   80 km: 16 dB (2.5 %)   100 km: 20 dB (1 %)

After 100 km, 1 % of the light is left: still detectable. Long-distance links place optical amplifiers (erbium-doped fiber amplifiers) every 50 to 100 km or so, which boost all the wavelengths at once without converting back to electricity. And a single fiber carries dozens of independent channels on different wavelengths (wavelength-division multiplexing).

The speed of light in glass also sets a floor on network latency: about 4.9 ms per 1,000 km, one way. A round trip across the Atlantic can't take less than a few tens of milliseconds, whatever the equipment.

Shannon: the capacity of a channel

How many bits per second can a channel carry? In 1948 Claude Shannon proved that a channel with bandwidth B hertz and signal-to-noise ratio SNR (a power ratio, not in dB) has a maximum error-free rate — its capacity:

C = B × log₂(1 + SNR)

No coding scheme can beat it; good modern codes come close. Doubling the bandwidth doubles the capacity; doubling the signal power adds only about one bit per second per hertz.

C = lambda B, snr_db: B * math.log2(1 + 10 ** (snr_db / 10))
print(C(3100, 35))      # telephone line: 36,044 bit/s
print(C(20e6, 25))      # 20 MHz radio channel at 25 dB: 166 Mbit/s
print(C(1e6, 0))        # SNR = 1 (0 dB): C = B = 1 Mbit/s
print(10 * math.log10(2 ** (33600 / 3100) - 1))   # 32.6 dB needed for 33.6 kbit/s

A telephone voice channel passes roughly 300 to 3,400 Hz, about 3.1 kHz of bandwidth. With an SNR of 35 dB, Shannon allows about 36 kbit/s. The last generation of analog modems ran at 33.6 kbit/s, which needs at least 32.6 dB: they were within a few percent of the limit. The "56k" modems that followed didn't beat Shannon; they avoided one analog conversion by having the provider's end connected digitally to the phone network. That story belongs to From modems to Ethernet, on the device level.

The same formula governs every link. A 20 MHz Wi-Fi channel with 25 dB of SNR can carry at most 166 Mbit/s per spatial stream; wider channels and several antennas (several streams) are how Wi-Fi gets faster. Wired links hit it too: PCIe 6.0 doubled PCIe 5.0's rate without doubling the frequency by sending PAM4 — four voltage levels, two bits per symbol — instead of two levels. With four levels in the same voltage range, the gap between adjacent levels is a third of the full swing, which costs 20 log₁₀ 3 ≈ 9.5 dB of noise margin; the link needs strong error correction to make up for it.

Takeaways

• Signals travel as electromagnetic waves at c/√εr: 30 cm/ns in vacuum, about 20 cm/ns in cable and fiber, 15–17 cm/ns on a circuit board. On-chip wires are slower still because of RC delay.
• A 30 cm board trace takes about 2 ns; a round trip to memory 10 cm away takes at least 1.3 ns.
• Fast traces are transmission lines with a characteristic impedance Z₀. A mismatched end reflects Γ = (ZL − Z₀)/(ZL + Z₀) of the wave; termination absorbs it.
• At 32 GT/s, a 30 cm trace holds 64 bits in flight, and losses rise with frequency (skin depth 2.1 µm at 1 GHz), so receivers equalize.
• Skew limits parallel buses; serial links embed the clock in the data. Differential pairs reject common-mode noise, allow small swings and radiate less.
• Fiber guides light by total internal reflection; single-mode fiber loses about 0.2 dB/km at 1550 nm (1 % left after 100 km).
• Shannon: C = B log₂(1 + SNR). A 3.1 kHz phone line at 35 dB allows 36 kbit/s — which 33.6 kbit/s modems nearly reached.