Main memory forgets everything when the power goes off, and it's never big enough. Below it in the memory hierarchy — the pyramid the caches chapter climbed from the top — sits secondary storage: slower, much larger, cheaper per byte, and non-volatile. For fifty years that meant spinning magnetic disks. Today it mostly means solid-state drives made of flash memory, with hard disks kept for bulk capacity. This chapter looks at how both work, how fast they really are, and how RAID combines several of them so that losing one loses nothing.
The hard disk
A hard disk drive (HDD) stores bits as tiny magnetized regions on the surface of rotating platters. A head floating a few nanometers above each surface magnetizes the regions to write and senses them to read (storing a bit, at the physics level, covers the magnetism). All the heads sit on one arm assembly that moves them together across the surfaces.
The geometry follows from that:
- A track is the ring of bits passing under a head while the arm stays still.
- Each track is divided into sectors, the smallest unit the drive reads or writes. Each sector carries a preamble for synchronization, the data, and an error-correcting code — a Reed–Solomon or more powerful code, so that a scratch or weak spot can be corrected (the idea is explained in the foundations chapter on error correction).
- The set of tracks under all the heads at one arm position is a cylinder.
- Outer tracks are longer than inner ones, so drives use zones: tracks further out hold more sectors. The disk turns at a constant speed, so the outer tracks deliver data faster.
Tanenbaum describes 512-byte sectors. That was the standard until around 2010; drives since then use Advanced Format 4,096-byte physical sectors, which need less space for gaps and codes per byte stored. Many still pretend to have 512-byte sectors for compatibility, doing a read–modify–write internally when the host writes less than 4 KiB. The largest drives today use heat-assisted magnetic recording (HAMR), in which a laser briefly heats each spot as it's written: in 2025 Seagate announced drives of up to 36 TB, built from ten platters of 3.6 TB each. Many large drives are sealed and filled with helium instead of air, which lets thinner platters spin with less turbulence.
How long a read takes
Reading a sector takes three steps:
- Seek: move the arm to the right cylinder. Averaged over random positions, this takes several milliseconds.
- Rotational latency: wait for the sector to come round under the head. On average, that's half a revolution.
- Transfer: read the bits as they pass.
Let's work it out for a 7,200 RPM drive, assuming an 8 ms average seek and a sustained rate of 200 MB/s (both typical, but assumptions, not measurements):
| Step | Computation | Time |
|---|---|---|
| one revolution | 60 s / 7,200 | 8.33 ms |
| rotational latency | half a revolution | 4.17 ms |
| seek | assumed | 8 ms |
| transfer 4 KiB | 4,096 B / 200 MB/s | 0.02 ms |
| total | 12.2 ms |
A random 4 KiB read takes about 12 ms, so the drive manages about 82 random reads per second — 0.34 MB/s. Read the same data sequentially and it streams at 200 MB/s, 600 times faster. Almost all the time goes into mechanics: the transfer is 0.2% of the total. That's why file systems and databases work so hard to keep related data together on disk, and why a hard disk as the system drive made computers feel slow.
Tanenbaum lists rotation speeds of 5,400, 7,200 and 10,800 RPM. The fast enterprise drives actually spun at 10,000 and 15,000 RPM, and those have essentially disappeared: an SSD beats them on every measure except capacity per dollar. The drives still sold turn at 5,400–7,200 RPM and compete on capacity.
From IDE and SCSI to SATA and NVMe
The book spends several pages on IDE, EIDE, ATA and SCSI, the parallel-cable interfaces of the 1980s–2000s. They're history now, but a few ideas from them live on:
- Logical block addressing (LBA): instead of cylinder, head and sector numbers, the drive exposes a flat array of blocks numbered from 0, and does the geometry itself. With 48-bit LBAs and 512-byte blocks, the limit is 128 PiB, far beyond today's drives.
- SATA (serial ATA), which the book announces as the future, did take over. Its third generation runs at 6 Gbit/s, which after encoding overhead carries at most about 600 MB/s.
- The SCSI command set outlived the SCSI bus: SAS (serial attached SCSI) is the server-drive interface, and USB storage devices carry SCSI commands too.
The fast drives of today skip all of these and use NVMe (non-volatile memory express), a protocol designed for flash, over PCI Express. SATA's command interface was designed for a disk that can do one thing at a time; NVMe gives the drive up to 65,535 queues of up to 65,536 commands each, so that every core can submit requests to its own queue without locking, and the drive can work on hundreds of them in parallel. That parallelism, as we'll measure below, is exactly what flash needs.
Inside an SSD
A solid-state drive stores bits as electric charge trapped in NAND flash cells: transistors with an extra, insulated storage layer. Charge in that layer shifts the voltage at which the transistor turns on, and the drive reads the bit by testing that threshold. How the cells work as chips is covered in the chapter on SRAM, DRAM, ROM and flash chips, at the digital logic level; here we look at the drive built from them.
Flash has three awkward properties:
- Reads and writes work on pages — typically 4 to 16 KiB — not bytes.
- A page can't be overwritten. Before it's written again, it must be erased, and erasing works only on a whole block of hundreds of pages, several megabytes.
- Each block wears out after a limited number of program/erase cycles.
An SSD hides all three behind a flash translation layer (FTL), firmware running on the drive's own processor. The host sees an ordinary array of logical blocks. The FTL keeps a map from each logical block to a physical page, and:
- Writes go to fresh pages. Rewriting logical block 42 writes the new data to an already-erased page elsewhere, updates the map, and marks the old page as stale. No erase is needed on the write path.
- Garbage collection runs in the background: it picks a block with many stale pages, copies its remaining valid pages elsewhere, and erases it. The drive keeps spare capacity, over-provisioning, so that there's always somewhere to write.
- Wear leveling spreads the erases evenly over all the blocks, including ones holding data that never changes, which it moves from time to time.
The file system helps with the TRIM command (called deallocate in NVMe): when a file is deleted, the operating system tells the drive which logical blocks no longer hold anything useful, so garbage collection doesn't waste effort copying dead data. On this Mac, system_profiler SPNVMeDataType shows the internal drive and TRIM Support: Yes.
A cell can store more than one bit by distinguishing more charge levels: SLC stores 1 bit per cell (2 levels), MLC 2 bits (4 levels), TLC 3 bits (8 levels), and QLC 4 bits (16 levels). More bits per cell is cheaper per gigabyte, but the levels are closer together, so reads are slower and the cells tolerate fewer erase cycles. Since the mid-2010s, flash is also built in three dimensions: the cells are stacked in more than a hundred layers, sometimes several hundred, on one chip.
What the book says about SSDs, and what changed
Tanenbaum's SSD section is from the early 2010s, and several points need correcting:
- He says a cell is programmed by hot-carrier injection. That's how NOR flash is programmed; the NAND flash in SSDs is programmed and erased by Fowler–Nordheim tunneling, in which a strong field pulls electrons through the thin insulator. And most 3D NAND doesn't use a conducting floating gate at all, but a charge-trap layer of insulator that holds the electrons in place.
- He gives an endurance of about 100,000 writes per cell. That's a figure for SLC flash. Most SSDs today use TLC or QLC cells, rated for roughly a thousand to a few thousand program/erase cycles; wear leveling and over-provisioning are what make that enough.
- "Multilevel flash cells" encode multiple bits per cell, not "per byte" as the book says, and they now go up to four bits, not two.
- He says an SSD runs "two to three times faster" than a 100 MB/s disk, and costs "one to three dollars per gigabyte". The price has fallen by an order of magnitude or more since, and the speed has risen by far more than that, as the next section shows.
Measured: this Mac's SSD
The M2 Ultra this chapter was written on has a 1 TB internal SSD (APPLE SSD AP1024Z); its flash controller is built into the M2 Ultra chip, and macOS reports it as NVMe. diskutil info / shows Solid State: Yes and a Device Block Size of 4,096 bytes.
A small C program measured it. It writes a 2 GiB file with F_NOCACHE (so macOS doesn't keep the data in its page cache) followed by F_FULLFSYNC (so the data really reaches the flash), reads it back sequentially in 8 MiB chunks, then does 20,000 random 4 KiB reads, one at a time. Six runs:
| Test | Result |
|---|---|
| sequential write | 956–1,412 MB/s in five runs, 240 MB/s in one |
| sequential read | 5.0–5.4 GB/s |
| random 4 KiB read, one at a time | 9,700–11,300 per second; median latency 80–85 µs |
The drive was 98% full during the test, which leaves the FTL little spare room for garbage collection, and probably explains the unsteady write speed — the slow run was the very first. Reads were stable.
The random reads are the interesting number. 82 µs per random read, against about 12 ms for the hard disk above: about 150 times faster, with no moving parts. But 10,000 reads per second at 4 KiB is only about 40 MB/s, far below the sequential 5 GB/s. The drive is waiting too: one read at a time can't use its many flash chips in parallel. A second program issued the same reads from several threads at once:
| Threads issuing reads | Random 4 KiB reads per second |
|---|---|
| 1 | about 12,000 (4,100 in one run) |
| 4 | about 49,000 |
| 16 | 126,000–141,000 |
| 32 | 141,000–156,000 |
With 16 requests in flight, the drive does over ten times as many reads per second. The latency of each read barely changes; the drive works on many at once. That's what NVMe's deep queues are for, and why storage software keeps many requests in flight. The test files were deleted after each run.
RAID
In 1988, Patterson, Gibson and Katz proposed replacing one large, expensive disk with a redundant array of inexpensive disks — RAID — which looks like one disk to the operating system but spreads the data across several. Industry later changed "inexpensive" to "independent". The different arrangements are called RAID levels, although, as Tanenbaum notes, they're alternatives, not a hierarchy.
| Level | Arrangement | Usable capacity (n disks) | Survives |
|---|---|---|---|
| 0 | striping: consecutive strips go to consecutive disks | n | no failure at all |
| 1 | mirroring: every strip written to two disks | n/2 | one disk per pair |
| 5 | striping plus one parity strip per stripe, rotated across disks | n − 1 | any one disk |
| 6 | striping plus two independent parity strips | n − 2 | any two disks |
| 10 | a stripe (RAID 0) across mirrored pairs (RAID 1) | n/2 | one disk per pair |
RAID 0 is fast for large transfers, because all disks work in parallel, but any failure loses everything: with four disks, a failure is four times as likely as with one. RAID 1 reads from either copy and survives a failure trivially, at the cost of half the capacity. Tanenbaum also describes levels 2, 3 and 4. Levels 2 and 3 spread every word bit by bit across disks that spin in lockstep; nobody builds them any more. Level 4 is level 5 with all the parity on one disk, which becomes a bottleneck for writes; NetApp's storage systems are the best-known users of a variant of it.
Parity with XOR
RAID 5 protects data with the exclusive OR. The parity strip is the XOR of the data strips in the same stripe:
P = D0 ⊕ D1 ⊕ D2
Because x ⊕ x = 0, any one strip is the XOR of all the others, including the parity. If disk 1 dies, D1 = D0 ⊕ D2 ⊕ P. For three strips holding the bytes RAID, five and XOR!, the first byte of each gives:
D0 'R' 01010010
D1 'f' 01100110
D2 'X' 01011000
P 01101100 (0x6c) = D0 ^ D1 ^ D2
and XORing D0, D2 and P gives back 01100110, the lost f. The same thing, running on the emulator — three data disks of four bytes, a parity disk, then disk 1 wiped and rebuilt:
Try it: Press Step to run one instruction, Run to animate or Continue to finish; the L2–L7 buttons zoom in and out one level at a time.
- char disk[4][4]; /* 3 data disks + 1 parity disk, 4 bytes each */
- int main() {
- strcpy(disk[0], "RAI");
- strcpy(disk[1], "D-5");
- strcpy(disk[2], "ok!");
- for (int i = 0; i < 4; i++) /* parity = D0 ^ D1 ^ D2 */
- disk[3][i] = disk[0][i] ^ disk[1][i] ^ disk[2][i];
- memset(disk[1], 0, 4); /* disk 1 dies */
- for (int i = 0; i < 4; i++) /* rebuild it from the survivors */
- disk[1][i] = disk[0][i] ^ disk[2][i] ^ disk[3][i];
- printf("rebuilt: %s\n", disk[1]);
- return disk[3][0] & 0xff;
- }
It prints rebuilt: D-5 and exits with 121, the first parity byte: 'R' ⊕ 'D' ⊕ 'o' = 0x52 ⊕ 0x44 ⊕ 0x6F = 0x79. Parity is one XOR per byte, done by the ALU or by vector instructions many bytes at a time, so computing it is cheap.
A single parity strip can only fix a known missing strip — an erasure — not find an unknown error. That's fine for RAID: when a disk dies, the controller knows which one.
The small-write penalty
Updating one strip in RAID 5 doesn't require reading the whole stripe. Since P is an XOR, the new parity is:
P_new = P_old ⊕ D_old ⊕ D_new
In the example, replacing five by six! changes the parity from 6c 67 6d 00 to 79 67 63 44, whether it's recomputed from all strips or from the old parity, old data and new data. But it still costs four disk operations for one logical write: read the old data, read the old parity, write the new data, write the new parity. That's the small-write penalty Tanenbaum describes, and it's why databases with many small writes prefer RAID 10.
RAID 6 and rebuild times
RAID 6 adds a second parity strip, computed not with XOR but with a Reed–Solomon code over the bytes, so that any two lost strips can be recomputed. It has become the norm for arrays of large hard disks, for a reason the book couldn't stress in 2013: rebuilding a failed disk means reading every sector of every surviving disk. At 200 MB/s, merely reading one 36 TB drive takes 50 hours, and with others working the array during the rebuild it takes longer. A second failure, or an unreadable sector, during that window would lose data with single parity.
RAID is not a backup. It protects against a disk failing, not against deleting a file, a bug that corrupts data, or ransomware: all of those are faithfully written to every disk. The next chapter's tapes are one answer to that.
Takeaways
- A hard disk reads a sector by seeking, waiting half a rotation on average, then transferring. For a 7,200 RPM drive with an 8 ms seek, a random 4 KiB read takes about 12 ms: 82 per second, versus 200 MB/s sequentially.
- Sectors are 4 KiB today (the book's 512 bytes is pre-2010); IDE, EIDE and parallel SCSI are history; SATA tops out around 600 MB/s; fast drives use NVMe over PCIe, with thousands of queues.
- An SSD's flash translation layer remaps every write to a fresh page, garbage-collects erase blocks, wear-levels, and uses TRIM. NAND is programmed by tunneling (not hot-carrier injection as the book says), and TLC/QLC cells endure thousands of cycles, not 100,000.
- Measured on this Mac's SSD: 5.0–5.4 GB/s sequential reads, about 1–1.4 GB/s writes, and 80–85 µs random reads, 150 times faster than the disk; parallel requests raise random reads from about 12,000 to over 140,000 per second.
- RAID 0 stripes, 1 mirrors, 5 adds XOR parity (any one strip = XOR of the others), 6 adds a second, Reed–Solomon parity, 10 stripes over mirrors. RAID 5's small writes cost four I/Os. RAID isn't a backup.