Flags, cmp and conditional jumps

A CPU has no if. What it has is a two-step routine that every compiled decision is built from:

1. an instruction records facts about a result — was it zero? negative? did it overflow? — in a register called RFLAGS;
2. a conditional jump reads those facts and either jumps somewhere else or falls through to the next instruction.

Once you can read those two steps, every if, while, for and switch in a disassembly becomes readable.

The flags register

RFLAGS is a 64-bit register, but the bits that matter for decisions are six status flags. Arithmetic and logic instructions (add, sub, cmp, and, test, inc…) write them as a side effect. mov does not touch them.

Textbooks call these bits condition codes, kept in a flags register often named the PSW (program status word). RFLAGS is x86's PSW, and ZF, SF, OF and CF are its versions of the classic Z, N, V and C bits.

ZF, SF, CF and OF are the ones conditional jumps use. PF and AF exist for historical reasons; you will rarely see them tested.

cmp is a subtraction you throw away

cmp a, b computes a − b, sets the flags from the result, and discards the result. Nothing but RFLAGS changes. That is the whole trick: comparing two numbers means subtracting them and looking at what the subtraction would have produced.

The demo below has already executed mov eax, 7 and cmp eax, 5. Look at the flags panel: 7 − 5 = 2, which is not zero, not negative, and didn't borrow — so all six status flags are 0. Nothing is lit, but every one of them is outlined: the cmp wrote them all, it just wrote zeros. Press Step to run the next two comparisons.

• cmp eax, 7: 7 − 7 = 0, so ZF = 1 — the two values are equal.
• cmp eax, 9: 7 − 9 = −2, so SF = 1 (negative) and CF = 1 (subtracting a bigger unsigned number borrowed).

eax stays 7 the whole time: cmp only writes flags.

Conditional jumps read the flags

A conditional jump is written j + a condition code, and each condition is a small formula over the flags:

The same conditions also exist as setcc (write 0 or 1 into a byte register) and cmovcc (move only if the condition holds).

Same bits, two different questions

Why are there two families, jl/jg and jb/ja? Because the CPU doesn't know whether your 32 bits are a signed or an unsigned number — the instruction you choose decides.

Take 0xffffffff. As a signed int it is −1. As an unsigned int it is 4 294 967 295. Compare it with 1 and you get two different, both correct, answers:

After the cmp, SF = 1 and OF = 0, and CF = 0. Step twice:

• setl bl tests SF ≠ OF → 1 ≠ 0 → bl = 1: as signed numbers, −1 is less than 1.
• setb cl tests CF = 1 → CF is 0 → cl = 0: as unsigned numbers, 4 294 967 295 is not below 1.

This is exactly what a compiler does with your C types: comparing two ints produces jl/jg, comparing two unsigneds produces jb/ja. When you read a disassembly, the jump tells you the signedness of the variables.

Why OF exists

If "less than" just meant "the subtraction came out negative", jl could test SF alone. It can't, because a signed subtraction can overflow and flip the sign bit.

The smallest int is −2 147 483 648 (0x80000000). Subtract 1 and the true answer, −2 147 483 649, doesn't fit in 32 bits: the result wraps around to 0x7fffffff, a positive number.

After the cmp, SF = 0 (the wrapped result looks positive) but OF = 1 (the signed result overflowed). jl and setl test SF ≠ OF, which is 0 ≠ 1 → true: INT_MIN is correctly less than 1. sets cl, which looks at the sign alone, gets it wrong and writes 0. OF is the correction that makes signed comparisons right even when the subtraction overflows.

How a C if becomes cmp + a jump

Here is if (x > 10) return 1; else return 0; written the way a compiler lays it out, with x = 12:

Notice the jump is jle, the opposite of the C condition >. The compiler lays the "then" block right after the comparison and jumps over it when the condition is false. The flags panel shows it before you step: next: jle tests ZF = 0, SF = 0, OF = 0 → falls through. Change 12 to 7 (Edit → Load) and the same jle is taken.

This inversion is the first thing to internalize when reading compiled code: the condition you see in the jump is usually the negation of the one in the source.

test: comparing against zero and checking bits

test a, b is to and what cmp is to sub: it computes a & b, sets ZF, SF and PF from the result, clears CF and OF, and throws the result away.

test eax, eax     ; eax & eax = eax, so ZF = 1 exactly when eax == 0
jz   is_zero      ; if (eax == 0) goto is_zero

test reg, reg followed by jz/jnz is how compilers write if (x == 0) and if (ptr) when optimizing. test eax, 4 checks a single bit — the idiom behind if (flags & 4).

Cheat sheet

In an unoptimized build, remember to invert: the jump you read skips the block when the condition is false.

To practice, open the simulator and pick the Signed vs unsigned compare example: it is C code whose two ifs compile to jge and jae, and you can step through both.