A function call is a contract. The caller has to put the arguments somewhere the callee will look, the callee has to leave the result somewhere the caller will read it, and both have to agree on which registers may be overwritten. That contract is the calling convention. The CPU doesn't enforce it — compilers do, so that code built by different compilers can call each other.
Two conventions cover most of what you'll read on Linux and macOS: System V AMD64 for 64-bit code and cdecl for 32-bit code.
call and ret
Everything starts with two instructions:
call fpushes the return address — the address of the instruction right after thecall— onto the stack, then jumps tof.retpops that address back intorip, so execution resumes right after thecall.
The stack is how a function knows where to go back to. That's also why corrupting it is so dangerous: overwrite the return address and ret jumps wherever the new value points.
System V x86-64: arguments in registers
In 64-bit Linux and macOS, the first six integer or pointer arguments go in registers, in this order:
| Argument | 1st | 2nd | 3rd | 4th | 5th | 6th |
|---|---|---|---|---|---|---|
| Register | rdi | rsi | rdx | rcx | r8 | r9 |
Further arguments are pushed on the stack. Floating-point arguments use xmm0–xmm7. The return value comes back in rax (xmm0 for floating point).
Here is add2(3, 4) in its simplest form. The demo stops right after the call:
- _start:
- mov edi, 3 ; 1st argument
- mov esi, 4 ; 2nd argument
- call add2 ; push the return address, jump
- ret
- add2:
- push rbp ; prologue: save the caller's frame pointer
- mov rbp, rsp ; and start our own frame
- mov eax, edi
- add eax, esi ; the return value goes in eax
- pop rbp ; epilogue: restore the caller's frame pointer
- ret ; pop the return address into rip
Look at the stack: call has pushed 0x401003, labelled return address → _start+3. Keep stepping:
push rbpsaves the caller's frame pointer just below it.mov rbp, rspmakesrbppoint at that saved value. From here on, the return address is always at[rbp+8].mov eax, edi/add eax, esicompute 3 + 4 ineax.pop rbpandretundo the frame and jump back to_start+3witheax = 7.
The stack frame
push rbp / mov rbp, rsp is the classic prologue, and leave (or mov rsp, rbp + pop rbp) followed by ret is the epilogue. Between them, a function that needs local variables subtracts from rsp to reserve space, and addresses everything relative to rbp:
| Location | Holds |
|---|---|
[rbp+16] and up | stack arguments (7th and beyond) |
[rbp+8] | return address |
[rbp] | caller's saved rbp |
[rbp-4], [rbp-8], … | local variables |
This layout is what makes debuggers' backtraces work: each saved rbp points to the previous frame, forming a linked list up the stack. Optimized code often skips rbp altogether and addresses locals from rsp instead.
A few more System V rules you'll see the effects of:
- Alignment:
rspmust be a multiple of 16 just before acall, which is why functions sometimes subtract "too much" fromrsp. - Red zone: a function that calls nothing may use the 128 bytes below
rspwithout movingrspat all. - For variadic functions like
printf,alholds the number of vector registers used — that's themov eax, 0compilers put right beforecall printf.
32-bit cdecl: arguments on the stack
32-bit x86 has too few registers to spare, so cdecl passes every argument on the stack, pushed right to left — so the first argument ends up closest to the return address. The return value is in eax, and the caller removes the arguments afterwards.
- _start:
- push 4 ; arguments right to left…
- push 3 ; …so the 1st ends up on top
- call add2
- add esp, 8 ; the caller removes its 2 arguments
- ret
- add2:
- push ebp
- mov ebp, esp
- mov eax, DWORD PTR [ebp+8] ; 1st argument
- add eax, DWORD PTR [ebp+12] ; 2nd argument
- pop ebp
- ret
The demo stops just after mov ebp, esp. With ebp = 0x7fffffcc, the stack reads, from the top:
| Address | Relative to ebp | Holds |
|---|---|---|
0x7fffffcc | [ebp] | saved ebp |
0x7fffffd0 | [ebp+4] | return address |
0x7fffffd4 | [ebp+8] | 3 — 1st argument |
0x7fffffd8 | [ebp+12] | 4 — 2nd argument |
That [ebp+8] = first argument pattern is one of the most recognizable sights in 32-bit disassembly. After ret, the caller's add esp, 8 throws the two arguments away.
Other 32-bit conventions differ in the details: stdcall (used by the Win32 API) makes the callee clean up with ret 8, and fastcall passes the first two arguments in ecx and edx.
Who saves which registers
A convention also splits registers into two groups:
| System V x86-64 | cdecl (32-bit) | |
|---|---|---|
| Callee-saved — a function must restore them before returning | rbx, rbp, r12–r15 (and rsp) | ebx, esi, edi, ebp |
| Caller-saved — may be overwritten by any call | rax, rcx, rdx, rsi, rdi, r8–r11 | eax, ecx, edx |
So if a function uses rbx, you'll see it push rbx in the prologue and pop rbx before returning. And if a caller needs a value in rcx to survive a call, it has to save it first — or keep it in a callee-saved register.
Windows is different
64-bit Windows uses its own convention: the first four arguments go in rcx, rdx, r8, r9, the caller reserves 32 bytes of "shadow space" on the stack, and rsi/rdi are callee-saved. Before naming the arguments of a function in a disassembly, check which OS the binary targets.
Practice
In the simulator, the Function call and Recursion (factorial) examples are compiled C. Switch between 64-bit and 32-bit to watch the same function receive its arguments in edi or at [ebp+8], and follow the frames pile up in the stack view.