The System V Calling Convention

The System V Calling Convention

When one function calls another, they have to agree on the details. Where does the first argument go? The second? Where will the result come back? If the caller stuffs an argument into one register but the callee looks for it in another, everything falls apart. The set of rules that prevents this is called a calling convention, and on Linux, macOS, and the BSDs running on x86-64, that convention is the System V AMD64 ABI.

You do not need to memorise the whole specification. But a handful of its rules explain almost everything you will see around a call, so they are worth learning well.

Arguments Go in Registers (Mostly)

The first and most useful rule: the first six integer or pointer arguments are passed in registers, in this exact order:

A handy way to remember the start of the sequence is "di, si, d, c" for rdi, rsi, rdx, rcx, then the numbered registers r8, r9. When an argument is int-sized (4 bytes) the compiler uses the 32-bit name of the same register: edi, esi, edx, ecx, r8d, r9d. This is why the very first thing you saw a function do, back in Chapter 1, was read its argument out of edi: that is simply where the first argument lives.

A seventh argument and beyond do not fit in this list, so they are passed on the stack instead. That is the slower path, and one practical reason functions with few parameters tend to be a touch more efficient.

(Floating-point arguments use a separate set of registers, xmm0 through xmm7. We focus on integer and pointer arguments here, since those cover the vast majority of what you will read early on.)

The Return Value Comes Back in rax

The second rule you already know: an integer or pointer result is returned in rax (or its 32-bit name eax for an int). The caller knows to look there the moment the callee returns. Return values get their own lesson shortly, but the headline is just that: results come home in rax.

Caller-Saved vs Callee-Saved Registers

Here is the rule that is less obvious but explains a lot of the pushing and popping you will see. There are only so many registers, and both the caller and the callee want to use them. So the convention splits the general-purpose registers into two groups, deciding who is responsible for preserving each one across a call.

Caller-saved (volatile) registers can be freely clobbered by the called function. If the caller has something valuable in one of these and wants it to survive the call, the caller must save it first (typically by pushing it) and restore it afterward. These include the argument registers and rax:

rax, rdi, rsi, rdx, rcx, r8, r9, r10, r11

The logic makes sense: the callee is going to use rdi, rsi, and friends to receive its arguments and build its result, so the caller cannot expect them to be untouched.

Callee-saved (non-volatile) registers are the opposite. The called function may use them, but if it does, it must restore their original values before returning, so the caller sees them unchanged. These are:

rbx, rbp, r12, r13, r14, r15

This is exactly why the prologue did push rbp: rbp is callee-saved, so a function that wants to use it as a frame pointer must save the caller's copy first and restore it (via pop rbp or leave) before returning. If you ever see a function open with several pushes like push rbx / push r12, you are watching it honour the callee-saved rule for the registers it is about to borrow.

Why This Matters for Reading Assembly

Once these rules are in your head, a lot of code stops being mysterious:

• Values flowing into rdi, rsi, rdx, ... right before a call are arguments being set up.
• A value read out of rax right after a call is the return value being collected.
• A cluster of push/pop of rbx, r12–r15 around a function body is it preserving callee-saved registers it borrowed.
• The compiler avoiding the argument and rax registers for values it needs after a call (or saving them first) is it respecting the caller-saved rule.

These are not arbitrary instructions; they are a conversation between caller and callee conducted according to a shared rulebook. The next two lessons zoom in on the two halves you will see most: arguments going in, and the result coming out.

A Quick Note on Other Platforms

System V is the convention on Unix-like systems. 64-bit Windows uses a different convention (the Microsoft x64 ABI), where the first four integer arguments go in rcx, rdx, r8, r9 instead. This course targets System V because it matches Linux, macOS, and Compiler Explorer's defaults, but it is worth knowing the rules change if you ever read Windows assembly.

Key Takeaways

• A calling convention is the shared rulebook for how functions pass arguments and return values; on Unix-like x86-64 it is the System V AMD64 ABI.
• The first six integer/pointer arguments go in rdi, rsi, rdx, rcx, r8, r9, in that order; further arguments go on the stack.
• Integer and pointer return values come back in rax (eax for int).
• Caller-saved registers (rax, rdi, rsi, rdx, rcx, r8–r11) may be clobbered by a call; the caller saves them if needed.
• Callee-saved registers (rbx, rbp, r12–r15) must be preserved by the called function; this is why the prologue pushes rbp.
• 64-bit Windows uses a different convention (rcx, rdx, r8, r9 for the first four arguments).