Functions and the stack recap

This chapter demystified the boilerplate you had been skimming since Chapter 1: the pushes and pops around every function, the arguments arriving in edi, the results landing in eax. All of it is a conversation between caller and callee, conducted on the call stack according to a shared rulebook. Let's review the key ideas from each lesson.

What Is the Stack?

The call stack is a region of memory holding function arguments, local variables, and return addresses, with rsp (the stack pointer) always pointing at its current top. On x86-64 the stack grows downward: push subtracts from rsp then stores, and pop reads then adds back. call pushes the return address and jumps; ret pops it and jumps back, so the stack rises and falls in step with the call chain. Runaway recursion drives rsp off the end of the region: a stack overflow.

Prologue and Epilogue

Almost every unoptimized function opens with the prologue push rbp / mov rbp, rsp, which saves the caller's frame pointer and establishes a new stack frame, often followed by sub rsp, N to reserve local space. rbp (the frame pointer) stays parked so locals can be addressed as stable offsets like -4[rbp] no matter how rsp moves. The epilogue undoes it all: either pop rbp / ret, or leave / ret, where leave equals mov rsp, rbp followed by pop rbp.

The System V Calling Convention

On Linux, macOS, and the BSDs, the System V AMD64 ABI dictates how functions communicate. The first six integer or pointer arguments travel in rdi, rsi, rdx, rcx, r8, r9 (in that order; the seventh and beyond go on the stack), and results come back in rax. Caller-saved registers (rax, the argument registers, r10, r11) may be clobbered by any call, while callee-saved registers (rbx, rbp, r12-r15) must be restored before returning, which is exactly why the prologue pushes rbp. 64-bit Windows uses a different convention.

Passing Arguments in Registers

The canonical call pattern is a cluster of moves into the argument registers followed by a call:

    mov  edx, 9    ; third argument
    mov  esi, 8    ; second argument
    mov  edi, 7    ; first argument
    call combine(int, int, int)

The order of the moves does not matter, only which register each value lands in. At -O0 the callee immediately spills each argument into a stack slot, in argument order; at -O2 the spilling disappears and the arguments are used straight out of their registers, but the convention itself never changes.

Return Values

A return compiles to "put the value in rax, then ret"; there is no separate hardware return mechanism, only the convention. The register name follows the type's size: al for a bool or char, ax for a short, eax for an int, the full rax for a long or a pointer. A read of eax immediately after a call is the pattern for collecting a result, and rax is caller-saved precisely because carrying results out is its job.

Floating Point and the xmm Registers

float and double never touch the general-purpose registers: they live in the 128-bit SSE registers xmm0-xmm15. Scalar instructions carry a suffix, ss for a float and sd for a double, giving movsd, addsd, mulsd, and friends. Floating-point arguments go in xmm0-xmm7 and results return in xmm0, counted independently of the integer sequence, so in f(int a, double b, int c) the arguments land in edi, xmm0, and esi. cvtsi2sd and cvttsd2si convert between the two worlds, and comisd compares doubles, setting the ordinary flags for the unsigned-style jumps (ja, jbe).

Local Variables on the Stack

At -O0, every local lives in a stack slot at a fixed negative offset from rbp, with int slots stepping down in 4-byte (DWORD) increments: -4[rbp], -8[rbp], -12[rbp]. Counting the distinct [rbp-N] offsets gives a quick estimate of how many locals and spilled arguments a function has. Locals sit in memory for debuggability, not speed; at -O2 the optimizer keeps them in registers or eliminates them entirely, and a whole function can collapse to mov eax, 30 / ret.

Key Terminology

  • Call stack: The memory region holding arguments, locals, and return addresses for active calls
  • rsp (stack pointer): Points at the current top of the stack; decreasing it grows the stack
  • Return address: The address call pushes so ret knows where to resume
  • Stack overflow: Running off the end of the stack region, typically from runaway recursion
  • Prologue / epilogue: Sets up (push rbp / mov rbp, rsp) and tears down (leave or pop rbp, then ret) a stack frame
  • rbp (frame pointer): Parked at a fixed spot so locals are stable offsets like -4[rbp]
  • Stack frame: One call's slice of the stack: saved rbp, locals, spilled arguments
  • Calling convention: The shared rulebook for passing arguments and returning results
  • System V AMD64 ABI: The convention on Unix-like x86-64 systems
  • Argument registers: rdi, rsi, rdx, rcx, r8, r9 for the first six integer/pointer arguments
  • Caller-saved (volatile) registers: May be clobbered by a call; the caller preserves them if needed
  • Callee-saved (non-volatile) registers: rbx, rbp, r12-r15; the callee must restore them
  • Spill: Storing a register value into a stack slot, as -O0 does with every incoming argument
  • xmm registers: The 128-bit SSE register file where float and double values live
  • ss / sd suffixes: Scalar single (float) and scalar double (double) instruction variants
  • cvtsi2sd / cvttsd2si: Convert integer to double and back; the extra t means truncation
  • comisd: Compares doubles, setting the ordinary flags for the unsigned-style jumps

Looking Forward

You can now read a complete function: its frame, its arguments, its locals, and its result. The next chapter turns from code to data: where global and local storage differ, how pointers and references look in assembly, how array indexing becomes scaled addressing, and how struct members turn into fixed offsets. The [rbp-N] addressing you mastered here is the foundation for all of it.