12.12: Calling Convention

The function prologue is the code at the beginning of a function and the function epilogue is the code at the end of a function. The operations performed by the prologue and epilogue are generally specified by the standard calling convention and deal with stack, registers, passed arguments (if any), and stack dynamic local variables (if any).

The general idea is that the program state (i.e., contents of specific registers and the stack) are saved, the function executed, and then the state is restored. Of course, the function will often require extensive use of the registers and the stack. The prologue code helps save the state and the epilogue code restores the state.

Parameter Passing

As noted, a combination of registers and the stack is used to pass parameters to and/or from a function.

The first six integer arguments are passed in registers as follows:

Argument Number	Argument Size
	64-bits	32-bits	16-bits	8-bits
1	rdi	edi	di	dil
2	rsi	esi	si	sil
3	rdx	edx	dx	dl
4	rcx	ecx	cx	cl
5	r8	r8d	r8w	r8b
6	r9	r9d	r9w	r9b

The seventh and any additional arguments are passed on the stack. The standard calling convention requires that, when passing arguments (values or addresses) on the stack, the arguments should be pushed in reverse order. That is “someFunc (one, two, three, four, five, six, seven, eight, nine)” would imply a push order of: nine, eight, and then seven.

For floating-point arguments, the floating-point registers xmm0 to xmm7 are used in that order for the first eight float arguments.

Additionally, when the function is completed, the calling routine is responsible for clearing the arguments from the stack. Instead of doing a series of pop instructions, the stack pointer, rsp, is adjusted as necessary to clear the arguments off the stack. Since each argument is 8 bytes, the adjustment would be adding [(number of arguments) * 8] to the rsp.

For value returning functions, the result is placed in the A register based on the size of the value being returned.

Specifically, the values are returned as follows:

Return Value Size	Location
byte	al
word	ax
double-word	eax
quadword	rax
floating-point	xmm0

The rax register may be used in the function as needed as long as the return value is set appropriately before returning.

Register Usage

The standard calling convention specifies the usage of registers when making function calls. Specifically, some registers are expected to be preserved across a function call. That means that if a value is placed in a preserved register or saved register, and the function must use that register, the original value must be preserved by placing it on the stack, altered as needed, and then restored to its original value before returning to the calling routine. This register preservation is typically performed in the prologue and the restoration is typically performed in the epilogue.

The following table summarizes the register usage.

Register	Usage
rax	Return Value
rbx	Callee Saved
rcx	\(4^{th}\) Argument
rdx	\(3^{rd}\) Argument
rsi	\(2^{nd}\) Argument
rdi	\(1^{st}\) Argument
rbp	Callee Saved
rsp	Stack Pointer
r8	\(5^{th}\) Argument
r9	\(6^{th}\) Argument
r10	Temporary
r11	Temporary
r12	Callee Saved
r13	Callee Saved
r14	Callee Saved
r15	Callee Saved

The temporary registers (r10 and r11) and the argument registers (rdi, rsi, rdx, rcx, r8, and r9) are not preserved across a function call. This means that any of these registers may be used in the function without the need to preserve the original value.

Additionally, none of the floating-point registers are preserved across a function call. Refer to Chapter 18 for more information regarding floating-point operations.

12.8.3 Call Frame

The items on the stack as part of a function call are referred to as a call frame (also referred to as an activation record or stack frame). Based on the standard calling convention, the items on the stack, if any, will be in a specific general format.

The possible items in the call frame include:

• Return address (required).
• Preserved registers (if any).
• Passed arguments (if any).
• Stack dynamic local variables (if any).

Other items may be placed in the call frame such as static links for dynamically scoped languages. Such topics are outside the scope of this text and will not be discussed here. For some functions, a full call frame may not be required. For example, if the function:

• Is a leaf function (i.e., does not call another function).
• Passes its arguments only in registers (i.e., does not use the stack).
• Does not alter any of the saved registers.
• Does not require stack-based local variables.

This can occur for simpler, smaller leaf functions. However, if any of these conditions is not true, a full call frame is required.

For more non-leaf or more complex functions, a more complete call frame is required.

The standard calling convention does not explicitly require use of the frame pointer register, rbp. Compilers are allowed to optimize the call frame and not use the frame pointer. To simplify and clarify accessing stack-based arguments (if any) and stack dynamic local variables, this text will utilize the frame pointer register. This is similar to how many other architectures use a frame pointer register.

As such, if there are any stack-based arguments or any local variables needed within a function, the frame pointer register, rbp, should be pushed and then set pointing to itself. As additional pushes and pops are performed (thus changing rsp), the rbp register will remain unchanged. This allows the rbp register to be used as a reference to access arguments passed on the stack (if any) or stack dynamic local variables (if any).

For example, assuming a function call has eight (8) arguments and assuming the function uses rbx, r12, and r13 registers (and thus must be pushed), the call frame would be as follows:

The stack-based arguments are accessed relative to the rbp. Each item push is a quadword which uses 8 bytes. For example, [rbp+16] is the location of the first passed argument (\(7^{th}\) integer argument) and [rbp+24] is the location of the second passed argument (\(8^{th}\) integer argument).

In addition, the call frame would contain the assigned locations of local variables (if any). The section on local variables details the specifics regarding allocating and using local variables.

Zone

In the Linux standard calling convention, the first 128-bytes after the stack pointer, rsp, are reserved. For example, extending the previous example, the call frame would be as follows:

This red zone may be used by the function without any adjustment to the stack pointer. The purpose is to allow compiler optimizations for the allocation of local variables. This does not directly impact programs written directly in assembly language.