Intel x86 Assembly Language & Microarchitecture Optimization Linux system calls with less bloat
Example
In 32-bit Linux, system calls are usually done by using the sysenter instruction (I say usually because older programs use the now deprecated int 0x80) however, this can take up quite alot of space in a program and so there are ways that one can cut corners in order to shorten and speed things up.
This is usually the layout of a system call on 32-bit Linux:
mov eax, <System call number>
mov ebx, <Argument 1> ;If applicable
mov ecx, <Argument 2> ;If applicable
mov edx, <Argument 3> ;If applicable
push <label to jump to after the syscall>
push ecx
push edx
push ebp
mov ebp, esp
sysenter
That's massive right! But there are a few tricks we can pull to avoid this mess.
The first is to set ebp to the value of esp decreased by the size of 3 32-bit registers, that is, 12 bytes. This is great so long as you are ok with overwriting ebp, edx and ecx with garbage (such as when you will be moving a value into those registers directly after anyway), we can do this using the LEA instruction so that we do not need to affect the value of ESP itself.
mov eax, <System call number>
mov ebx, <Argument 1>
mov ecx, <Argument 2>
mov edx, <Argument 3>
push <label to jump to after the syscall>
lea ebp, [esp-12]
sysenter
However, we're not done, if the system call is sys_exit we can get away with not pushing anything at all to the stack!
mov eax, 1
xor ebx, ebx ;Set the exit status to 0
mov ebp, esp
sysenter
