How Linux System Calls Work

Let us take a deep dive into how system calls are executed by the operating system.

The first thing that happens working with system calls is that the process first switches from user mode to kernel mode. When in user mode the OS is executing user level instructions and in order to execute kernel level instructions it would need certain elevated privileges.

Step 1: The system needs to switch to the kernel mode and for that to happen, the wrapper puts the system call into a specific register. We have a sys-call table which identifies the system call by a unique identifier.

Step 2: The system then executes a trap instruction which switches the system from the user mode to the kernel mode and then begins executing the code at location 0x80 of the systems trap vector.

Step 3: Now that the instruction is in memory in response to the trap instruction it makes a system_call() with certain parameters to handle the trap. This handler does certain things

• Checks for the validity of the system call

• Saves register values on the kernel stack (We have two stacks inside an OS one is the kernel stack and one is the user stack)

• Checks for the arguments in system call, if the memory calls are valid and not out of bounds.

• After the sys-call has returned it makes the appropriate changes to the registers and then puts the return value back onto the stack, to be returned back to the wrapper.

• Returns the processor back to the user mode.

Step 4: If the return value of the system call service routine indicated an error, the wrapper function sets the global variable errno using this value. The wrapper function then returns to the caller, providing an integer return value indicating the success or failure of the system call