We mentioned how one of the major tasks of the operating system is to implement security; that is to not allow one application or user to interfere with any other that is running in the system. This means applications should not be able to overwrite each others memory or files, and only access system resources as dictated by system policy.
However, when an application is running it has exclusive use of the processor. We see how this works when we examine processes in the next chapter. Ensuring the application only accesses memory it owns is implemented by the virtual memory system, which we examine in the chapter after next. The essential point is that the hardware is responsible for enforcing these rules.
The system call interface we have examined is the
gateway to the application getting to system resources. By
forcing the application to request resources through a system
call into the kernel, the kernel can enforce rules about what
sort of access can be provided. For example, when an
application makes an
system call to open a file on disk, it will check the
permissions of the user against the file permissions and
allow or deny access.
Hardware protection can usually be seen as a set of concentric rings around a core set of operations.
In the inner most ring are the most protected
instructions; those that only the kernel should be allowed to
call. For example, the
instruction to halt the processor should not be allowed to be
run by a user application, since it would stop the entire
computer from working. However, the kernel needs to be able
to call this instruction when the computer is legitimately
Each inner ring can access any instructions protected by a further out ring, but not any protected by a further in ring. Not all architectures have multiple levels of rings as above, but most will either provide for at least a "kernel" and "user" level.
386 protection model
The 386 protection model has four rings, though most operating systems (such as Linux and Windows) only use two of the rings to maintain compatibility with other architectures that do now allow as many discrete protection levels.
386 maintains privileges by making each piece of application code running in the system have a small descriptor, called a code descriptor, which describes, amongst other things, its privilege level. When running application code makes a jump into some other code outside the region described by its code descriptor, the privilege level of the target is checked. If it is higher than the currently running code, the jump is disallowed by the hardware (and the application will crash).
Applications may only raise their privilege level by specific calls that allow it, such as the instruction to implement a system call. These are usually referred to as a call gate because they function just as a physical gate; a small entry through an otherwise impenetrable wall. When that instruction is called we have seen how the hardware completely stops the running application and hands control over to the kernel. The kernel must act as a gatekeeper; ensuring that nothing nasty is coming through the gate. This means it must check system call arguments carefully to make sure it will not be fooled into doing anything it shouldn't (if it can be, that is a security bug). As the kernel runs in the innermost ring, it has permissions to do any operation it wants; when it is finished it will return control back to the application which will again be running with its lower privilege level.
Fast System Calls
One problem with traps as described above is that they are very expensive for the processor to implement. There is a lot of state to be saved before context can switch. Modern processors have realised this overhead and strive to reduce it.
To understand the call-gate mechanism described above requires investigation of the ingenious but complicated segmentation scheme used by the processor. The original reason for segmentation was to be able to use more than the 16 bits available in a register for an address, as illustrated in Figure 4.4, “x86 Segmentation Addressing”.
When x86 moved to 32 bit registers, the segmentation scheme remained but in a different format. Rather than fixed segment sizes, segments are allowed to be any size. This means the processor needs to keep track of all these different segments and their sizes, which it does using descriptors. The segment descriptors available to everyone are kept in the global descriptor table or GDT for short. Each process has a number of registers which point to entries in the GDT; these are the segments the process can access (there are also local descriptor tables, and it all interacts with task state segments, but that's not important now). The overall situation is illustrated in Figure 4.5, “x86 segments”.
Since the operating system assigns the segment registers as part of the process state, the processor hardware knows what segments of memory the currently running process can access and can enforce protection to ensure the process doesn't touch anything it is not supposed to. If it does go out of bounds, you receive a segmentation fault, which most programmers are familiar with.
The picture becomes more interesting when running code needs to make calls into code that resides in another segment. As discussed in the section called “386 protection model”, x86 does this with rings, where ring 0 is the highest permission, ring 3 is the lowest, and inner rings can access outer rings but not vice-versa.
As discussed in the section called “Raising Privilege”, when ring 3 code wants to jump into ring 0 code, it is essentially modifying its code segment selector to point to a different segment. To do this, it must use a special far-call instruction which hardware ensures passes through the call gate. There is no other way for the running process to choose a new code-segment descriptor, and hence the processor will start executing code at the known offset within the ring 0 segment, which is responsible for maintaining integrity (e.g. not reading arbitrary and possibly malicious code and executing it. Of course nefarious attackers will always look for ways to make your code do what you did not intend it to!).
This allows a whole hierarchy of segments and
permissions between them. You might have noticed a cross
segment call sounds exactly like a system call. If you've
ever looked at Linux x86 assembly the standard way to make
a system call is
0x80, which raises interrupt
0x80. An interrupt stops
the processor and goes to an interrupt gate, which then
works the same as a call gate -- it changes privilege
level and bounces you off to some other area of code
The problem with this scheme is that it is slow. It takes a lot of effort to do all this checking, and many registers need to be saved to get into the new code. And on the way back out, it all needs to be restored again.
On a modern x86 system segmentation and the four-level
ring system is not used thanks to virtual memory,
discussed fully in Chapter 6, Virtual Memory. The
only thing that really happens with segmentation switching
is system calls, which essentially switch from mode 3
(userspace) to mode 0 and jump to the system call handler
code inside the kernel. Thus the processor provides extra
fast system call instructions called
sysexit to get back)
which speed up the whole process over a
int 0x80 call by removing
the general nature of a far-call — that is the
possibility of transitioning into any segment at any ring
level — and restricting the call to only transition
to ring 0 code at a specific segment and offset, as stored
Because the general nature has been replaced with so much prior-known information, the whole process can be speed up, and hence we have a the aforementioned fast system call. The other thing to note is that state is not preserved when the kernel gets control. The kernel has to be careful to not to destroy state, but it also means it is free to only save as little state as is required to do the job, so can be much more efficient about it. This is a very RISC philosophy, and illustrates how the line blurs between RISC and CISC processors.
For more information on how this is implemented in the Linux kernel, see the section called “Kernel Library”.
Other ways of communicating with the kernel
about proc, sysfs, debugfs, etc
 What happens when a "naughty" application calls that instruction anyway? The hardware will usually raise an exception, which will involve jumping to a specified handler in the operating system similar to the system call handler. The operating system will then probably terminate the program, usually giving the user some error about how the application has crashed.