08-E.10.3: CPU and Memory Troubleshooting - OOM & swap
- Page ID
- 38095
The Out of Memory Manager (OOM)
The OOM manager checks to see if there is enough available memory to verify that the system is truly out of memory and if so, selects a process to kill. This is a controversial part of the VM and it has been suggested that it be removed on many occasions. Regardless of whether it exists in the latest kernel, it still is a useful system to examine as it touches off a number of other subsystems.
Checking Available Memory
For certain operations, such as expanding the heap or remapping an address space, the system will check if there is enough available memory to satisfy a request.
Unless the system administrator has specified that the system should over-commit memory, the amount of available memory will be checked. To determine how many pages are potentially available, Linux sums up the following bits of data:
- Total page cache as page cache is easily reclaimed
- Total free pages because they are already available
- Total free swap pages as user space pages may be paged out
- Total pages managed by swapper_space although this double-counts the free swap pages. This is balanced by the fact that slots are sometimes reserved but not used
- Total pages used by the dentry cache as they are easily reclaimed
- Total pages used by the inode cache as they are easily reclaimed
If the total number of pages added here is sufficient for the request, vm_enough_memory() returns true to the caller. If false is returned, the caller knows that the memory is not available and usually decides to return -ENOMEM to userspace.
Determining OOM Status
When the machine is low on memory, old page frames will be reclaimed, but despite reclaiming pages it may find that it was unable to free enough pages to satisfy a request even when scanning at highest priority.
Unfortunately, it is possible that the system is not out memory and simply needs to wait for IO to complete or for pages to be swapped to backing storage. This is unfortunate not because the system has memory, but because the function is being called unnecessarily, opening the possibly of processes being unnecessarily killed. Before deciding to kill a process, it goes through the following checklist:
- Is there enough swap space left? If yes, not OOM
- Has it been more than 5 seconds since the last failure? If yes, not OOM
- Have we failed within the last second? If no, not OOM
- If there haven't been 10 failures at least in the last 5 seconds, we're not OOM
- Has a process been killed within the last 5 seconds? If yes, not OOM
Only after the above tests have passed is a process selected to kill.
To avoid OOM killing processes there are two steps that can be taken:
- The fastest and easiest option is to upgrade the memory in the server.
- Reduce memory usage to stop any processes running that are not needed. This would entail finding all the processes that are started at boot time, looking at what processes are run by cron, and what users are running.
Swap Space Configurations
What is swap?
Swap space is the area on a hard disk. It is a part of your machine's Virtual Memory, which is a combination of accessible physical memory (RAM) and the swap space. Swap holds memory pages that are temporarily inactive. Swap space is used when your operating system decides that it needs physical memory for active processes and the amount of available (unused) physical memory is insufficient. When this happens, inactive pages from the physical memory are then moved into the swap space, freeing up that physical memory for other uses. Note that the access time for swap is slower, depending on the speed of the hard drive. Do not consider it to be a complete replacement for the physical memory. Swap space can be a dedicated swap partition (recommended), a swap file, or a combination of swap partitions and swap file(s).
Types of Linux Swap
Linux provides for two types of swap space. By default, most Linux installations create a swap partition, but it is also possible to use a specially configured file as a swap file. A swap partition is just what its name implies—a standard disk partition that is designated as swap space by the mkswap command.
A swap file can be used if there is no free disk space in which to create a new swap partition or space in a volume group where a logical volume can be created for swap space. This is just a regular file that is created and pre-allocated to a specified size. Then the mkswap command is run to configure it as swap space.
The mkswap Command
The mkswap sets up a Linux swap area on a device or in a file.
Syntax:
mkswap [ OPTIONS ] device [size]
Command Options:
| Options | Option Meaning |
|---|---|
| -a, --all | All devices marked as ''swap'' in /etc/fstab are made available, except for those with the ''noauto'' option. Devices that are already being used as swap are silently skipped. |
| -e, --ifexists | Silently skip devices that do not exist. |
| --show | Display a definable table of swap areas. See the --help output for a list of available columns. |
If a system doesn’t have swap space or if the swap space is not adequate enough, a swap file can be created on Linux. You can create multiple swap files as well.
pbmac@pbmac-server $ sudo fallocate -l 1G /swapfile
pbmac@pbmac-server $ sudo chmod 600 /swapfile
pbmac@pbmac-server $ sudo mkswap /swapfile
pbmac@pbmac-server $ sudo swapon /swapfile
pbmac@pbmac-server $ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 1024M 0B -2
You can see in the above example the use of the swapon command. There are a couple of options available in swapon.
There is also a swapoff command. This command disable devices and files for paging and swapping. The most common option is -a which deactivates all known swap devices and files (as found in /proc/swaps or /etc/fstab).
Adapted from:
"Chapter 13 Out Of Memory Management" by Unknown, Geeks for Geeks is in the Public Domain, CC0
"SwapFaq" by Jose Bartolome, Ubuntu Community Help Wiki is licensed under CC BY-SA 4.0
"An introduction to swap space on Linux systems" by David Both, opensource.com is licensed under CC BY-SA 4.0