08-E.10.1: CPU and Memory Troubleshooting - sar & Memory Issues

Last updated
Save as PDF

Page ID: 37697

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

The sar Command

The sar command can be used to monitor Linux system’s resources like CPU usage, memory utilization, I/O devices consumption, network monitoring, Disk usage, process and thread allocation, battery performance, plug and play devices, processor performance, file system and more. Linux system monitoring and analyzing aids in understanding system resource usage, which can help to improve system performance to handle more requests.

By default sar command displays results on the output screen; in addition results can also be stored in the file specified by the -o filename option. Any user can collect information about system performance using system activities flags. The sar command will show only CPU monitoring activity if any flag is not specified by the user.

pbmac@pbmac-server $ sar -u 2 5
Linux 4.15.0-91-generic (pbmac-server)     09/15/2020     _x86_64_    (2 CPU)

05:26:47 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
05:26:49 PM     all     13.99      0.00      1.02      1.02      0.00     83.97
05:26:51 PM     all      6.94      0.00      0.51      0.00      0.00     92.54
05:26:53 PM     all      7.42      0.00      1.02      1.53      0.00     90.03
05:26:55 PM     all      6.89      0.00      1.28      0.51      0.00     91.33
05:26:57 PM     all      7.73      0.00      0.77      1.55      0.00     89.95
Average:        all      8.60      0.00      0.92      0.92      0.00     89.55

Common Memory Issues

There are numerous issues that can be caused by memory problems, or that are related to memory issues.

It is possible to load down a system to the point that there is not enough total memory to service all processes at once.
If the system is a busy system, or does not have sufficient memory, then there may not be enough free memory to service new processes.
There is the potential for processes to be unable to access memory even though it is available.
Swap partitions can up too much or not enough storage space.
Other memory issues can impact the system, but there are tools to help find and resolve memory issues.

The /proc/meminfo File

This file contains all the data about the memory usage. It provides the current memory usage details rather than old stored values. It is one of the more commonly used files in the /proc directory, as it reports a large amount of valuable information about the systems' RAM usage.

The following sample /proc/meminfo virtual file is from a system with 256 MB of RAM and 512 MB of swap space:

MemTotal:       255908 kB
MemFree:         69936 kB
Buffers:         15812 kB
Cached:         115124 kB
SwapCached:          0 kB
Active:          92700 kB
Inactive:        63792 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       255908 kB
LowFree:         69936 kB
SwapTotal:      524280 kB
SwapFree:       524280 kB
Dirty:               4 kB
Writeback:           0 kB
Mapped:          42236 kB
Slab:            25912 kB
Committed_AS:   118680 kB
PageTables:       1236 kB
VmallocTotal:  3874808 kB
VmallocUsed:      1416 kB
VmallocChunk:  3872908 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     4096 kB

Much of the information here is used by the free, top, and ps commands. In fact, the output of the free command is similar in appearance to the contents and structure of /proc/meminfo. But by looking directly at /proc/meminfo, more details are revealed:

MemTotal — Total amount of physical RAM, in kilobytes.
MemFree — The amount of physical RAM, in kilobytes, left unused by the system.
Buffers — The amount of physical RAM, in kilobytes, used for file buffers.
Cached — The amount of physical RAM, in kilobytes, used as cache memory.
SwapCached — The amount of swap, in kilobytes, used as cache memory.
Active — The total amount of buffer or page cache memory, in kilobytes, that is in active use. This is memory that has been recently used and is usually not reclaimed for other purposes.
Inactive — The total amount of buffer or page cache memory, in kilobytes, that is free and available. This is memory that has not been recently used and can be reclaimed for other purposes.
HighTotal and HighFree — The total and free amount of memory, in kilobytes, that is not directly mapped into kernel space. The HighTotal value can vary based on the type of kernel used.
LowTotal and LowFree — The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used.
SwapTotal — The total amount of swap available, in kilobytes.
SwapFree — The total amount of swap free, in kilobytes.
Dirty — The total amount of memory, in kilobytes, waiting to be written back to the disk.
Writeback — The total amount of memory, in kilobytes, actively being written back to the disk.
Mapped — The total amount of memory, in kilobytes, which have been used to map devices, files, or libraries using the mmap command.
Slab — The total amount of memory, in kilobytes, used by the kernel to cache data structures for its own use.
Committed_AS — The total amount of memory, in kilobytes, estimated to complete the workload. This value represents the worst case scenario value, and also includes swap memory.
PageTables — The total amount of memory, in kilobytes, dedicated to the lowest page table level.
VMallocTotal — The total amount of memory, in kilobytes, of total allocated virtual address space.
VMallocUsed — The total amount of memory, in kilobytes, of used virtual address space.
VMallocChunk — The largest contiguous block of memory, in kilobytes, of available virtual address space.
HugePages_Total — The total number of hugepages for the system. The number is derived by dividing Hugepagesize by the megabytes set aside for hugepages specified in /proc/sys/vm/hugetlb_pool. This statistic only appears on the x86, Itanium, and AMD64 architectures.
HugePages_Free — The total number of hugepages available for the system. This statistic only appears on the x86, Itanium, and AMD64 architectures.
Hugepagesize — The size for each hugepages unit in kilobytes. By default, the value is 4096 KB on uniprocessor kernels for 32 bit architectures. For SMP, hugemem kernels, and AMD64, the default is 2048 KB. For Itanium architectures, the default is 262144 KB. This statistic only appears on the x86, Itanium, and AMD64 architectures.

Adapted from:
"C.2.18. /proc/meminfo" by Multiple Contributors, Fedora Docs is licensed under CC BY-SA 4.0