04-F.13.2: Storage Issues - More IO Scheduling

Last updated
Save as PDF

Page ID: 26855

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

Selecting I/O Schedulers

Prior to Ubuntu 19.04 with Linux 5.0 or Ubuntu 18.04.3 with Linux 4.15, the multiqueue I/O scheduling was not enabled by default, and just the deadline, cfq and noop I/O schedulers were available by default.

For Ubuntu 19.10 with Linux 5.0 or Ubuntu 18.04.3 with Linux 5.0 onwards, multiqueue is enabled by default providing the bfq, kyber, mq-deadline and none I/O schedulers. For Ubuntu 19.10 with Linux 5.3 the deadline, cfq and noop I/O schedulers are deprecated.

With the Linux 5.0 kernels, one can disable these and fall back to the non-multiqueue I/O schedulers using a kernel parameter. For example for SCSI devices one can use:

scsi_mod.use_blk_mq=0

Add this to the GRUB_CMDLINE_LINUX_DEFAULT string in /etc/default/grub and run sudo update-grub to enable this option.

Changing an I/O scheduler is performed on a per block device basis. For example, for non-multi queue device /dev/sda one can see the current I/O schedulers available using the following:

pbmac@pbmac-server $ cat /sys/block/sda/queue/scheduler
noop deadline [cfq]

To change this to deadline use:

pbmac@pbmac-server $ echo "deadline" | sudo tee /sys/block/sda/queue/scheduler

For multiqueue devices the default will show:

pbmac@pbmac-server $ cat /sys/block/sda/queue/scheduler 
[mq-deadline] none

To use kyber, install the module:

pbmac@pbmac-server $ sudo modprobe kyber-iosched
pbmac@pbmac-server $ cat /sys/block/sda/queue/scheduler 
[mq-deadline] kyber none

and enable it:

pbmac@pbmac-server $ echo "kyber" | sudo tee /sys/block/sda/queue/scheduler

To use bfq, install the module:

pbmac@pbmac-server $ sudo modprobe bfq
pbmac@pbmac-server $ cat /sys/block/sda/queue/scheduler 
[mq-deadline] kyber none

and enable it:

pbmac@pbmac-server $ echo "bfq" | sudo tee /sys/block/sda/queue/scheduler

Tuning I/O Schedulers

Each I/O scheduler has a default set of tunable options that may be adjusted to help improve performance or fair sharing for your particular use case. The following kernel documentation covers these per-I/O scheduler tunable options:

deadline (and mq-deadline) deadline-iosched.txt
cfq cfq-iosched.txt
bfq bfq-iosched.txt
kyber kyber-iosched.txt

Best I/O scheduler to use

Different I/O requirements may benefit from changing from the Ubuntu distro default. A quick start guide to select a suitable I/O scheduler is below. The results are based on running 25 different synthetic I/O patterns generated using fio on ext4, xfs and btrfs with the various I/O schedulers using the 5.3 kernel.

SSD or NVME drives

It is worth noting that there is little difference in throughput between the mq-deadline/none/bfq I/O schedulers when using fast multi-queue SSD configurations or fast NVME devices. In these cases it may be preferable to use the 'none' I/O scheduler to reduce CPU overhead.

HDD

Avoid using the none/noop I/O schedulers for a HDD as sorting requests on block addresses reduce the seek time latencies and neither of these I/O schedulers support this feature. mq-deadline has been shown to be advantageous for the more demanding server related I/O, however, desktop users may like to experiment with bfq as it has been shown to load some applications faster.

Of course your use-case may differ - the above are just suggestions to start with based on some synthetic tests. You may find other choices with adjustments to the I/O scheduler tunables produce better results.

Adapted from:
"IOSchedulers" by Colin Ian King, Ubuntu Wiki is licensed under CC BY-SA 4.0