One of the challenges of testing the performance of your storage system or file system is to find a tool that can generate realistic and flexible read and write workloads. FIO is such a tool that can create various scenarios of write operations and report various statistics such as bandwidth, latency, and IOPS.
This article describes how to use FIO to test read and write workloads on your storage system or file system and below are few command like options you can use with FIO and their meanings.
Arguments | Meaning |
---|---|
--name=str |
FIO will create a file with specified name to run the test on it. |
--ioengine=str |
Defines how the job issues I/O to the test file. - libaio - Linux native asynchronous block level I/O- solarisaio - Solaris native asynchronous block level I/O- windowsaio - Windows native asynchronous block level I/O |
--size |
The size of the file on which the FIO will run the benchmarking test. |
--rw=str |
Specifies the type of I/O pattern. - read - sequential read- write - sequential write- randread - random reads- randwrite - random writes- rw - sequential mix of read and writes- randrw - random mix of read and writes |
--rwmixread=int |
Specifies read/write distribution ration.rwmixread=30 would mean that 30% of I/O will be reads and 70% will be writes. |
--bs=int |
Defines the block size that the test will be using for generating I/O. Default 4k |
--direct=bool |
Decides wheather to use buffered I/O or Direct I/O - 1 means true and it will use the direct I/O- 0 means false and it will use the buffered I/O |
--numjobs=int |
The number of threads spawned be the test. |
--iodepth=int |
Controls how many I/O requests it issues to the OS at any given time. A sequential job with --iodepth=2 will submit two sequential IO requests at a time. |
--runtime=int |
The amount of time the test will be running in seconds. |
--time_based |
If specified, run for the specified duration even if the files are completely read of written. This same workload will be repeated as many times as runtime is allowed. |
--startdelay |
Add a delay in seconds between the intial test file created and the actual test. We can use some delay to avoid reading from write cache. |
Example: Read Workload
- Running fiotest with
libaio
engine and performing sequential read with direct I/O - The Size of the file is 1Gb
- Number of
jobs
is 4 andiodepth
is 8fio --name=fiotest --ioengine=libaio --size 1Gb --rw=read --bs=1M --direct=1 --numjobs=4 --iodepth=8 --runtime=60 --startdelay=60 --group_reporting
Starting 4 processes
fiotest: Laying out IO file(s) (1 file(s) / 1024MB) <-- 1
fiotest: Laying out IO file(s) (1 file(s) / 1024MB) <-- 1
fiotest: Laying out IO file(s) (1 file(s) / 1024MB) <-- 1
fiotest: Laying out IO file(s) (1 file(s) / 1024MB) <-- 1
Jobs: 3 (f=3): [R(2),_(1),R(1)] [98.9% done] [129.0MB/0KB/0KB /s] [129/0/0 iops] [eta 00m:01s]
fiotest: (groupid=0, jobs=4): err= 0: pid=21786: Mon Aug 22 11:32:58 2022
read : io=4096.0MB, bw=135418KB/s, iops=132, runt= 30973msec <-- 2
slat (usec): min=37, max=146352, avg=29015.91, stdev=26314.40 <-- 3
clat (msec): min=9, max=412, avg=207.45, stdev=57.88 <-- 4
lat (msec): min=10, max=491, avg=236.47, stdev=65.97 <-- 5
clat percentiles (msec):
| 1.00th=[ 18], 5.00th=[ 115], 10.00th=[ 116], 20.00th=[ 165],
| 30.00th=[ 212], 40.00th=[ 223], 50.00th=[ 223], 60.00th=[ 225],
| 70.00th=[ 233], 80.00th=[ 235], 90.00th=[ 235], 95.00th=[ 277],
| 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 363], 99.95th=[ 367],
| 99.99th=[ 412]
lat (msec) : 10=0.02%, 20=2.95%, 50=0.20%, 100=1.25%, 250=90.14% <-- 6
lat (msec) : 500=5.44% <-- 6
cpu : usr=0.01%, sys=0.21%, ctx=5686, majf=0, minf=8234 <-- 7
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0% <-- 8
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% <-- 9
complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% <-- 10
issued : total=r=4096/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 <-- 11
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: io=4096.0MB, aggrb=135418KB/s, minb=135418KB/s, maxb=135418KB/s, mint=30973msec, maxt=30973msec <-- 12
Disk stats (read/write):
nvme0n1: ios=16353/12, merge=0/2, ticks=1694705/756, in_queue=1695460, util=99.69% <-- 13Example 3: Write Workload
fio --name=fiotest --ioengine=libaio --size 1Gb --rw=write --bs=1M --direct=1 --numjobs=4 --iodepth=8 --runtime=60 --startdelay=60 --group_reporting
fiotest: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
...
fio-2.14
Starting 4 processes
fiotest: Laying out IO file(s) (1 file(s) / 1024MB)
fiotest: Laying out IO file(s) (1 file(s) / 1024MB)
fiotest: Laying out IO file(s) (1 file(s) / 1024MB)
fiotest: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 3 (f=3): [_(1),W(3)] [97.8% done] [0KB/129.0MB/0KB /s] [0/129/0 iops] [eta 00m:02s]
fiotest: (groupid=0, jobs=4): err= 0: pid=1654: Wed Aug 24 04:27:55 2022
write: io=4096.0MB, bw=135422KB/s, iops=132, runt= 30972msec
slat (usec): min=44, max=259568, avg=28738.66, stdev=24563.74
clat (msec): min=4, max=608, avg=207.46, stdev=72.17
lat (msec): min=4, max=633, avg=236.20, stdev=80.91
clat percentiles (msec):
| 1.00th=[ 38], 5.00th=[ 63], 10.00th=[ 115], 20.00th=[ 169],
| 30.00th=[ 188], 40.00th=[ 210], 50.00th=[ 212], 60.00th=[ 217],
| 70.00th=[ 235], 80.00th=[ 235], 90.00th=[ 293], 95.00th=[ 338],
| 99.00th=[ 392], 99.50th=[ 445], 99.90th=[ 545], 99.95th=[ 562],
| 99.99th=[ 611]
lat (msec) : 10=0.07%, 20=0.12%, 50=3.27%, 100=3.44%, 250=76.17%
lat (msec) : 500=16.75%, 750=0.17%
cpu : usr=0.23%, sys=0.17%, ctx=5730, majf=0, minf=44
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=4096/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=135422KB/s, minb=135422KB/s, maxb=135422KB/s, mint=30972msec, maxt=30972msec
Disk stats (read/write):
nvme0n1: ios=0/16358, merge=0/0, ticks=0/1389281, in_queue=1389282, util=99.66%Interpreting the FIO output:
In the Read Workload section refer to the
<--
prefixes which correspond to the line items below-
This will create 4 files with 1Gb of size.
-
fiotest.0.0
,fiotest.1.0
,fiotest.2.0
,fiotest.3.0
-
-
This following line is important
read : io=4096.0MB, bw=135418KB/s, iops=132, runt= 30973msec
- FIO did 8GB I/O at 135MB/s for a total 132 iops and ran for 31 seconds
-
slat
is submission latency.- Basically means, How long it take to submit this I/O to the kernel for processing.
slat (usec): min=37, max=146352, avg=29015.91, stdev=26314.40
- usec means microseconds.
-
clat
is completion latency.- This is the time between submission to kernel and time when I/O is completed.
clat (msec): min=9, max=412, avg=207.45, stdev=57.88
-
msec
means milliseconds
-
lat
is total latency.- This is the time between I/O creation and its completion. This is the sum of
slat
andclat
. lat (msec): min=10, max=491, avg=236.47, stdev=65.97
- This is the time between I/O creation and its completion. This is the sum of
-
lat
distribution.- This is distribution of I/O completion latencies. This is the time from when I/O leaves FIO and when it gets completed.
-
10=0.02%
means0.02%
of the I/O completed under10
milliseconds. -
20=2.95%
means2.95%
of the I/O completed under20
milliseconds. -
250=90.14%
means90.14%
of the I/O completed under250
milliseconds.
-
cpu
- This is user/system CPU percentages followed by context switches then major and minor page faults.
cpu : usr=0.01%, sys=0.21%, ctx=5686, majf=0, minf=8234
-
IO depths
:- This is the distribution of I/O depths over the job lifetime.
1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
-
1=0.1%
means0.1%
of IOs issued1
I/O request at a time. -
2=0.2%
means0.2%
of IOs issued2
I/O requests at a time. -
8=99.3%
means99.3%
of IOs issued8
I/O requests at a time. -
"4="
covers depths from4
to7
-
submit
:- Number of submitted IOs at a time.
- Each entry denotes that amount and below.
-
4=100%
means that we submitted anywhere between1
to4
I/O at a time.
-
complete
:- Like the above submit number, but for completions instead.
-
issued
:- The number of read/write/trim requests issued, and how many of them were short or dropped.
-
Run status group
:- Stats for all jobs.
READ: io=4096.0MB, aggrb=135418KB/s, minb=135418KB/s, maxb=135418KB/s, mint=30973msec, maxt=30973msec
-
io
= Number of I/O performed -
aggrb
= Aggregate bandwidth of threads. -
minb
= Minimum average bandwidth of threads. -
maxb
= Maximum average bandwidth of threads. -
mint
= Shortest runtime of threads in the group. -
maxt
= Longest runtime of threads in the group.
-
Disk stats
:nvme0n1: ios=16353/12, merge=0/2, ticks=1694705/756, in_queue=1695460, util=99.69%
-
I/O
= number of I/O performed by all group. -
merge
= number of merges in the I/O scheduler. -
ticks
= number of ticks we kept the disk busy. -
io_queue
= total time spent on disk queue. -
util
= disk utilization
-
Comments
0 comments
Please sign in to leave a comment.