Skip to main content

Running and understanding write workload using FIO

Yogesh Jadhav
Updated

 

One of the challenges of testing the performance of your storage system or file system is to find a tool that can generate realistic and flexible read and write workloads. FIO is such a tool that can create various scenarios of write operations and report various statistics such as bandwidth, latency, and IOPS.

This article describes how to use FIO to test read and write workloads on your storage system or file system and below are few command like options you can use with FIO and their meanings.

 

 

Arguments Meaning
--name=str FIO will create a file with specified name to run the test on it.
--ioengine=str Defines how the job issues I/O to the test file.
- libaio - Linux native asynchronous block level I/O
- solarisaio - Solaris native asynchronous block level I/O
- windowsaio - Windows native asynchronous block level I/O
--size The size of the file on which the FIO will run the benchmarking test.
--rw=str Specifies the type of I/O pattern.
- read -  sequential read
- write - sequential write
- randread - random reads
- randwrite -  random writes
- rw -  sequential mix of read and writes
- randrw -  random mix of read and writes
--rwmixread=int Specifies read/write distribution ration.
rwmixread=30 would mean that 30% of I/O will be reads and 70% will be writes.
--bs=int Defines the block size that the test will be using for generating I/O. Default 4k
--direct=bool Decides wheather to use buffered I/O or Direct I/O
- 1 means true and it will use the direct I/O
- 0 means false and it will use the buffered I/O
--numjobs=int The number of threads spawned be the test.
--iodepth=int Controls how many I/O requests it issues to the OS at any given time. A sequential job with --iodepth=2 will submit two sequential IO requests at a time.
--runtime=int The amount of time the test will be running in seconds.
--time_based If specified, run for the specified duration even if the files are completely read of written. This same workload will be repeated as many times as runtime is allowed.
--startdelay

Add a delay in seconds between the intial test file created and the actual test. We can use some delay to avoid reading from write cache.

 

 

Example: Read Workload

 

  • Running fiotest with libaio engine and performing sequential read with direct I/O
  • The Size of the file is 1Gb
  • Number of jobs is 4 and iodepth is 8

     

    fio --name=fiotest --ioengine=libaio --size 1Gb --rw=read --bs=1M --direct=1 --numjobs=4 --iodepth=8 --runtime=60 --startdelay=60 --group_reporting
    Starting 4 processes
    fiotest: Laying out IO file(s) (1 file(s) / 1024MB) <-- 1
    fiotest: Laying out IO file(s) (1 file(s) / 1024MB) <-- 1
    fiotest: Laying out IO file(s) (1 file(s) / 1024MB) <-- 1
    fiotest: Laying out IO file(s) (1 file(s) / 1024MB) <-- 1
    Jobs: 3 (f=3): [R(2),_(1),R(1)] [98.9% done] [129.0MB/0KB/0KB /s] [129/0/0 iops] [eta 00m:01s]
    fiotest: (groupid=0, jobs=4): err= 0: pid=21786: Mon Aug 22 11:32:58 2022
    read : io=4096.0MB, bw=135418KB/s, iops=132, runt= 30973msec <-- 2
    slat (usec): min=37, max=146352, avg=29015.91, stdev=26314.40 <-- 3
    clat (msec): min=9, max=412, avg=207.45, stdev=57.88 <-- 4
    lat (msec): min=10, max=491, avg=236.47, stdev=65.97 <-- 5
    clat percentiles (msec):
    | 1.00th=[ 18], 5.00th=[ 115], 10.00th=[ 116], 20.00th=[ 165],
    | 30.00th=[ 212], 40.00th=[ 223], 50.00th=[ 223], 60.00th=[ 225],
    | 70.00th=[ 233], 80.00th=[ 235], 90.00th=[ 235], 95.00th=[ 277],
    | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 363], 99.95th=[ 367],
    | 99.99th=[ 412]
    lat (msec) : 10=0.02%, 20=2.95%, 50=0.20%, 100=1.25%, 250=90.14% <-- 6
    lat (msec) : 500=5.44% <-- 6
    cpu : usr=0.01%, sys=0.21%, ctx=5686, majf=0, minf=8234 <-- 7
    IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0% <-- 8
    submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% <-- 9
    complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% <-- 10
    issued : total=r=4096/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 <-- 11
    latency : target=0, window=0, percentile=100.00%, depth=8

    Run status group 0 (all jobs):
    READ: io=4096.0MB, aggrb=135418KB/s, minb=135418KB/s, maxb=135418KB/s, mint=30973msec, maxt=30973msec <-- 12

    Disk stats (read/write):
    nvme0n1: ios=16353/12, merge=0/2, ticks=1694705/756, in_queue=1695460, util=99.69% <-- 13

     

    Example 3: Write Workload

    fio --name=fiotest --ioengine=libaio --size 1Gb --rw=write --bs=1M --direct=1 --numjobs=4 --iodepth=8 --runtime=60 --startdelay=60 --group_reporting
    fiotest: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=8
    ...
    fio-2.14
    Starting 4 processes
    fiotest: Laying out IO file(s) (1 file(s) / 1024MB)
    fiotest: Laying out IO file(s) (1 file(s) / 1024MB)
    fiotest: Laying out IO file(s) (1 file(s) / 1024MB)
    fiotest: Laying out IO file(s) (1 file(s) / 1024MB)
    Jobs: 3 (f=3): [_(1),W(3)] [97.8% done] [0KB/129.0MB/0KB /s] [0/129/0 iops] [eta 00m:02s]
    fiotest: (groupid=0, jobs=4): err= 0: pid=1654: Wed Aug 24 04:27:55 2022
    write: io=4096.0MB, bw=135422KB/s, iops=132, runt= 30972msec
    slat (usec): min=44, max=259568, avg=28738.66, stdev=24563.74
    clat (msec): min=4, max=608, avg=207.46, stdev=72.17
    lat (msec): min=4, max=633, avg=236.20, stdev=80.91
    clat percentiles (msec):
    | 1.00th=[ 38], 5.00th=[ 63], 10.00th=[ 115], 20.00th=[ 169],
    | 30.00th=[ 188], 40.00th=[ 210], 50.00th=[ 212], 60.00th=[ 217],
    | 70.00th=[ 235], 80.00th=[ 235], 90.00th=[ 293], 95.00th=[ 338],
    | 99.00th=[ 392], 99.50th=[ 445], 99.90th=[ 545], 99.95th=[ 562],
    | 99.99th=[ 611]
    lat (msec) : 10=0.07%, 20=0.12%, 50=3.27%, 100=3.44%, 250=76.17%
    lat (msec) : 500=16.75%, 750=0.17%
    cpu : usr=0.23%, sys=0.17%, ctx=5730, majf=0, minf=44
    IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
    submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    issued : total=r=0/w=4096/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
    latency : target=0, window=0, percentile=100.00%, depth=8

    Run status group 0 (all jobs):
    WRITE: io=4096.0MB, aggrb=135422KB/s, minb=135422KB/s, maxb=135422KB/s, mint=30972msec, maxt=30972msec

    Disk stats (read/write):
    nvme0n1: ios=0/16358, merge=0/0, ticks=0/1389281, in_queue=1389282, util=99.66%

    Interpreting the FIO output:

    In the Read Workload section refer to the <-- prefixes which correspond to the line items below

    1. This will create 4 files with 1Gb of size.

      • fiotest.0.0, fiotest.1.0, fiotest.2.0, fiotest.3.0

    2. This following line is important

      • read : io=4096.0MB, bw=135418KB/s, iops=132, runt= 30973msec
      • FIO did 8GB I/O at 135MB/s for a total 132 iops and ran for 31 seconds

    3. slat is submission latency.

      • Basically means, How long it take to submit this I/O to the kernel for processing.
      • slat (usec): min=37, max=146352, avg=29015.91, stdev=26314.40
      • usec means microseconds.

    4. clat is completion latency.

      • This is the time between submission to kernel and time when I/O is completed.
      • clat (msec): min=9, max=412, avg=207.45, stdev=57.88
      • msec means milliseconds

    5. lat is total latency.

      • This is the time between I/O creation and its completion. This is the sum of slat and clat.
      • lat (msec): min=10, max=491, avg=236.47, stdev=65.97

    6. lat distribution.

      • This is distribution of I/O completion latencies. This is the time from when I/O leaves FIO and when it gets completed.
      • 10=0.02% means 0.02% of the I/O completed under 10 milliseconds.
      • 20=2.95% means 2.95% of the I/O completed under 20 milliseconds.
      • 250=90.14% means 90.14% of the I/O completed under 250 milliseconds.

    7. cpu

      • This is user/system CPU percentages followed by context switches then major and minor page faults.
      • cpu : usr=0.01%, sys=0.21%, ctx=5686, majf=0, minf=8234

    1. IO depths:

      • This is the distribution of I/O depths over the job lifetime.
      • 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
      • 1=0.1% means 0.1% of IOs issued 1 I/O request at a time.
      • 2=0.2% means 0.2% of IOs issued 2 I/O requests at a time.
      • 8=99.3% means 99.3% of IOs issued 8 I/O requests at a time.
      • "4=" covers depths from 4 to 7

    2. submit:

      • Number of submitted IOs at a time.
      • Each entry denotes that amount and below.
      • 4=100% means that we submitted anywhere between 1 to 4 I/O at a time.

    3. complete:

      • Like the above submit number, but for completions instead.

    4. issued:

      • The number of read/write/trim requests issued, and how many of them were short or dropped.

    5. Run status group:

      • Stats for all jobs.
      • READ: io=4096.0MB, aggrb=135418KB/s, minb=135418KB/s, maxb=135418KB/s, mint=30973msec, maxt=30973msec
      • io = Number of I/O performed
      • aggrb = Aggregate bandwidth of threads.
      • minb = Minimum average bandwidth of threads.
      • maxb = Maximum average bandwidth of threads.
      • mint = Shortest runtime of threads in the group.
      • maxt = Longest runtime of threads in the group.

    6. Disk stats:

      • nvme0n1: ios=16353/12, merge=0/2, ticks=1694705/756, in_queue=1695460, util=99.69%
      • I/O = number of I/O performed by all group.
      • merge = number of merges in the I/O scheduler.
      • ticks = number of ticks we kept the disk busy.
      • io_queue = total time spent on disk queue.
      • util = disk utilization

     

Related articles

Comments

0 comments

Please sign in to leave a comment.

Powered by Zendesk