Consider understanding the performance of a C program which writes blocks between 12 and 16 KiB to a few files sequentially within each file.
It produces the following iostat -x line1 from a one second interval for a disk under a fairly heavy load of ~16 KiB writes:
Device w/s wkB/s wrqm/s %wrqm w_await wareq-sz aqu-sz %util
nvme1n1 7497.00 105250.50 0.00 0.00 0.03 14.04 0.23 100.00
It shows 100% utilization and aqu-sz of 0.23. This is typical reading for this workload: the queue size generally hovers around 0.20 to 0.25 and the utilizatio is usually 100%, but for an occasional measurement interval somewhat less like 90% or 95%.
As I understand it, utilization is "the fraction of time at least 1 request was outstanding to the disk"2. So 100% means that at least 1 request was outstanding at all times. The aqu-sz or average queue size, as I understand it, is the time-averaged size of the requests outstanding queue, i.e., the average number of in-flight requests submitted to the block layer which haven't yet completed.
Something is wrong though: this reading is not consistent with these definitions. If there were at least 1 request outstanding at all times (implied by utilization 100%) then the average queue size must be at least 1.0, since it is never less than 1.0 at any moment.
So what's up with that?
1 I've cut out the read and discard stats since they are both ~0 for this run.
2 Unlike single-head disks, for HDDs with multiple independent heads and for modern SSDs which can both handle several outstanding requests, a utilization of 100% does not correspond to "saturated device".