Skip to content

20251023 steady state add latency #2003

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

20251023 steady state add latency #2003

wants to merge 6 commits into from

Conversation

@mcgrof
Copy link

@mcgrof mcgrof commented Oct 23, 2025

This adds fio steady state latency support.

@vincentkfu
Copy link
Collaborator

vincentkfu commented Oct 23, 2025

Seems like some rebasing went awry.

@mcgrof mcgrof force-pushed the 20251023-steady-state-add-latency branch 2 times, most recently from 6604c32 to 61894dd Compare October 23, 2025 23:47
@mcgrof
Copy link
Author

mcgrof commented Oct 23, 2025

Seems like some rebasing went awry.

yikes yes, sorry about that. Fixed and and pushed.

vincentkfu
vincentkfu reviewed Oct 28, 2025
View reviewed changes
mcgrof and others added 6 commits October 30, 2025 09:54
@mcgrof
configure: conditionally add gnutls for libnfs >= 6.0.0
9a225e0 
gnutls is required newwer version of libnfs, and so we need to check
for this. libnfs version 6.0.0 and later added TLS support. Earlier versions
like 5.0.x do not require gnutls.

Add a version check in the configure script to detect the libnfs version
and only add gnutls as a dependency when libnfs >= 6.0.0 is detected.

For systems with older libnfs versions, the build will work without
requiring gnutls. For systems with libnfs 6.0.0+, gnutls will be
properly detected and linked.

Generated-by: Claude AI
Reviewed-by: Vincent Fu <[email protected]>
Signed-off-by: Luis Chamberlain <[email protected]>
@mcgrof
fio: refactor steady state validation check
3353f8d 
Extract the long if condition in str_steadystate_cb() into a separate
helper function is_valid_steadystate() for better readability and
maintainability.

Generated-by: Claude AI
Reviewed-by: Vincent Fu <[email protected]>
Signed-off-by: Luis Chamberlain <[email protected]>
@mcgrof
fio: refactor duplicate code in steadystate_*_mean functions
81a80cb 
Extract the common logic from steadystate_bw_mean() and steadystate_iops_mean()
into a helper function steadystate_data_mean() to reduce code duplication and
improve maintainability.

Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <[email protected]>
@mcgrof
fio: add latency steady state detection
e4275de 
Add fio latency steady state support. The implementation calculates
weighted average latency across all I/O directions and supports both
maximum mean deviation and slope-based detection methods.

Tested successfully against NVMe device with debug output confirming
proper latency calculation and steady state evaluation.

Quick demo with the null engine:

fio --name=test --ioengine=null --size=1G --rw=randread --bs=4k --iodepth=64 --steadystate=lat:100us --ss_dur=10s --runtime=120s --time_based=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=64
fio-3.41-20-gf2b2e-dirty
Starting 1 process
Jobs: 1 (f=1): [f(1)][100.0%][r=15.6GiB/s][r=4078k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1207074: Tue Oct  7 11:47:27 2025
  read: IOPS=4054k, BW=15.5GiB/s (16.6GB/s)(153GiB/9899msec)
    slat (nsec): min=18, max=33347, avg=22.20, stdev=64.49
    clat (nsec): min=135, max=42283, avg=152.74, stdev=173.53
     lat (nsec): min=156, max=42306, avg=174.94, stdev=185.32
    clat percentiles (nsec):
     |  1.00th=[  139],  5.00th=[  141], 10.00th=[  141], 20.00th=[  143],
     | 30.00th=[  143], 40.00th=[  145], 50.00th=[  147], 60.00th=[  153],
     | 70.00th=[  155], 80.00th=[  157], 90.00th=[  163], 95.00th=[  171],
     | 99.00th=[  217], 99.50th=[  221], 99.90th=[  239], 99.95th=[  270],
     | 99.99th=[ 2096]
   bw (  MiB/s): min=15344, max=15944, per=100.00%, avg=15835.92, stdev=132.94, samples=19
   iops        : min=3928298, max=4081694, avg=4053996.63, stdev=34033.65, samples=19
  lat (nsec)   : 250=99.93%, 500=0.06%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=99.93%, sys=0.03%, ctx=88, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40129848,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
  steadystate  : attained=yes, bw=15.2GiB/s (15.9GB/s), iops=3976k, lat=152nsec, lat mean dev=2.700

Run status group 0 (all jobs):
   READ: bw=15.5GiB/s (16.6GB/s), 15.5GiB/s-15.5GiB/s (16.6GB/s-16.6GB/s), io=153GiB (164GB), run=9899-9899msec

Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <[email protected]>
@mcgrof
fio: add mock test framework for isolated unit testing
8afdaf9 
Introduce a mock test framework to validate specific algorithms and
calculations in isolation. This allows testing numerical precision,
edge cases, and algorithmic correctness without requiring the full
FIO infrastructure.

Features:
- TAP (Test Anything Protocol) output for CI integration
- Isolated testing of specific calculations
- Clear documentation of test purposes and goals
- Simple make target: 'make mock-tests'

The first test validates the numerical precision improvements in
steady state latency calculations, demonstrating how the framework
can catch subtle precision and overflow issues.

Structure:
- mock-tests/lib/tap.h: TAP output support
- mock-tests/tests/: Individual test programs
- mock-tests/Makefile: Build system
- mock-tests/README.md: Comprehensive documentation

This framework complements the existing integration tests by focusing
on unit-level validation of critical calculations.

Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <[email protected]>
@vincentkfu @mcgrof
mock-tests: assess per second latency recovery
3954bbb 
When fio has issued 884 billion IOs it is still aable to recover per
second latencies with less than 0.1% error using a latency value of
13ms. This is evidence that fio's strategy for estimating per second
latency for steady state detection works reasonably well.

Signed-off-by: Vincent Fu <[email protected]>
@mcgrof mcgrof force-pushed the 20251023-steady-state-add-latency branch from 61894dd to 3954bbb Compare October 30, 2025 16:54
vincentkfu
vincentkfu reviewed Oct 31, 2025
View reviewed changes
HOWTO.rst
**lat**
Collect completion latency data and calculate the maximum mean
deviation. Stop the job if the deviation falls below the specified
limit. The latency values are weighted by the number of I/O samples
Copy link
Collaborator

@vincentkfu vincentkfu Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what the last sentence about weighting means here or below. When the overall mean is calculated for the mean deviation criterion, the values are not weighted. Each period is given the same weight regardless of how many IOs completed during that period. I suggest removing the sentence about weighting.

vincentkfu
vincentkfu reviewed Oct 31, 2025
View reviewed changes
configure
libnfs_cflags="$libnfs_cflags $(pkg-config --cflags gnutls)"
libnfs_libs="$libnfs_libs $(pkg-config --libs gnutls)"
else
if test "$libnfs" = "yes" ; then
Copy link
Collaborator

@vincentkfu vincentkfu Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This if is unnecessary because libnfs is already set to "yes"

@vincentkfu
Copy link
Collaborator

vincentkfu commented Oct 31, 2025

There is still a bit more than needs to be added to get this working in client/server mode.

  1. Please bump FIO_SERVER_VER since struct thread_stat has changed
  2. In client.c:fio_handle_client(), the switch block for FIO_NET_CMD_TS has code to ingest steadystate BW and IOPS data. Similar code for latency should be added.
  3. Similarly, in server.c:fio_server_send_ts() is code to send steadystate BW and IOPS data. Code for sending latency data should be added. ss_extra_size needs to be 3 * ts->ss_dur... instead of 2 * ts->ss_dur...

vincentkfu
vincentkfu reviewed Oct 31, 2025
View reviewed changes
stat.c
free(p1);
free(p1alt);
free(p2);
if (p3)
Copy link
Collaborator

@vincentkfu vincentkfu Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could just get rid of the if since free(null) will just be a noop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vincentkfu vincentkfu vincentkfu left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mcgrof @vincentkfu