10 x performance differences between 2.2.6 and 2.3.3

[jxdn]

System information

Type	Version/Name
Distribution Name	RHEL
Distribution Version	9.4
Kernel Version	5.14.0-362.8.1.el9_3.x86_64
Architecture	x86_64
OpenZFS Version	zfs-2.3.3-1

Describe the problem you're observing

I run 2 nodes : node 1 and node 2

node 1 :

zpool status
pool: data
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:

    NAME                                                        STATE     READ WRITE CKSUM
    data                                                        ONLINE       0     0     0
      raidz1-0                                                  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_v2_AGN_RI_U.2_7.68TB_S6CMNA0TC14131  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_v2_AGN_RI_U.2_7.68TB_S6CMNA0TC14133  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_v2_AGN_RI_U.2_7.68TB_S6CMNA0TC14135  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_v2_AGN_RI_U.2_7.68TB_S6CMNA0TC14137  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_v2_AGN_RI_U.2_7.68TB_S6CMNA0TC14182  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_v2_AGN_RI_U.2_7.68TB_S6CMNA0TC14184  ONLINE       0     0     0

node 2:

pool: data
state: ONLINE
config:

    NAME                                                   STATE     READ WRITE CKSUM
    data                                                   ONLINE       0     0     0
      raidz1-0                                             ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_CM7_U.2_RI_7.68TB_9E20A0UR04A3  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_CM7_U.2_RI_7.68TB_9E20A0US04A3  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_CM7_U.2_RI_7.68TB_9E20A0UT04A3  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_CM7_U.2_RI_7.68TB_9E20A0UU04A3  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_CM7_U.2_RI_7.68TB_9E20A0UW04A3  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_CM7_U.2_RI_7.68TB_9E20A0UX04A3  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_CM7_U.2_RI_7.68TB_9E20A0XU04A3  ONLINE       0     0     0
        nvme-Dell_Ent_NVMe_CM7_U.2_RI_7.68TB_9E20A0XV04A3  ONLINE       0     0     0

i run dd on first node:

dd if=testrand4.fio of=/dev/null bs=128k count=163840 iflag=direct
163840+0 records in
163840+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 5.06304 s, 4.2 GB/s

the run on second node (newest versions)

dd if=testrand4.fio of=/dev/null bs=128k count=163840 iflag=direct
163840+0 records in
163840+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 83.3492 s, 258 MB/s

Describe how to reproduce the problem

jjust install the difference version without setting anything

Include any warning/errors/backtraces from the system logs

[amotin]

ZFS 2.3 got support for Direct I/O, so iflag=direct is no longer meaningless. Since Direct I/O does not use speculative prefetch, it depends on how much data you read at once by your application (dd here). I suppose one request of 128KB is not enough to saturate RAIDZ1 of 8 devices, even if they are NVMe. You should test it without iflag=direct, or with setting direct=disabled dataset property.