Skip to content

CI: run ztest on compressed zpool #17501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

behlendorf
Copy link
Contributor

@behlendorf behlendorf commented Jul 1, 2025

Motivation and Context

Keep zloop CI tests green.

Description

When running ztest under the CI a common failure mode is for the underlying filesystem to run out of available free space. Since the storage associated with a GitHub-hosted running is fixed, we instead create a pool and use a compressed ZFS dataset to store the ztest vdev files. This significantly increases the available capacity since the data written by ztest is highly compressible. A compression ratio of over 40:1 is conservatively achieved using the default lz4 compression. Autotrimming is enabled to ensure freed blocks are discarded from the backing cipool vdev file.

Lastly, the default zloop run time is increased to an hour or up to 12 iterations, and each individual ztest run is increased to 5 minutes.

How Has This Been Tested?

Testing in my fork here: https://github.com/behlendorf/zfs/actions/runs/15988074405/job/45096145143

Additionally, overnight local testing of zloop showed several runs that created over 80G of vdev files. After compression it required less than 2G on disk to these file vdevs.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Performance enhancement (non-breaking change which improves efficiency)
  • Code cleanup (non-breaking change which makes code smaller or more readable)
  • Quality assurance (non-breaking change which makes the code more robust against bugs)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
  • Documentation (a change to man pages or other documentation)

Checklist:

@behlendorf behlendorf added Component: Test Suite Indicates an issue with the test framework or a test case Status: Code Review Needed Ready for review and testing labels Jul 1, 2025
@behlendorf behlendorf force-pushed the zloop-ci branch 2 times, most recently from 2704c4e to ddc126a Compare July 2, 2025 16:36
@behlendorf
Copy link
Contributor Author

Dropped increased run time until some additional ztest fixes are merged.

@behlendorf
Copy link
Contributor Author

@mcmilk would you mind taking a look at this.

@mcmilk
Copy link
Contributor

mcmilk commented Jul 2, 2025

We should use the provided scratch-space of the runners for testings.
This space is bigger and faster then rootfs @ /var/tmp

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        72G   48G   25G  66% /
tmpfs           7.9G   84K  7.9G   1% /dev/shm
tmpfs           3.2G  1.2M  3.2G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdb16      881M   60M  760M   8% /boot
/dev/sdb15      105M  6.2M   99M   6% /boot/efi
/dev/sda1        74G  4.1G   66G   6% /mnt
tmpfs           1.6G   12K  1.6G   1% /run/user/1001

root@pkrvmbietmlfzoi:~# hdparm -t /dev/sda1
/dev/sda1:
 Timing buffered disk reads: 1286 MB in  3.04 seconds = 422.88 MB/sec

root@pkrvmbietmlfzoi:~# hdparm -t /dev/sdb
/dev/sdb:
 Timing buffered disk reads: 236 MB in  3.00 seconds =  78.62 MB/sec

So instead of this:

truncate -s 256G /var/tmp/vdev
sudo zpool create cipool -m $WORK_DIR -O compression=on -o autotrim=on /var/tmp/vdev

This variant would be on the fast disk:

truncate -s 256G /mnt/vdev
sudo zpool create cipool -m $WORK_DIR -O compression=on -o autotrim=on /mnt/vdev

This variant will remove the extra ext4 layer (untested, maybe I forgot sth.):

# remove default swapfile and /mnt
sudo swapoff -a
sudo umount -l /mnt
DISK="/dev/disk/cloud/azure_resource-part1"
sudo sed -e "s|^$DISK.*||g" -i /etc/fstab
sudo wipefs -aq $DISK
sudo systemctl daemon-reload

# partition the disk as needed
DISK="/dev/disk/cloud/azure_resource"
sudo sgdisk --zap-all $DISK
sudo sgdisk -p \
 -n 1:0:+16G -c 1:"swap" \
 -n 2:0:0    -c 2:"tests" \
$DISK
sync
sleep 1

# swap with same size as RAM (16GiB)
sudo mkswap $DISK-part1 && sudo swapon $DISK-part1
sudo zpool create cipool -m $WORK_DIR -O compression=on -o autotrim=on $DISK-part1

When running ztest under the CI a common failure mode is for the
underlying filesystem to run out of available free space.  Since
the storage associated with a GitHub-hosted running is fixed, we
instead create a pool and use a compressed ZFS dataset to store
the ztest vdev files.  This significantly increases the available
capacity since the data written by ztest is highly compressible.
A compression ratio of over 40:1 is conservatively achieved using
the default lz4 compression.  Autotrimming is enabled to ensure
freed blocks are discarded from the backing cipool vdev file.

Signed-off-by: Brian Behlendorf <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Test Suite Indicates an issue with the test framework or a test case Status: Code Review Needed Ready for review and testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants