HDD performance is poor #257
Description
pdu performs about 2x worse on my HDD than single-threaded du. I'm testing on an old home directory of mine on a mechanical hard drive, with about 712 gigabytes of data in around 150,000 files. The size difference reported by the two programs is due to hard links.
$ echo 1 | sudo tee /proc/sys/vm/drop_caches
$ time pdu
...
765.0G ┌─┴.
pdu 0.69s user 2.93s system 4% cpu 1:18.21 total
Compared to du:
$ echo 1 | sudo tee /proc/sys/vm/drop_caches
$ time du -sh .
712G .
du -sh . 0.28s user 1.46s system 3% cpu 47.405 total
I'm not positive on the source of this difference, but I believe it's due to the directory traversal order used by the two programs. du uses a depth-first search whereas pdu seems to use breadth-first search through rayon, although I can't tell for sure. Interestingly, pdu is comparable to du when manually limited to a single thread:
$ echo 1 | sudo tee /proc/sys/vm/drop_caches
$ time RAYON_NUM_THREADS=1 pdu
...
765.0G ┌─┴.
RAYON_NUM_THREADS=1 pdu 0.46s user 1.87s system 5% cpu 46.078 total