You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
pdu performs about 2x worse on my HDD than single-threaded du. I'm testing on an old home directory of mine on a mechanical hard drive, with about 712 gigabytes of data in around 150,000 files. The size difference reported by the two programs is due to hard links.
$ echo 1 | sudo tee /proc/sys/vm/drop_caches
$ time pdu
...
765.0G ┌─┴.
pdu 0.69s user 2.93s system 4% cpu 1:18.21 total
Compared to du:
$ echo 1 | sudo tee /proc/sys/vm/drop_caches
$ time du -sh .
712G .
du -sh . 0.28s user 1.46s system 3% cpu 47.405 total
I'm not positive on the source of this difference, but I believe it's due to the directory traversal order used by the two programs. du uses a depth-first search whereas pdu seems to use breadth-first search through rayon, although I can't tell for sure. Interestingly, pdu is comparable to du when manually limited to a single thread:
$ echo 1 | sudo tee /proc/sys/vm/drop_caches
$ time RAYON_NUM_THREADS=1 pdu
...
765.0G ┌─┴.
RAYON_NUM_THREADS=1 pdu 0.46s user 1.87s system 5% cpu 46.078 total
The text was updated successfully, but these errors were encountered:
pdu was never really designed to run on HDD (I forgot to mention it in README.md). But if there's an easy way to detect HDD and limit rayon thread to 1, I'll be happy to accept a pull request. Unless multi-threaded du is still faster on HDD for some reason.
pdu
performs about 2x worse on my HDD than single-threadeddu
. I'm testing on an old home directory of mine on a mechanical hard drive, with about 712 gigabytes of data in around 150,000 files. The size difference reported by the two programs is due to hard links.Compared to du:
I'm not positive on the source of this difference, but I believe it's due to the directory traversal order used by the two programs.
du
uses a depth-first search whereaspdu
seems to use breadth-first search through rayon, although I can't tell for sure. Interestingly,pdu
is comparable todu
when manually limited to a single thread:The text was updated successfully, but these errors were encountered: