Use pbzip2/pigz for asset decompression

[brainstorm]

Description of feature

Currently asset decompression is limited to single core?:

[brainstorm]

Looking at the workflows, I'd pinpoint this bottleneck at oncoanalyser/modules/local/custom/extract_tarball, which is https://github.com/conda-forge/tar-feedstock ... I'd add parallel decompressors right in that feedstock (if reviewers/community agree). It can definitely shave off a lot of time from the following timeline and third party users of that particular tar feedstock:

[scwatts]

Thanks for the suggestion @brainstorm. Given a sufficiently well-resourced executor where each job can run independently I would expect to see a runtime improvement of upwards of ~5 minutes

This on the backdrop of full sized samples having a runtime in the order of hours so the relative improvement won't be hugely impactful. Additionally, users typically pre-stage reference data so decompression should generally only occur once

[brainstorm]

Healthy skepticisim for 4 cores (as plotted above), I agree, but assuming linear core scaling of pigz/bzip2 I would expect significantly better than that at higher core counts IF the executor manages to get those tasks overlapped (also not considering faster download speeds on higher core count instances)?

decompression should generally only occur once

That kind of decompression occurs MANY times ONCE if you are in CI environment and/or testing. That matters too if you aggregate runs like I've been dealing with recently for dev and test purposes! ;)

[scwatts]

Agreed, in that ideal scenario the improvement in decompression runtime would be at most ~10 minutes

Generally I would also recommend for dev/testing that you configure oncoanalyser to not continually retrieve and decompress reference data for every single test run