Build suggestions etc

[miraep8]

Hello!

I (believe) I just successfully built the medi database (after a lot of attempts on my part!) Since I know you said it would be useful to have some job build parameters, I thought I would pass along the params I used for the build, as well as some other tweaks I made to improve the build process:

Downloading:

• 72:00 requested
• 50 cores
• 8 GB per core (so 400 in total)

Building:

• 120:00 time requested
• 60 cores
• 32 GB per core (1920 in total)

No claims that the above are the absolute minimum necessary required for this build! (but they were sufficient - the memory in the second bit in particular was problematic).

In addition I ran into some issues using the conda version of kraken recommended in the env setup. (specifically it would hang at the self_classify stage). I found that manually installing kraken (and of course bracken) into the env worked better than relying on the conda version.

I also think it is possible to speed things up by parallelizing the self_classify step!

Lastly, just wanted to point out that (as of right now) its essential that you build the medi db in the medi repo (this is because the publishDir in database.nf switches back and forth between "$baseDir/data/" and "${launchDir}/data/" so unless baseDir = launchDir, sad times. Would be an easy fix though!

Best + excited to get started trying out Medi! 😃

Mirae

[cdiener]

Hi Mirae,

Good catch with the threads in self_classify. I had this in there for debugging and never updated it. I will add it. The baseDir/launchDir issue should be fixed by now.

[robbueck]

I'm currently running build_kraken.nf with 550G (and --max_db_size=500) on 29 cores. It took almost 3 days to reach the end of add_sequences. Now it took 2 days for build_kraken_db and currently the self_classify step already is running for 2 days.
The database directory is currently at 2.6 TB.
Are those reasonable numbers?

[DEH114]

I would like an answer for this as well. I cannot tell if the self_classify step is hung up or if it is just a large step.

I am running on 1TB of memory with 25 cores.

[cdiener]

Hi, sorry about that. I am currently overhauling the database building workflow. Unfortunately, NCBI has changed some things (download links, dropped the kingdom rank in the NCBI taxonomy) which create some problems for Kraken2/Bracken. So right now it requires some manual patching of Kraken2 which is why did not release it yet.

Regarding the self classification, this step is really memory hungry and unfortunately it is a bit unclear how much memory it will need because it also depends on the HPC setup a lot, specifically on whether memory caching of files will be included in the limits. The most recent build we did requested ~900GB of RAM but actually only used a peak RSS of 614GB and peak Vmem of 645GB. It ran in under 10h using 12 cores.

I can also provide you with a download for the db version from the paper. Just shoot me a mail to lab (at) cdiener.com and provide me with a Gmail address so I can share the file with you.

Here the full statistics:

submit

2025-05-02 15:25:56

Timestamp when the task has been submitted
start

2025-05-02 15:26:25

Timestamp when the task execution has started
complete

2025-05-03 00:53:32

Timestamp when task execution has completed
duration

9 h 27 m 36 s

Time elapsed to complete since the submission i.e. including scheduling time
realtime

9 h 26 m 42 s

Task script execution time
Resources requested
Label Value Description
container

Container image name used to execute the task
queue

The queue that the executor attempted to run the process on
cpus

12

The cpus number request for the task execution
memory

867.4 GB

The memory request for the task execution
disk

The disk space request for the task execution
time

5 d

The time request for the task execution
executor

slurm

The Nextflow executor used to carry out this task
machineType

The virtual machine type used to carry out by this task
cloudZone

The cloud zone where the job get executed
priceModel

The price model used to charge the job computation
cost

The estimated cost to compute this task
Resources usage
Label Value Description
pcpu

501.4

Percentage of CPU used by the process
rss

613.5 GB

Real memory (resident set) size of the process
peakRss

613.7 GB

Peak of real memory
vmem

636.0 GB

Virtual memory size of the process
peakVmem

644.2 GB

Peak of virtual memory
rchar

787.7 GB

Number of bytes the process read, using any read-like system call from files, pipes, tty, etc
wchar

268.5 GB

Number of bytes the process wrote, using any write-like system call.
readBytes

1.2 TB

Number of bytes the process directly read from disk
writeBytes

268.5 GB

Number of bytes the process originally dirtied in the page-cache (assuming they will go to disk later).
syscr

63353659

Number of read-like system call invocations that the process performed
syscw

212549

Number of write-like system call invocations that the process performed
volCtxt

425,661

Number of voluntary context switches
invCtxt

20,242

Number of involuntary context switches