Stuck at step "Building the hashes" for 10 days

[Feanor-Silmarillion]

Description

Hi authors,

I encountered a problem when running the pipeline. The process gets stuck at step 3 (Building the hashes) for over 10 days, and it seems no error is reported.

Reproduction

• I followed the instructions at: https://github.com/Gibbons-Lab/medi
• My command was run with this parameter:
  The log shows:
  executor > local (2)
  [d3/59982b] build_kraken_db (1) | 1 of 1 ✔
  [b8/8f2114] self_classify (1) | 0 of 1
  [- ] build_bracken -
  [- ] add_info

It seems to hang at self_classify and never proceeds. The size of the sequences is not huge.

My Question

• Is this normal?
• Roughly how long does the self_classify step take when --max_db_size=10?
• Any suggestions for debugging or speeding it up?

Thanks a lot!

Additional Info

• HPC node: CentOS 7, 20 threads, 500 GB memory
• Log file: attached or available upon request

[cdiener]

Hi, there is some discussion right now in #24 . The self classify step needs a lot of RAM. It seems to be some odd issue when using memory I/O and large inputs in Kraken2. For instance, for the latest db we built I used 860GB of RAM for this step and 12 CPUs. It completed in under 10h this way.