Missing Genome in matches.csv

[ktmbiome]

Hi MEDI developers,

Thank you for making such a cool tool! I was exploring it several months back and was able to generate a database and results without issue.

I am now rebuilding the database for various reasons, and it seems that one of the genomes, GCA_036512215.2_SLM_r2.1, was perhaps removed or modified recently. When I visit: https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/036/512/215/GCA_036512215.2_SLM_r2.1/, there are no genomic.fna.gz files, which causes an error (I found this by having the download fail and then visiting .command.log).

I wanted to let you know in case there's something you need to do for the database. For now, I'll remove it from matches.csv and see if that is acceptable to the Nextflow pipeline.

--Katie

[cdiener]

Seems similar to issue #11. Can you try with rerunning without -resume to get the matches from scratch? Usually this is caused if a genome was removed or not yet uploaded between the matching and downloading. It looks like the genome in question has been deposited now because in your link it does show the fna.gz now.

If that does not work you can also shoot me a mail at lab[at]cdiener.com and I can send you a link to download the database.

[ktmbiome]

Indeed it is back! I'll re-run that part now too... I have been doing a lot of investigating since I wrote last week, as I am actively having trouble with the other kraken databases for similar reasons (prokaryote, even!). I get errors for different genomes that are missing every time I run build_kraken.nf (no --resume option), because the assembly_summary that kraken downloads would still include missing genomes (that would disappear or reappear).

I've written to NCBI about whether there are maintenance windows that we can avoid, since the instability can be quite problematic at times, really for anyone trying to build a database.

[cdiener]

Yes I agree. In the initial version MEDI would just silently ignore genomes that couldn't be downloaded but the caveat was that it would build a different database every run which sometimes would miss an entire food item entirely. So that did not seem too reproducible either.

The best solution would be to provide the database for download but we haven't found a repository that allows you to deposit 500GB. Open for suggestions here.

But it is a priority for us and we are working on it.