Persistent error in database.nf

[luhugerth]

Since some other errors suggested it could be a hiccup in the server I've tried a couple of times, but the error is consistent.

I run

nextflow run /ceph/home/luhugerth/medi/database.nf --max-cores 32 --keep-going

and it goes on for a while until

INFO [2024-12-06 18:43:38] Downloaded genome for assembly GCA_034641835.2_WBP_1.0...    
  INFO [2024-12-06 18:49:22] Downloaded genome for assembly GCA_044510145.1_ASM4451014v1...
  INFO [2024-12-06 18:51:46] Downloaded genome for assembly GCA_958299785.1_drMedLupu1.1...    
  INFO [2024-12-06 18:54:09] Downloaded genome for assembly GCA_900312955.1_ASM90031295v1...
executor >  local (7)                                                                  
[21/c073c7] process > download               [100%] 1 of 1 ✔                               
[54/0c24a0] process > get_taxids             [100%] 1 of 1 ✔                               
[13/eb2062] process > download_taxa_dbs      [100%] 1 of 1 ✔                               
[9c/134f3e] process > get_lineage (1)        [100%] 1 of 1 ✔                                 
[b3/3ba2d8] process > match_taxids (1)       [100%] 1 of 1 ✔                               
[58/aef707] process > download_sequences (1) [100%] 1 of 1, failed: 1 ✘                    
[-        ] process > sketch                 -                                        
[-        ] process > ANI                    -                                             
[51/db5e1a] process > food_mappings (1)      [100%] 1 of 1 ✔                               
ERROR ~ Error executing process > 'download_sequences (1)'                                                                                                                                                   
                                                                                           
Caused by:                                                                                 
  Process `download_sequences (1)` terminated with an error exit status (1)                 

Which seems to be downstream of:

INFO [2024-12-06 18:54:09] Downloaded genome for assembly GCA_900312955.1_ASM90031295v1...
  INFO [2024-12-06 18:57:41] Downloaded genome for assembly GCA_038419695.1_ASM3841969v1...
  INFO [2024-12-06 18:59:22] Downloaded genome for assembly GCA_045280955.1_ASM4528095v1...
  INFO [2024-12-06 19:00:49] Downloaded genome for assembly GCA_040801865.1_FNU-M-1-Hap-a...
  Warning message:
  In parallel::mclapply(gb[, unique(id)], function(i) download_genome(gb[id ==  :
    scheduled core 5 encountered error in user code, all values of the job will be affected
  Error in rbindlist(dls) :
    Item 5 of input is not a data.frame, data.table or list
  Execution halted

Any idea what item 5 is and why is preventing me from building the DB?

[cdiener]

I suspect a similar issue as in #11 and #15. This is usually an issue with delayed NCBI updates where the sequence summary lists genomes that have either not been uploaded yet or have been removed. Usually waiting a day or two and rerunning the full pipeline fixes this.

In the meanwhile you can also send me a mail to lab[at]cdiener.com and I can send you a link to download the built database.

[tanaes]

I'm having a similar problem, with the exception that there's an additional line complaining about integer overflow:

 Warning message:
  In nchar(str, "bytes") * 4L : NAs produced by integer overflow
  INFO [2025-04-20 17:57:19] Downloading 368 genomes with 8 threads.
  Warning message:
  In parallel::mclapply(gb[, unique(id)], function(i) download_genome(gb[id ==  :
    all scheduled cores encountered errors in user code
  Error in rbindlist(dls) : 
    Item 1 of input is not a data.frame, data.table or list
  Execution halted

Does that strike you also as an NCBI-side error?