download genome accession error: Error: [gateway], net/http: TLS handshake timeout

[owenfhale]

Thanks for developing this tool!
I've been using it for a while and it's been great, but recently I've run into errors when trying to download data.
I'm using version 18.3.1.
I'm working on my university's HPC trying to download some genomes using the following command:

datasets download genome \
    accession \
    --inputfile input_data/accessions.txt \
    --filename input_data/genomes.zip \
    --include genome,rna,protein,cds,gff3,gtf,gbff \
    --dehydrated \
    --debug

I keep getting this error:

Error: [gateway] Post "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report": POST https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report giving up after 1 attempt(s): Post "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report": net/http: TLS handshake timeout

I looked at some similar opened issues and it seems like sometimes the size of the request can be the problem, but I still get the error if I try a dehydrated run with a single accession.

Any suggestions on how to get this to work?

Here's the --debug output for the run with all the accessions:

2025/06/25 10:20:52 
POST /datasets/v2/genome/dataset_report HTTP/1.1
Host: api.ncbi.nlm.nih.gov
User-Agent: OpenAPI-Generator/1.0.0/go
Content-Length: 2368
Accept: application/json
Content-Type: application/json
Ncbi-Phid: 7C332F20F4358C7C74A8C509
X-Datasets-Client: datasets-cli
X-Datasets-Client-Arch: amd64
X-Datasets-Client-Cmd: download genome accession --inputfile input_data/accessions.txt --filename input_data/genomes.zip --include genome,rna,protein,cds,gff3,gtf,gbff --dehydrated --debug
X-Datasets-Client-Os: linux
X-Datasets-Client-Version: 18.3.1
Accept-Encoding: gzip

{"accessions":["GCF_000005845.2","GCF_000008865.2","GCF_002853715.1","GCF_000210475.1","GCF_041879485.1","GCF_003019295.1","GCF_031191795.1","GCF_037889275.1","GCF_902381645.1","GCF_037951365.1","GCF_000007985.2","GCF_019904315.1","GCF_024205665.1","GCF_003574175.1","GCF_000210155.1","GCF_001190745.1","GCF_020736185.1","GCF_000009085.1","GCF_017357025.1","GCF_037680265.1","GCF_003030185.1","GCF_046529325.1","GCF_034298135.1","GCF_032917925.1","GCF_033569435.1","GCF_000011985.1","GCF_000389675.2","GCF_008245085.1","GCF_000007305.1","GCF_000275605.1","GCF_025998455.1","GCF_017821355.1","GCF_016757555.1","GCF_022925795.1","GCF_022921295.1","GCF_022374915.2","GCF_020150655.1","GCF_900638745.1","GCF_000013925.1","GCF_020616615.1","GCF_000144405.1","GCF_018128045.1","GCF_020735905.1","GCF_002208725.2","GCF_003609775.1","GCF_000025985.1","GCF_000965785.1","GCF_000724805.2","GCF_023702735.1","GCF_040142445.1","GCF_000015485.1","GCF_000195755.1","GCF_027947635.1","GCF_000166115.1","GCF_902167245.1","GCF_044361425.1","GCF_025147485.1","GCF_018289175.1","GCF_024760085.1","GCF_046603925.1","GCF_000012825.1","GCF_024759565.1","GCF_036419465.1","GCF_020091465.1","GCF_008728395.1","GCF_001314995.1","GCF_918258265.1","GCF_020091525.1","GCF_024759225.1","GCF_024759785.1","GCF_031172035.1","GCF_018291945.1","GCF_020091385.1","GCF_020091345.1","GCF_018292125.1","GCF_001318345.1","GCF_020091405.1","GCF_025263845.1","GCF_025146565.1","GCF_019930665.1","GCF_018292205.1","GCF_016726305.1","GCF_002222615.2","GCF_025146315.1","GCF_016889745.1","GCF_013009555.1","GCF_018292105.1","GCF_013009875.1","GCF_013010095.1","GCF_001640865.1"],"filters":{"assembly_source":"all","assembly_version":"all_assemblies","exclude_atypical":false,"exclude_multi_isolate":false,"has_annotation":false,"is_metagenome_derived":"METAGENOME_DERIVED_UNSET","is_type_material":false,"reference_only":false,"search_text":[],"type_material_category":"NONE"},"include_tabular_header":"INCLUDE_TABULAR_HEADER_FIRST_PAGE_ONLY","page_size":1,"returned_content":"ASSM_ACC","sort":[{"direction":"SORT_DIRECTION_ASCENDING","field":"organismName"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRepGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefseq"},{"direction":"SORT_DIRECTION_ASCENDING","field":"accession"}]}

Error: [gateway] Post "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report": POST https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report giving up after 1 attempt(s): Post "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report": net/http: TLS handshake timeout

Here's the --debug output for the run with a single accession:

2025/06/25 10:29:47 
POST /datasets/v2/genome/dataset_report HTTP/1.1
Host: api.ncbi.nlm.nih.gov
User-Agent: OpenAPI-Generator/1.0.0/go
Content-Length: 757
Accept: application/json
Content-Type: application/json
Ncbi-Phid: 91084F7ED7114536C61A9E8D
X-Datasets-Client: datasets-cli
X-Datasets-Client-Arch: amd64
X-Datasets-Client-Cmd: download genome accession GCF_000005845.2 --assembly-version latest --filename input_data/GCF_000005845.2.zip --dehydrated --debug
X-Datasets-Client-Os: linux
X-Datasets-Client-Version: 18.3.1
Accept-Encoding: gzip

{"accessions":["GCF_000005845"],"filters":{"assembly_source":"all","assembly_version":"current","exclude_atypical":false,"exclude_multi_isolate":false,"has_annotation":false,"is_metagenome_derived":"METAGENOME_DERIVED_UNSET","is_type_material":false,"reference_only":false,"search_text":[],"type_material_category":"NONE"},"include_tabular_header":"INCLUDE_TABULAR_HEADER_FIRST_PAGE_ONLY","page_size":1,"returned_content":"ASSM_ACC","sort":[{"direction":"SORT_DIRECTION_ASCENDING","field":"organismName"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRepGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefseq"},{"direction":"SORT_DIRECTION_ASCENDING","field":"accession"}]}

Error: [gateway] Post "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report": POST https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report giving up after 1 attempt(s): Post "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report": net/http: TLS handshake timeout

[ericcox1]

Hi @owenfhale,

Thank you for opening this issue. I have not been able to reproduce this problem but we will try to investigate soon using the debug output that you provided and I will share any updates as they become available.

Best,
Eric

[ericcox1]

Hi @owenfhale,

We want to try to rule out a firewall or connection issue on your end. Could you please try the following curl command and report what you see?

curl -X POST "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report"  -H 'accept: application/json' -H 'content-type: application/json'  -d '{"accessions":["GCF_000005845"],"returned_content":"ASSM_ACC"}'

Best,
Eric

[owenfhale]

Thanks for the quick reply Eric!

Here's what I get:

{"reports":[{"accession":"GCF_000005845.2","source_database":"SOURCE_DATABASE_REFSEQ"}],"content_type":"ASSM_ACC","total_count":1}

Also, I'm not sure it matters but I installed the tool with conda. Here's the environment info:

# Name                    Version                   Build  Channel
ca-certificates           2025.1.31            hbcca054_0    conda-forge
ncbi-datasets-cli         18.3.1               ha770c72_0    conda-forge

[ericcox1]

Thanks @owenfhale, could you please try the datasets command with the single accession one more time and post the debug output here?

[owenfhale]

I re-ran this:

datasets download genome accession GCF_000005845.2 --assembly-version latest --filename input_data/GCF_000005845.2.zip --dehydrated --debug

and got this:

2025/06/25 13:40:35 
POST /datasets/v2/genome/dataset_report HTTP/1.1
Host: api.ncbi.nlm.nih.gov
User-Agent: OpenAPI-Generator/1.0.0/go
Content-Length: 757
Accept: application/json
Content-Type: application/json
Ncbi-Phid: 2627F5D855F124ECE868B91F
X-Datasets-Client: datasets-cli
X-Datasets-Client-Arch: amd64
X-Datasets-Client-Cmd: download genome accession GCF_000005845.2 --assembly-version latest --filename input_data/GCF_000005845.2.zip --dehydrated --debug
X-Datasets-Client-Os: linux
X-Datasets-Client-Version: 18.3.1
Accept-Encoding: gzip

{"accessions":["GCF_000005845"],"filters":{"assembly_source":"all","assembly_version":"current","exclude_atypical":false,"exclude_multi_isolate":false,"has_annotation":false,"is_metagenome_derived":"METAGENOME_DERIVED_UNSET","is_type_material":false,"reference_only":false,"search_text":[],"type_material_category":"NONE"},"include_tabular_header":"INCLUDE_TABULAR_HEADER_FIRST_PAGE_ONLY","page_size":1,"returned_content":"ASSM_ACC","sort":[{"direction":"SORT_DIRECTION_ASCENDING","field":"organismName"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRepGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefseq"},{"direction":"SORT_DIRECTION_ASCENDING","field":"accession"}]}

Error: [gateway] Post "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report": POST https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report giving up after 1 attempt(s): Post "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report": net/http: TLS handshake timeout

[ericcox1]

Thanks. What about datasets --version?

[owenfhale]

datasets version: 18.3.1