Ingest from INSDC

[huddlej]

Context

Avian influenza analyses ingest INSDC data. Additionally, the Oropouche ingest includes logic to group multiple segments of the same isolate.

GenSpectrum maintains curated INSDC data for seasonal influenza which might be a better starting point for our ingest.

Description

Seasonal influenza should also have an ingest workflow for INSDC data that will:

1. support all segments from H1N1pdm, H3N2, and Vic.
2. provide the standard metadata TSV per isolate and FASTA files per isolate and gene
3. provide Nextclade annotations and alignments

[joverlee521]

According to NCBI docs, they have a cached package for Alphainfluenza that can be downloaded via

datasets download virus genome taxon "Alphainfluenzavirus influenzae" --filename all_alphainfluenza.zip

No cached package for Betainfluenza since there's not as much data, but we can download via the taxon id

datasets download virus genome taxon 197912 --filename all_betainfluenza.zip

[joverlee521]

GenSpectrum already links the segments to a single record which makes it easier for us.

I briefly wondered if we could just skip ingest altogether and just start from the GenSpectrum API, but I think we still want ingest to pull in more strain names. We would also need ingest for custom annotations, e.g. we probably need strain name munging to link the records to titer data.

[joverlee521]

There are separate URLs for each subtype

• Influenza A/H1N1pdm: https://api.loculus.genspectrum.org/h1n1pdm
• Influenza A/H3N2: https://api.loculus.genspectrum.org/h3n2
• Influenza B/Victoria: https://api.loculus.genspectrum.org/b-victoria

but the endpoints for downloading metadata and sequences are the same.

Using BVic as an example. Download metadata as TSV:

curl -X 'GET' \
  'https://api.loculus.genspectrum.org/b-victoria/sample/details?dataFormat=TSV-ESCAPED&versionStatus=LATEST_VERSION' \
  -H 'accept: application/json' > /tmp/b-victoria-metadata.tsv

Download FASTA of unaligned sequences by segment (seg1 - seg8) with the GenSpectrum accession as the seq id:

curl -X 'GET' \
 'https://api.loculus.genspectrum.org/b-victoria/sample/unalignedNucleotideSequences/seg1?dataFormat=FASTA&fastaHeaderTemplate=%7Baccession%7D&versionStatus=LATEST_VERSION' \
  -H 'accept: application/json' > /tmp/b-victoria-seg1.fasta

[joverlee521]

Did some additional digging today into the GenSpectrum data:

• Since both GenSpectrum and PPX are using Loculus, GenSpectrum geolocations look similar to PPX geolocations so we will have to curate the locations. Maybe with similar solution as ingest: Fix PPX division rsv#100.
• There are some Nextclade columns (e.g. cladeHA and cladeNA), but I think we'd still want to run Nextclade in our own workflow to have control over which versions of Nextclade datasets we use (or test with unreleased versions).

[joverlee521]

Ah, GenSpectrum has the issue we originally saw in Pathoplexus: ambiguous dates default to the first of the month

GenBank accession	GenBank date	GenSpectrum accession	GenSpectrum date
CY006218	1968	GS_00BVC0L	1968-01-01
MF955471	Mar-1997	GS_00BLBWC	1997-03-01

[joverlee521]

Chatted about this work with @huddlej in the priorities meeting today. I'll start on a workflow that downloads metadata/sequences from GenSpectrum. Using the linked GenBank accessions, we can then join the metadata with some additional fields from Entrez (i.e. strain name and original collection date).

I did not dive into this yet, but I suspect passage info is also a custom field that we'll have to pull from Entrez.

[joverlee521]

I did not dive into this yet, but I suspect passage info is also a custom field that we'll have to pull from Entrez.

I am able to get some passage info from Entrez via the "note" field. This flags an issue in the GenSpectrum linkage of segments in that they are potentially linking segments that do not have the same passage history.

GenSpectrum accession	GenBank accession	segment	note
GS_00BKVR6	KY888350	HA	passage details: E1
	KY888377	NA	passage details: C2

[anna-parker]

Looking into the example above these two sequences are not grouped in an assembly so in Genspectrum (Loculus) they must have been grouped using the heuristic grouping algorithm.

I see that both of these samples have the same strain name (B/JiangxiAnyuan/1451/2015), collection date, authors and location. Strain name is often used as an isolate name so it does make sense that the heuristic grouping algorithm we use in Genspectrum would group these sequences.

Sadly the notes are not available in the NCBI CLI tool - so Im not sure how Genspectrum can improve this grouping without reading in data with Entrez - but maybe we need to start doing that!

{"accession":"KY888350.1","completeness":"PARTIAL","geneCount":1,"host":{"lineage":[{"name":"cellular organisms","taxId":131567},{"name":"Eukaryota","taxId":2759},{"name":"Opisthokonta","taxId":33154},{"name":"Metazoa","taxId":33208},{"name":"Eumetazoa","taxId":6072},{"name":"Bilateria","taxId":33213},{"name":"Deuterostomia","taxId":33511},{"name":"Chordata","taxId":7711},{"name":"Craniata","taxId":89593},{"name":"Vertebrata","taxId":7742},{"name":"Gnathostomata","taxId":7776},{"name":"Teleostomi","taxId":117570},{"name":"Euteleostomi","taxId":117571},{"name":"Sarcopterygii","taxId":8287},{"name":"Dipnotetrapodomorpha","taxId":1338369},{"name":"Tetrapoda","taxId":32523},{"name":"Amniota","taxId":32524},{"name":"Mammalia","taxId":40674},{"name":"Theria","taxId":32525},{"name":"Eutheria","taxId":9347},{"name":"Boreoeutheria","taxId":1437010},{"name":"Euarchontoglires","taxId":314146},{"name":"Primates","taxId":9443},{"name":"Haplorrhini","taxId":376913},{"name":"Simiiformes","taxId":314293},{"name":"Catarrhini","taxId":9526},{"name":"Hominoidea","taxId":314295},{"name":"Hominidae","taxId":9604},{"name":"Homininae","taxId":207598},{"name":"Homo","taxId":9605},{"name":"Homo sapiens","taxId":9606}],"organismName":"Homo sapiens","taxId":9606},"isAnnotated":true,"isolate":{"collectionDate":"2015-12-31"},"length":1749,"location":{"geographicLocation":"China: Jiangxi","geographicRegion":"Asia"},"nucleotide":{"sequenceHash":"755E17CC"},"proteinCount":1,"releaseDate":"2017-04-16T00:00:00Z","segment":"4","sourceDatabase":"GenBank","submitter":{"affiliation":"Jiangxi Province Center for Disease Control and Prevention","country":"China","names":["Li,J.","Gong,T.","Shi,Y.","Xiong,Y."]},"updateDate":"2017-04-16T00:00:00Z","virus":{"lineage":[{"name":"Viruses","taxId":10239},{"name":"Riboviria","taxId":2559587},{"name":"Orthornavirae","taxId":2732396},{"name":"Negarnaviricota","taxId":2497569},{"name":"Polyploviricotina","taxId":2497571},{"name":"Insthoviricetes","taxId":2497577},{"name":"Articulavirales","taxId":2499411},{"name":"Orthomyxoviridae","taxId":11308},{"name":"Betainfluenzavirus","taxId":197912},{"name":"Betainfluenzavirus influenzae","taxId":2955465},{"name":"Influenza B virus","taxId":11520},{"name":"Influenza B virus (B/JiangxiAnyuan/1451/2015)","taxId":1977768}],"organismName":"Influenza B virus (B/JiangxiAnyuan/1451/2015)","taxId":1977768}}