Switch builds to use new ingest outputs

[joverlee521]

TODOs

• update geolocation rules
• remove /trials/ingest from S3 prefix in manual-upload and nextstrain-automation
• update GH Action workflows
  • remove targets upload_all_raw_sequences, upload_all_sequences, upload_all_metadata in upload job
  • add ingest workflow to upload workflow (before run-nextclade)
• copy all files from s3://nextstrain-data-private/files/workflows/seasonal-flu/trials/ingest/gisaid-downloads/processed/ to prod S3 location
• add docs for curation (might reveal papercuts that we want to tackle before completing switch)
• save a copy of metadata.tsv and sequences.fasta produced from fauna on S3 (?)
• add rules/config for reprocessing all data in nextstrain-automation (prompted by Add overview info to ingest READMEs #237)

[huddlej]

One more sanity check before we switch to this ingest approach would be to cross-check the updated strain names against titer record strain names. It seems likely that strains the titer records have been renamed in GISAID. Any mismatch between those names would cause us to silently drop titer records.

[huddlej]

I just ran into a variant of the issue I mentioned above where H1N1pdm titer data include a reference strain named A/SouthAustralia/59/2023. We have this strain in our fauna metadata and sequences, but for the new ingest we only have the strain in the metadata and not in the sequences file.

[joverlee521]

We have this strain in our fauna metadata and sequences, but for the new ingest we only have the strain in the metadata and not in the sequences file.

Ah, this is a slightly different issue. The ingest workflow dedups strains by prioritizing the latest record, so it is using EPI_ISL_17811563, which only has the na and mp segments. We could prioritize the older record EPI_ISL_17691801, which has ha, na, and mp. This is not obvious in fauna data because the records get merged on the strain name.

[huddlej]

Thanks for figuring that one out, @joverlee521! I think we will usually want to prioritize records with HA over records without, then records with NA, and then everything else.

[joverlee521]

Thanks for figuring that one out, @joverlee521! I think we will usually want to prioritize records with HA over records without, then records with NA, and then everything else.

Thanks @huddlej, moving this into a separate issue that I've linked as a sub-issue.