updated to ensembl release 110; did some patching #1536

andrewkern · 2023-10-25T18:36:20Z

okay here is the update to ensembl 110. i had to do a bit of patching along the way, adding or modifying ensembl_ids in the species.py files of a few species.

in addition there is a horrible "duct tape" operation i did to patch the canis_familiaris situation. i'll point that out below.

andrewkern · 2023-10-25T18:37:09Z

maintenance/main.py

+    if ensembl_id == "canis_lupus_familiaris":
+        ensembl_id = "canis_familiaris"


this is a horrible, terrible thing that i've done, but i've done it because this single ensembl shift disagrees with all the other ensembl ids....

petrelharp · 2023-10-25T18:58:03Z

maintenance/main.py

    tmp = ensembl_id.split("_")[:2]
+    print(tmp, ensembl_id)


stray print

oops-- debugging

petrelharp · 2023-10-25T18:59:29Z

stdpopsim/catalog/GasAcu/genome_data.py

-        "MT": {"length": 16543, "synonyms": []},
-    },
-}
+data = {"assembly_accession": None, "assembly_name": "BROAD S1", "chromosomes": {}}


uh-oh, what happened to this one?

wow that's not good

it looks to have all the info there: http://rest.ensembl.org/info/assembly/gasterosteus_aculeatus?synonyms=1

petrelharp · 2023-10-25T18:59:50Z

stdpopsim/catalog/GasAcu/species.py

@@ -116,7 +116,7 @@

 _species = stdpopsim.Species(
    id="GasAcu",
-    ensembl_id="9307941",
+    ensembl_id="gasterosteus_aculeatus",


perhaps this is why the assembly dissappeared?

no, the original ensembl_id was incompatible with the REST API and the maintenance script...

petrelharp · 2023-10-25T19:00:10Z

stdpopsim/catalog/HelMel/genome_data.py

-    },
+    "assembly_accession": "GCA_000313835.1",
+    "assembly_name": "Hmel1",
+    "chromosomes": {},


And another one that dissappeared?

Looks like another manually manipulated one

Well this one has no karyotype entry:

--- assembly_accession: GCA_000313835.1 assembly_date: 2012-02 assembly_name: Hmel1 coord_system_versions: - Hmel1 default_coord_system_version: Hmel1 genebuild_initial_release_date: 2012-03 genebuild_last_geneset_update: 2012-03 genebuild_method: import genebuild_start_date: 2012-03-HGC golden_path: 273786188 karyotype: [] top_level_region: - coord_system: scaffold length: 163478 name: HE667775 synonyms: - dbname: INSDC name: HE667775.1

... and, it's just a bunch of scaffolds - nothing as long as in the assembly we've got.

petrelharp · 2023-10-25T19:00:47Z

stdpopsim/catalog/PonAbe/genome_data.py

-        "22": {"length": 35308119, "synonyms": []},
-        "X": {"length": 151242693, "synonyms": []},
-        # Mitochondria absent in ponAbe3, so length taken from ponAbe2.
-        "MT": {"length": 16499, "synonyms": []},


looks like we need to stick in this manually again?

nope - it's there, just not listed in the karyotype:

- coord_system: primary_assembly length: 16499 name: MT

petrelharp · 2023-10-25T19:01:00Z

stdpopsim/catalog/StrAga/genome_data.py

-    "chromosomes": {"1": {"length": 2065074, "synonyms": ["I"]}},
+    "assembly_accession": "GCA_001017915.1",
+    "assembly_name": "ASM101791v1",
+    "chromosomes": {},


another empty one?

yep - the current build for this has no chromosome-level assembly

petrelharp · 2023-10-25T19:01:54Z

The most alarming thing here is those species that now don't have any chromosomes - any idea what's up with that?

andrewkern · 2023-10-25T19:10:43Z

looks like the script isn't working quite right... will have to dig in further

jeromekelleher

Hmm, watch out for the manually edited files that are getting touched here.

It's pretty messy isn't it?

jeromekelleher · 2023-10-25T19:28:00Z

stdpopsim/catalog/DroSec/genome_data.py

@@ -1,13 +1,6 @@
-# File created manually from https://www.ncbi.nlm.nih.gov/assembly/GCF_004382195.1
+# File autogenerated from Ensembl REST API. Do not edit.


Looks like this file shouldn't be changed as was created manually

maybe we should add a species level attribute -- manually_added or something -- that would indicate if the maintenance script should go ahead with the download

I'd vote "no", so we are alerted when a species that previously didn't have a build gets one; otherwise just manually back out the change in such cases.

(btw the reason this doesn't work is because ensembl doesn't have a karyotype entry for DroSec, just some ~Mb scale scaffolds)

jeromekelleher · 2023-10-25T19:28:39Z

stdpopsim/catalog/HelMel/genome_data.py

-    },
+    "assembly_accession": "GCA_000313835.1",
+    "assembly_name": "Hmel1",
+    "chromosomes": {},


Looks like another manually manipulated one

igronau · 2023-11-21T18:07:31Z

stdpopsim/catalog/CanFam/genome_data.py

-        "38": {"length": 23914537, "synonyms": []},
-        "X": {"length": 123869142, "synonyms": []},
-        "MT": {"length": 16727, "synonyms": []},
+        "1": {"length": 123313939, "synonyms": []},


Looks like the chromosome lengths changed between these two builds of CanFam, and new build doesn't have a mitochondrial genome. We probably want to stick with old version to avoid conflicts with recombination map.

Hm - unlike in other species, it appears to legit not have MT any more.

petrelharp · 2023-12-12T17:13:42Z

stdpopsim/catalog/AnoCar/genome_data.py

+        "d": {"length": 1094478, "synonyms": ["chrLGd"]},
+        "f": {"length": 4257874, "synonyms": ["chrLGf"]},
+        "g": {"length": 424765, "synonyms": ["chrLGg"]},
+        "h": {"length": 248369, "synonyms": ["chrLGh"]},


wtf, the previous chromosome names (e.g., LGa) are not included in the synonyms. I vote to leave well enough alone there. But - why no mitochondria?

And gee, the mitochondria shows on ensembl:

ah I see what happened here: consulting http://rest.ensembl.org/info/assembly/anolis_carolinensis they've removed the mitochondria from the "karyotype" list of chromosomes, but it's still there, at the end of a list of a bajillion tiny contigs:

- coord_system: primary_assembly length: 17223 name: MT

petrelharp · 2023-12-12T17:17:25Z

stdpopsim/catalog/ApiMel/genome_data.py

-        "CM009944.2": {"length": 10670842, "synonyms": ["NC_037651.1"]},
-        "CM009945.2": {"length": 9534514, "synonyms": ["NC_037652.1"]},
-        "CM009946.2": {"length": 7238532, "synonyms": ["NC_037653.1"]},
-        "CM009947.2": {"length": 16343, "synonyms": ["NC_001566.1", "MT"]},


ensembl dropping the MT label here

and those synonyms are still there, labeled "RefSeq" or "GenBank":

- coord_system: primary_assembly length: 16343 name: CM009947.2 synonyms: - dbname: ensembl_internal_synonym name: NC_001566 - dbname: GenBank name: MT - dbname: INSDC name: CM009947.2 - dbname: ensembl_internal_synonym name: CM009947 - dbname: RefSeq name: NC_001566.1

petrelharp · 2023-12-12T17:48:04Z

stdpopsim/catalog/PanTro/genome_data.py

+        "22": {"length": 37823149, "synonyms": []},
+        "X": {"length": 155549662, "synonyms": []},
+        "Y": {"length": 26350515, "synonyms": []},
+        "MT": {"length": 16554, "synonyms": []},


looks like they just removed those synonyms, oh well

petrelharp · 2023-12-12T17:59:53Z

I've been through this. Notes:

I've added back in (just via git checkout main <file>) the manually added genome info.
Some genomes have MT now missing, even though the lengths are listed, because MT has been removed from their "karyotype". This could cause a problem if MT is listed in the genetic map? We could add some code to the ensembl maintenance parsing that checks if there's any scaffolds with name == "MT", and if so, adds them, even if not in the karyotype.
We don't have some synonyms we had previously, but could fixup the maintenance code to get those by including different dbNames.
Some genomes (two of them!) seem to have much worse assemblies on ensembl now - previously they had chromsome-level; now they don't. I don't know what's up with that.

This all seems straightforward... except maybe the last point. I guess we should just convert those species over to being "manually added"?

The other thing to deal with here is lifting over genetic maps...

updated to ensembl release 110; did some patching

ce96648

andrewkern commented Oct 25, 2023

View reviewed changes

andrewkern mentioned this pull request Oct 25, 2023

What to do about shifting ensembl build? #1521

Open

petrelharp reviewed Oct 25, 2023

View reviewed changes

jeromekelleher reviewed Oct 25, 2023

View reviewed changes

igronau reviewed Nov 21, 2023

View reviewed changes

petrelharp reviewed Dec 12, 2023

View reviewed changes

put back in manually added info

8962f3e

silastittes mentioned this pull request Mar 15, 2024

updates to hash codes and file names #1553

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

updated to ensembl release 110; did some patching #1536

updated to ensembl release 110; did some patching #1536

andrewkern commented Oct 25, 2023

andrewkern Oct 25, 2023

petrelharp Oct 25, 2023

andrewkern Oct 25, 2023

petrelharp Oct 25, 2023

andrewkern Oct 25, 2023

petrelharp Dec 12, 2023

petrelharp Oct 25, 2023

andrewkern Oct 25, 2023

petrelharp Oct 25, 2023

jeromekelleher Oct 25, 2023

petrelharp Dec 12, 2023

petrelharp Dec 12, 2023

petrelharp Oct 25, 2023

petrelharp Dec 12, 2023

petrelharp Oct 25, 2023

petrelharp Dec 12, 2023

petrelharp commented Oct 25, 2023

andrewkern commented Oct 25, 2023

jeromekelleher left a comment

jeromekelleher Oct 25, 2023

andrewkern Oct 25, 2023

petrelharp Dec 12, 2023

jeromekelleher Oct 25, 2023

igronau Nov 21, 2023

petrelharp Dec 12, 2023

petrelharp Dec 12, 2023

petrelharp Dec 12, 2023

petrelharp Dec 12, 2023

petrelharp Dec 12, 2023

petrelharp Dec 12, 2023 •

edited

Loading

petrelharp Dec 12, 2023

petrelharp commented Dec 12, 2023

		if ensembl_id == "canis_lupus_familiaris":
		ensembl_id = "canis_familiaris"

		@@ -1,13 +1,6 @@
		# File created manually from https://www.ncbi.nlm.nih.gov/assembly/GCF_004382195.1
		# File autogenerated from Ensembl REST API. Do not edit.

updated to ensembl release 110; did some patching #1536

Are you sure you want to change the base?

updated to ensembl release 110; did some patching #1536

Conversation

andrewkern commented Oct 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrelharp commented Oct 25, 2023

andrewkern commented Oct 25, 2023

jeromekelleher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrelharp Dec 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrelharp commented Dec 12, 2023

petrelharp Dec 12, 2023 •

edited

Loading