Change Conda channel order #3237
Conversation
|
👍 |
| @@ -203,7 +203,8 @@ paste.app_factory = galaxy.web.buildapp:app_factory | |||
| # Pass debug flag to conda commands. | |||
| #conda_debug = False | |||
| # conda channels to enable by default (http://conda.pydata.org/docs/custom-channels.html) | |||
| #conda_ensure_channels = conda-forge,r,bioconda,iuc | |||
| # the recommended channel order is the one from BioConda (https://github.com/bioconda/bioconda-recipes/blob/master/config.yml#L8) | |||
| #conda_ensure_channels = bioconda,r,defaults,conda-forge,iuc | |||
There was a problem hiding this comment.
Isn't iuc channel supposed to be used (also) to fix problems with packages in other channels? If we put it last, it's not very useful. It should probably be first, no?
There was a problem hiding this comment.
This really depends on what we want to use this channel for. My Intension was to put in old recipes, that are outdated and we don't want to have in bioconda, or some ugly hacks (but for new recipes). Not replacing/overwriting recipes from other channels.
Up for discussion I suppose.
There was a problem hiding this comment.
If iuc is only for old recipes, its position shouldn't matter, but if we are ever going to need to use it for some "ugly hack" it needs to be first.
Has any one else opinions here?
There was a problem hiding this comment.
I am +1 on having a channel we directly control as the first entry here. Can we create a new one for this purposes?
There was a problem hiding this comment.
I understand the desire to have like an IUC override channel for hacks. I don't think we need it yet though and it probably shouldn't hold up this PR. Can we create a low priority issue for that and continue with this correction?
There was a problem hiding this comment.
@martenson Do you mean an additional one? I think one "Galaxy" channel should be enough and iuc is mostly unused.
There was a problem hiding this comment.
@jmchilton @bgruening I don't think there is a rush to merge this or the need to backport it since channel order in Conda 3.19.3 matters only if 2 packages have the same version and build numbers. This changed in Conda 4.1.0, see http://conda.pydata.org/docs/channels.html
There was a problem hiding this comment.
@nsoranzo Does putting iuc first have any unwanted effects?
@jmchilton this targets 17.01 - no rush, right?
There was a problem hiding this comment.
@martenson iuc channel seems to have 3 packages, all in a version different from what's available in bioconda.
There was a problem hiding this comment.
I honestly don't know. You guys decide.
martenson
added
area/framework
area/tool-dependencies
kind/enhancement
status/review
and removed
triage
area/framework
labels
|
👍 |
| @@ -31,7 +31,7 @@ | |||
|
|
|||
| DEFAULT_BASE_PATH_DIRECTORY = "_conda" | |||
| DEFAULT_CONDARC_OVERRIDE = "_condarc" | |||
| DEFAULT_ENSURE_CHANNELS = "conda-forge,r,bioconda,iuc" | |||
| DEFAULT_ENSURE_CHANNELS = "iuc,bioconda,r,defaults,conda-forge" | |||
There was a problem hiding this comment.
Sorry for being late to the game, but is it possible that this list should be reversed?
It appears that currently conda-forge takes the lowest priority.
Admittedly this is a bit confusing, but it appears that galaxyproject/tools-devteam#375 fails because bioconda takes the highest priority, while conda-forge should be highest. This also explains passing tests in bioconda but failing when we try to use them in galaxy.
There was a problem hiding this comment.
We need to stick to the bioconda channel order and this was changed to give bioconda the opportunity to overwrite conda-forge. I don't agree with this move but it was decided this way and I think Galaxy should follow it. https://github.com/bioconda/bioconda-recipes/blob/master/config.yml#L11
There was a problem hiding this comment.
We need to stick to the bioconda channel order
absolutely. but I don't think we do, given that for example for perl-gd the import tests pass (because the dependencies are pulled in from conda-forge, I presume). I don't completely understand the details yet, but to me it looks like galaxy is not respecting the channel order. Changing the order in planemo with --conda_channels has no effect whatsoever.
There was a problem hiding this comment.
It's possible that there are two issues:
- For whatever reason the bioconda build system doesn't do what it's supposed to (either false positive test on the import test, or it does actually have conda-forge at a higher priority ... difficult to tell with the limited test output)
- We're unable to change channel priority with
planemo conda_install --conda_channels ...
So I guess @nsoranzo is right to pin perl-gd to a higher libgd build number
There was a problem hiding this comment.
I've noticed similar issues in my development environment when changing the channel order to pull in recipes I'm developing and uploading here https://anaconda.org/gregvonkuster/, but not related to planemo or galaxy.
The issue I've encountered is that if a package/version is is available in one of "iuc,bioconda,r,defaults,condo-forge", and I add an altered recipe for the package using the same version to my anaconda channel and update my channels to be "gregvonkuster,iuc,bioconda,r,defaults,condo-forge", the altered recipe is not pulled from my channel. Instead, the package is pulled from its original channel using the unchanged recipe.
This is all done from the command line without using planemo or galaxy. It seems that conda somehow is not accounting for my channel order.
There was a problem hiding this comment.
When adding channels using conda config --add they should be added in reverse (from low to high priority). How are you adding these channels @gregvonkuster?
@mvdbeek do you have updated your bioconda-utils installation, is everything up to date?
There was a problem hiding this comment.
I haven't used bioconda utils for this ... everything is done by travis, so that's whatever travis is using.
There was a problem hiding this comment.
Note that the condarc is hard code in Planemo and even if you specify all the channel to change the order, Planemo just add those that are missing
There was a problem hiding this comment.
I'm running on Linux and using .condarc. I'm still running miniconda2, version 4.2.13. I have not tried the conda config --add channels approach, so I'll try it and see if it makes a difference. Thanks!
After some discussions with conda-forge we agreed on this channel order for now. The
rchannel will go away at some point as we will try to get all packages from r into conda-forge.I think this PR needs to be backported.