conda

[infokiller]

Hi, and thanks for the great project- the guiding ideas really resonate with me.

In a better world, I would be able to only use pip to manage dependencies in Python projects, however sometimes conda is useful, either because it can handle stuff pip can't (notably different python versions per env), or because the conda packaging has better support from from the package's author (if the pip's package has exists at all).

In practice, for my projects I usually use a minimal environment.yml file:

name: myenv
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.8
  - pip
  - pip:
    - -r requirements.txt
    - -r requirements-dev.txt

If all the dependencies I need are available from pip (the best situation), they can all be managed with pip-compile. If not, I can add more dependencies to the conda file.

Is there a way to use zpy so that it integrates well with conda?
For example, in my setup conda stores envs in ~/.local/pkg/conda/envs, so zpy would need to use that, and with the name of the env, not the current hash (AFAIK conda finds myenv by checking ~/.local/pkg/conda/envs/myenv).

[AndydeCleyre]

Thanks so much, your issues and interest have made my day!

I'm not a conda user, so the more concrete workflow examples you can share, the better.

If you want to use venvs generated by conda, rather than zpy+venv, you probably want to avoid using envin, activate, pipup, vrun, vlauncher, and the vpy* ones. The other commands should generally use whatever venv is already activated for the shell. Alternatively, the conda venvs could be symlinked into the zpy-expected locations, and we might want to add a function to automate that.

The following actions can be done currently, assuming you have pip-compile installed in the conda venv:

• create venvs with conda
• activate venvs with conda
• install packages with conda, not pips/pipcs/pipacs/pip-sync
• use pipc and pipac to generate requirements txts

Given that, please let me know how you see a helpful conda integration looking.

Thanks!

[infokiller]

Thanks for the quick and helpful reply, and I'm happy it created enthusiasm!

The main use cases I have for using conda to manage python envs are:

• Install packages/dependencies that are either only packaged for conda, or better supported than pip (for example the pytorch quickstart recommends conda over pip). I think that originally pip didn't have good support for non-Python dependencies, which made installing some packages hard and inconsistent across operating systems (since you had to use the OS package manager for these deps, if you're lucky enough to have it packaged). Nowadays I see many python packages with non-Python deps that can be installed by pip (numpy, pytorch, tensorflow, etc.), so I guess the situation is much better now.

• Using a different python version and low level libraries per env (again, instead of relying on the OS package manager).

In general, I prefer pip and pip-tools over conda for managing deps. I don't have much experience with venv or pyenv to comment on how they compare to conda, though I will say my experience with pyenv wasn't great since it required me to build from source (in contrast, conda supports binary packages).

My current go-to approach for python projects is to use conda for defining the environment, but use pip (with pip-tools) for managing the deps. As mentioned above, sometimes it's better to install a dependency from conda, but it seems increasingly rare. So, in the common situation where all deps can be installed from pip, I define the environment like in my original comment:

name: myenv
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.8
  - pip
  - pip:
    - -r requirements.txt
    - -r requirements-dev.txt

The requirements*.txt files are generated using pip-compile.

[infokiller]

If you want to use venvs generated by conda, rather than zpy+venv, you probably want to avoid using envin, activate, pipup, vrun, vlauncher, and the vpy* ones. The other commands should generally use whatever venv is already activated for the shell. Alternatively, the conda venvs could be symlinked into the zpy-expected locations, and we might want to add a function to automate that.

The following actions can be done currently, assuming you have pip-compile installed in the conda venv:

• create venvs with conda
• activate venvs with conda
• install packages with conda, not pips/pipcs/pipacs/pip-sync
• use pipc and pipac to generate requirements txts

Given that, please let me know how you see a helpful conda integration looking.

Off the top of my head:

• Commands for creating/activating/listing envs from project dirs support conda as an option
• pipz can install packages using conda envs (and also specifying the python version)

I think the folder layout is similar in conda and venv, so I believe most of the code can be shared.

[AndydeCleyre]

This project holds very much with the pattern of mapping each folder/project to its own set of alternative venvs, one of them being default.

I think the conda pattern is to have a small handful of venvs that many projects can map to.

So my first take is that it doesn't really help to use zpy to create/activate conda envs, when you can use conda to do so, then use the venv-agnostic zpy commands to work with that.

The pipz business sounds like it may be worth playing with. Do you have an example package you would want to install with pipz+conda for me to test with?

[infokiller]

I use and advocate the exact same pattern, and I believe conda encourages that as well. I hope it uses hardlinks or whatever to save space when the same package is installed in different envs.

As for pipz+conda- any package you would install with pipz+pip will work, one example for me is ipython. My current process for installing a tool is similar to a manual version of how pipz works:

1. Manually create and activate a conda env from a new environment.yml file that looks similar to the example above
2. Install pip-tools in the new env
3. Create a requirements.in file with only the package of the tool
4. Run pip-compile and pip-sync

[AndydeCleyre]

As for pipz+conda- any package you would install with pipz+pip will work, one example for me is ipython.

ipython installs very well using plain pip (or now also uv), so I'd rather work around a concrete example where there's a benefit the user can notice by working with conda.