Packaging

[pdiercks]

Hi all,

I opened this issue to discuss how the packaging can be improved and collect ideas (I cannot start a discussion for some reason).
Currently there are two sources to define the package fenicsxconcrete and its runtime dependencies and optional dependencies: pyproject.toml and .conda/meta.yaml.

Since fenics-dolfinx is a non-python dependency, we aim at providing a conda package, such that stable versions can be installed simply via conda install --channel bam77 fenicsxconcrete or similar.

For the installation of the development version in the CI we currently use environment.yml including fenics-dolfinx as a basis and let pip handle the installation of the runtime dependencies (see pyproject.toml).
However, the same information is necessary in the recipe .conda/meta.yaml to build the conda package.
This is not ideal, because two files need to be maintained instead of just one. The reason for this is also that the package names (and in some cases also contents) on conda-forge and pypi are not the same.

Ideally, only one source of information (e.g. pyproject.toml) is defined/maintained to manage the package.
To accomplish this, my idea at the moment would be to load the data from pyproject.toml inside .conda/meta.yaml and rely on conda build to build the conda package for (a) the CI and (b) the user. (we simply do not rely on pip as we do at the moment).
The pyproject.toml would contain the names of the packages on conda-forge and therefore pip install -e . would not work.

Further info: I found a cookiecutter for conda python packages. Here are examples on the install commands and the recipe

[srosenbu]

Thank you for looking into this. I have one quick question: Why would we store the relevant packages in pyproject.toml if we plan on not using pip? And how would the data be read during the conda build?

[joergfunger]

So instead of using pip in the local development, we would build a local conda package that could be installed via
conda install --use-local .?

[pdiercks]

Thank you for looking into this. I have one quick question: Why would we store the relevant packages in pyproject.toml if we plan on not using pip? And how would the data be read during the conda build?

We would still use pip to install fenicsxconcrete, but not its dependencies. In the conda recipe there is usually a build command such as $PYTHON -m pip install --no-deps and for this to work a pyproject.toml or setup.py or similar needs to be present.
The requirements are read from the pyproject.toml during the build process. See recipe

[pdiercks]

So instead of using pip in the local development, we would build a local conda package that could be installed via conda install --use-local .?

yes, I guess instead of

conda env create -n fenicsxconcrete
conda env update -n fenicsxconcrete -f ./environment.yaml

We could do

conda build path/to/recipe --no-test --no-anaconda-upload
conda install --use-local fenicsxconcrete

in the CI

[pdiercks]

For the local development probably conda develop is better.
The build and local install would be good for the CI, although would not want to build the package several times (we already have a pipeline just for the build).

[joergfunger]

I have never used conda develop, but if you can make it work and share how you have done that, this would be great.

[pdiercks]

ok, I will try to implement the above in a PR.

[danielandresarcones]

Just as a follow-up to look into, we recently tried to install the environment from scratch using conda env create -n fenicsxconcrete -f ./environment.yaml and got plenty of error messages due to not being able to find boost dependencies during the installation of dolfinx. Installing the cloned repository through pip install -e .[tests, docs] worked fine. I am not sure why did that error happen or if it can be reproduced, but it is something to keep in mind (probably after the installation and packaging methods are stable).