Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Using Vcpkg/conan for dependencies and third party libraries #672

Open
RamadanAhmed opened this issue Jan 16, 2021 · 18 comments · May be fixed by #1413
Open

[Feature Request] Using Vcpkg/conan for dependencies and third party libraries #672

RamadanAhmed opened this issue Jan 16, 2021 · 18 comments · May be fixed by #1413

Comments

@RamadanAhmed
Copy link
Contributor

Feature

use vcpkg/conan instead of system packaging for dependencies and third party libraries

Motivation

System Packaging have the problem that it can't be updated till next release update of the system, or some custom repo with needed library. On the other hand we can use conan or vcpkg to build these library which is updated regularly, and if certain library build configuration is not suitable, you can edit it and publish the edited version to be used by other to build your library and its working cross-platform (linux-windows-macos) so one script fit all -which will help in CI-.

Pitch

I have used vcpkg and it worked for example

./vcpkg.exe install tbb metis boost eigen3 suitesparse[metis] ceres --triplet x64-windows

but I had to change some linking and includes to work. So if we can use one of these package managers it will remove the need for almost all Find*.cmake files and it will make build easier on windows, linux and macos

@RamadanAhmed RamadanAhmed changed the title Using Vcpkg/conan for dependencies and third party libraries [Feature Request] Using Vcpkg/conan for dependencies and third party libraries Jan 16, 2021
@varunagrawal
Copy link
Collaborator

Similar to #450

@varunagrawal
Copy link
Collaborator

@RamadanAhmed I believe this request would fall into the gtsam-packaging namespace. The current maintainers of that repo don't have expertise with Windows packaging and it is the only thing blocking this feature.

@varunagrawal
Copy link
Collaborator

It seems like you are already well versed in this. Can you please submit a PR?

@RamadanAhmed
Copy link
Contributor Author

RamadanAhmed commented Jan 22, 2021

Ok I will try to do it but if I needed to edit many of the current Handle*.cmake or add new ones, should I do each one in separte PRs or just do it in one PR ?

@ProfFan
Copy link
Collaborator

ProfFan commented Jan 22, 2021

I do think this is a very significant design choice. I personally is not a big fan of vcpkg, and I am concerned that this will impact ABI compatibility with system packages.

However, if this can be done without any behavioral change, it should be acceptable.

@varunagrawal
Copy link
Collaborator

I was under the impression that vcpkg would be used to create a homebrew like installation path for GTSAM on Windows. Looking at the docs this seems like zero changes to any C++ code.

@wolfv
Copy link

wolfv commented Mar 11, 2021

I would also be interested in adding gtsam to conda-forge: we're working towards having many robotics packages on conda-forge and the robostack project. It's cross-platform (Windows, OS X and Linux) and already has many data-science and HPC libraries.
Contrary to vcpkg, conda it's a binary package manager: that means no build time for the users (like apt on Ubuntu or dnf on Fedora).

Here is a recent blog post: https://medium.com/robostack/cross-platform-conda-packages-for-ros-fa1974fd1de3

@ameysutavani
Copy link

ameysutavani commented Mar 22, 2021

@wolfv +1 for getting gtsam on conda-forge. I recently switched to RoboStack (mamba + RoboStack + ros-neotic = 🔥 ) from ros-melodic (and source compiled gtsam). I have gotten pretty much everything replicated in a mamba environment; the only missing piece of the puzzle for me is gtsam. I would love to help-out on testing if you create a recipe for gtsam on conda-forge

@varunagrawal
Copy link
Collaborator

@wolfv we already have a python package for GTSAM. Would that suffice?

@ameysutavani
Copy link

@varunagrawal I had to modify @wolfv 's patch a little bit for getting latest develop (74e4fc3) to compile with boost 1.74

Maybe #634 needs a follow-up fix before this can be compiled on conda-forge.

@wolfv
Copy link

wolfv commented Mar 22, 2021

@ameysutavani thanks for the kind words! great that the robostack is useful for you!

@varunagrawal do you mean PyPI package? conda is not really python-specific, in fact you can ship any kind of binary (C, C++, Rust, Go ...) in a Conda-package. As such, a Python package on PyPI doesn't help much. Conda packages are more like Debian / RPM packages (but cross-platform).

I have an initial recipe that seems to work fine. I am going to post it shortly to 'staged-recipes' on conda-forge. If any of you are interested in becoming co-maintainers please let me know (usually not much to do, except when a release comes around, but even that is handled mostly by bots).

The last issue is that I need to un-vendor Ceres and Metis. Would be cool if that would be easily doable from CMake (as it is for Eigen).

@wolfv
Copy link

wolfv commented Mar 22, 2021

Here is an initial recipe submitted to conda-forge: conda-forge/staged-recipes#14339

I've tried to unvendor all libraries (except for metis).

We could also depend on TBB. Do you guys know if you're compatible with the latest 2021 TBB release? Others (gazebo etc.) are not yet.

@wolfv
Copy link

wolfv commented Mar 22, 2021

btw it's kinda surprising that PyPI has an unreleased gtsam version, if I understand that correctly?

@ProfFan
Copy link
Collaborator

ProfFan commented Mar 22, 2021

Yes, there is a PyPI dev version of 4.1.1, but that should not be the default version you install by pip.

@varunagrawal
Copy link
Collaborator

@RamadanAhmed @wolfv you might want to check out gtsam-packaging. Currently it only supports Ubuntu PPAs, but the idea is that we can have CI systems in that repo that build and push to the various package indices.

I highly recommend creating PRs over there so we can regularly publish to vcpkg, conda-forge, homebrew, etc.

@wolfv
Copy link

wolfv commented Mar 23, 2021

if we have a package on conda-forge, bots will watch this repository and create an automated version bump on the "feedstock" git repository whenever a new version of gtsam is tagged. the packages on conda-forge have to come from the feedstocks (you could start your own channel where you publish gtsam though, and use a recipe in your git repo).

Here is such an feedstock for e.g. rosdistro: https://github.com/conda-forge/rosdistro-feedstock
And you can see some of these automated version bumps by the bots here: https://github.com/conda-forge/rosdistro-feedstock/pulls?q=is%3Apr+sort%3Aupdated-desc+is%3Aclosed

It looks like we can't build the tests against boost 1.74 (current default on conda-forge).
We can force gtsam to use boost 1.72 (still compatible with many packages on conda-forge) or we can use boost 1.75 (not compatible with any other packages on conda-forge).

Or we can ignore the tests and not build them. Although I am not sure if users of gtsam could build their applications against that released version then.

What do you think?

@ameysutavani
Copy link

ameysutavani commented Mar 24, 2021

@wolfv Does the problem with tests persist even with the patch?

@talregev talregev linked a pull request Jan 24, 2023 that will close this issue
@Tobias-Fischer
Copy link
Contributor

FYI - if people are looking at this issue, gtsam has been packaged on conda-forge for a while now, see https://github.com/conda-forge/gtsam-feedstock/ (or simply mamba install gtsam).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants