[question] Workspaces with lockfiles and CMakeToolchain generator

[ibrawada]

What is your question?

Hello,

while experimenting with conan workspaces i noticed several things that didn't work or i couldn't make them work. I had taken the libraries from here: https://github.com/conan-io/examples2/tree/main/ci/game and added external dependencies and lockfiles.

For WS files i have used the content from here: https://docs.conan.io/2/tutorial/developing_packages/workspaces.html#id2

The first thing that didn't happen was that the generation of individual toolchain files for each library. The setting of preprocessor definition (eg. in ai: https://github.com/conan-io/examples2/blob/main/ci/game/ai/conanfile.py#L24) wasn't done. This was easily overcome by adding the line of code inside WS conanws.py. But this will not scale with a lot of editable libraries and also they will be globally available and not defined per library.

When performing conan workspace super-install --lockfile-out <file>.lock conan didn't generate a lockfile. Im not sure if im missing something in the command. This is not a problem since the lockfile can easily be created for the end product.

After doing modifications and creating new packages with conan workspace create i wanted to have an updated lockfile for the game but couldn't get it generated by appending --lockfile-out <file.lock> to create command.
I tried then to generate the lockfile as to i would normally do with conan lock create game/conanfile.py but that didn't work because the packages are editable and the lockfile i get contains no real info. I was able to generate it outside of my WS folder but the problem there is that the lockfile can be polluted with package RREVs coming from other projects. Would it be possible to have something like conan workspace deactivate? Now i do it by renaming conanws.py to _conanws.py

Thanks

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @ibrawada

Thanks for your feedback.

The first thing that didn't happen was that the generation of individual toolchain files for each library. The setting of preprocessor definition (eg. in ai: https://github.com/conan-io/examples2/blob/main/ci/game/ai/conanfile.py#L24) wasn't done.

This is kind of expected and a result of having a super-build project. The moment that conan workspace super-install is called, the assumption is that there are no individual projects and packages, and there is no a toolchain for each one. There is one single project ,the "super-project" with one single toolchain file for it. Because that is the only way that CMake works, it can only have 1 toolchain file for the whole project.

This is why the super project logic should take care of the things that the possible subdirectories/subprojects might need, and this is the reason why the https://docs.conan.io/2/reference/workspace_files.html#conanws-py-super-build-options exists, allowing to collect the values of options from the workspace packages with workspace_packages_options, in order to do something.
Likewise, if the packages of the super-build require some preprocessor definition, this should be defined by the super-project in conanws.py.
Note in this case is particularly noticeable that the definition of the preprocessor directive PKG_VERSION is not good to be used in a super-project. Because if many different packages define that PKG_VERSION, but the project can only define one value, the different sub-projects would get incorrect values.
In order for something like this to work:

• Each package should define a better preprocessor definition like PKG_VERSION_MYGAME and PKG_VERSION_MYENGINE, for each package being different. Or maybe use another mechanism not based in preprocessor
• Then, the conanws.py should define its own generate() using CMakeToolchain like in https://docs.conan.io/2/tutorial/developing_packages/workspaces.html, to define the values it wants, possibly using the .packages() method

This was easily overcome by adding the line of code inside WS conanws.py. But this will not scale with a lot of editable libraries and also they will be globally available and not defined per library.

If there are tons of editable packages and using a "super-build" project, then some consensus, conventions and alignment is necessary, and the super-build needs some understanding of the underlying packages and the capability of defining the super-build at that scale, typically via automation. This is why the workspace_packages_options was added. Maybe there are other points that need or can benefit from other data interfaces for automation, this needs to be discovered and built. The preprocessor definitions defined in recipe code like generate() would need some conventions in order to be defined from the "super-build", it is mostly impossible to automatically gather that information from Conan recipes generate() methods that are not even evaluated at all.

So it would be interesting to learn about the use cases. Because the above PKG_VERSION in the CI-tutorial isn't really that necessary, it is there for convenience and simplifying the CI tutorial, requiring less code changes, but in real-world, the version is not really injected by Conan that way, but it is reflected in actual C/C++ source code changes.

(I'll respond about lockfiles in another comment, to keep it a bit more reduced and clean)

[memsharded]

I have verified it, you are right that the workspace commands do not generate the lockfiles with --lockfile-out, the argument is there because it is added together with all the lockfile arguments, but it is not implemented yet.

We have to investigate this, because the workspace can contain different dependency graphs, with different "root" packages having even completely disjoint dependency graphs. I think the lockfiles can support this use case, but better check it carefully. I am assigning this for next 2.23 release to investigate.

[ibrawada]

Hello @memsharded,

as always, thanks for your in-depth reply.

to understand workspace_packages_options better(which has slipped my attention). Lets say that engine has an option with_o1 = False.
inside conanfile.py

def generate()
...
cache_variable["WITH_O1"] = self.options.with_o1  

inside CMakeLists.txt

if(WITH_O1)
  target_compile_definition(engine PRIVATE WITH_O1)
endif()

All other repos might have similar options definitions.

I also have a global profile for the whole project, something like this

include(default)
[options]
engine/*:with_o1=True
ai/*:with_boost=True 
etc..
# This is not possible in Monorepo because the shared/static is controlled by one global variable BUILD_SHARED_LIBS
engine/*:shared=True
ai/*:shared=False

So in order to forward the options to corresponding CMakeLists.txt i need to implement generate() in conanws.py as following:

def generate(self):
  tc = CMakeToolchain(self)  
  for pkg, options in self.workspace_packages_options.items():
     for k, v in options.items():
           if k == "with_o1":
              tc.cache_variable["WITH_O1] = v
           if k == "with_boost":  
              tc.cache_variable["WITH_BOOST] = v
              
              and so on

From reading the docs i unterstand that this is the suggested way to do it.
This could get very messy and error prone with increasing number of packages as well as if their options keep on changing and extending

[ibrawada]

What i also tried and seems to work is following:

I have created for each library a "libname"_tc.cmake file. Inside, i set the options and variable for the corresponding lib.
eg. engine_tc.cmake

set(WITH_O1 ON)
set(BUILD_SHARED_LIBS ON)

eg. ai_tc.cmake

set(WITH_BOOST ON)
set(BUILD_SHARED_LIBS OFF)

The CMakeLists in the root of the Workspace has the usual loop to include all the project with additional include(build/${pkg}_tc.cmake)

foreach(pair ${CONAN_WS_ANOTHER_BUILD_ORDER})
    string(FIND "${pair}" ":" pos)
    string(SUBSTRING "${pair}" 0 "${pos}" pkg)
    math(EXPR pos "${pos} + 1")  # Skip the separator
    string(SUBSTRING "${pair}" "${pos}" -1 folder)

    **include(build/${pkg}_tc.cmake)**
    add_project(${pkg} ${folder})
endforeach()

After CMake configure/generate the engine has the preproc WITH_O1 defined and is built as shared and ai is building as static lib.
It looks like this would make it possible to have exactly the same configurations defined by the global profile when working in a WS and when compiling each lib as standalone

Question:
Would this (minimalistic pseudo) code be possible to implement in Conan?

def generate(self):
    tc = CMakeToolchain(self)

    for dep in self.dependencies:
        tc.create_tc(f"{dep.name}_tc.ame", dep.options)
    tc.generate()

[memsharded]

Hi @ibrawada

Quick update.

yes, the functionality you proposed has already been implemented as:

from conan import ConanFile
            from conan import Workspace

            class MyWs(ConanFile):
                settings = "arch", "build_type"
                def generate(self):
                    for pkg, dep in self.workspace_packages.items():
                        for k, v in dep.options.items():
                            self.output.info(f"Generating with opt {pkg}:{k}={v}!!!!")

So now it is possible to have access to workspace packages full recipes.
It is also possible to re-use their toolchain logic, see the section in the docs: https://docs.conan.io/2/reference/workspace_files.html#conanws-py-super-build-workspace-packages

I keep this ticket open for the lockfiles part, but I think for the toolchain parts it can be considered solved. Please have a look and create new tickets as necessary for the workspace_packages new feature.