Skip to content

Conflicting versions of toml11 and nlohmann_json when used as internally shipped dependencies at multiple points in the software stack #5355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
franzpoeschel opened this issue Apr 28, 2025 · 4 comments · May be fixed by #5367
Open

Conflicting versions of toml11 and nlohmann_json when used as internally shipped dependencies at multiple points in the software stack #5355

franzpoeschel opened this issue Apr 28, 2025 · 4 comments · May be fixed by #5367

Comments

@franzpoeschel
Copy link
Contributor

Both PIConGPU and the openPMD-api internally ship with the header-only dependencies toml11 and nlohmann_json. These internally shipped versions are used by default in both software packages.

When setting up a default installation of PIConGPU with openPMD-api enabled, this may lead to conflicts when both ship with different versions of one such library. (This is even when they are header-only, since they might still contain symbols with the same name, leading to undefined behavior since the loader jumps to the wrong function, or the same datatype might exist in two non-compatible layouts, …).

I'm not really sure how to best solve this, some ideas:

  1. Include nlohmann_json and toml11 into their own namespaces in PIConGPU to avoid symbol clashes
  2. By default, look for an installed version of nlohmann_json / toml11; use internally-shipped version only if no installed version was found
  3. Reexport the headers in openPMD, include them from there in PIConGPU
  4. Check in PIConGPU which version of toml11/nlohmann_json was used by openPMD, disallow use of shipped dependency if the dependencies do not match (or if the version cannot be figured out)

I don't really like any of these ideas, so feel free to suggest solutions.
@chillenzer
Copy link
Contributor

This is a case often encountered (for example with alpaka). We typically make sure that PIConGPU explicitly sets all the shipped dependencies in all packages to its own versions. For example, we force mallocMC to use the version shipped with PIConGPU here:

picongpu/include/pmacc/PMaccConfig.cmake

Line 388 in f101f25

set(mallocMC_USE_alpaka "${PMacc_DIR}/../../thirdParty/alpaka" CACHE STRING "Select which alpaka is used for mallocMC")

I would suggest that we do the same for openPMD-api.

@franzpoeschel
Copy link
Contributor Author

This solution only works for header-only libraries; the conflicting versions of nlohmann-json and toml11 are at this point already compiled as part of libopenPMD.so. I guess that we will need to do this the other way around, re-export both libraries (or at least their versions) from openPMD so PIConGPU can pick them up in the right versions; also find some way to deal with current openPMD versions that do not reexport the libraries.

@chillenzer
Copy link
Contributor

Does openPMD-api have those libraries in the interface?

This was referenced May 15, 2025
Open
Open
@franzpoeschel
Copy link
Contributor Author

franzpoeschel commented May 15, 2025 •
edited
Loading

Does openPMD-api have those libraries in the interface?

No, but this does not fix the problem, which occurs not at an API level, but at an ABI level. The current setup means that both libopenPMD.so and picongpu will internally have symbols compiled into them from the same header-only library in possibly different versions. The usual approach to work around such problems for header-only libraries seems to be the use of inline namespace to qualify the compiled symbols with a version number, but neither toml11 nor nlohmann_json do this. Hence, the same symbols will exist with different meaning in both binaries, which motivates the loader to do curious things.

The most instantly applicable workaround seems to be patching the internally shipped dependencies with inline namespaces, see #5367. I'm investigating a more long-term solution on openPMD/openPMD-api#1757. Might also open PRs to add inline namespaces to toml11 and nlohmann_json..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@franzpoeschel @chillenzer