[feature] Use PKG_CONFIG_LIBDIR and PKG_CONFIG_SYSROOT_DIR

[jwillikers]

What is your suggestion?

Working on cross-compilation problems with libselinux in conan-io/conan-center-index#22952, I discovered that there's another pkg-config environment variable for configured the search paths PKG_CONFIG_LIBDIR. Apparently, PKG_CONFIG_LIBDIR is for setting the primary search paths while PKG_CONFIG_PATH is for setting the secondary search paths. In the case of libselinux, it appears that setting PKG_CONFIG_LIBDIR fixes issues with system directories being picked up for the build when cross-compiling.

libselinux uses an ad-hoc Makefiles-based build system.
It locates the necessary compiler flags for the pcre2 dependency by running pkg-config --cflags libpcre2-8.
When cross-compiling and using the default PKG_CONFIG_PATH variable to point to Conan's generators directory, pkg-config locates the system's pcre2 pkg-config file and adds -I/usr/include to the compiler flags, breaking cross-compilation.
The next thing I tried was to set PKG_CONFIG_SYSROOT_DIR to the path to my sysroot directory.
This resulted in pcre2 being picked up from my sysroot and compiling without error, although it still should have been using the pcre2 package from Conan.
Finally, I set PKG_CONFIG_LIBDIR appropriately and viola, pkg-config found Conan's pcre2 package as expected.

It occurred to me that Conan's handling of these environment variables could be greatly improved, especially since it only works with PKG_CONFIG_PATH. Here are the improvements I think that could be made. Note that handling of these variables might be better handled in the build system. Meson already makes several accommodations for these variables, such as:

Since 0.54.0 The pkg_config_libdir property may point to a list of path used internally by Meson to set the PKG_CONFIG_LIBDIR environment variable for pkg-config. This prevents pkg-config from searching cross dependencies in system directories.

Setting sys_root in the [properties] section of your cross file will now set PKG_CONFIG_SYSROOT_DIR automatically for host system dependencies when cross compiling.

I think that it would make sense for Conan to follow suite for the pkg-config sysroot directory by doing the following:

1. Define sys_root according to the sysroot conf variable in the Meson cross file when it is defined.
2. For the AutotoolsToolchain, define PKG_CONFIG_SYSROOT_DIR automatically to the value of the sysroot conf option.

The handling of the PKG_CONFIG_LIBDIR variable is more complicated in that user's might want to incorporate system pkg-config files, especially when using platform_requires. However, the PKG_CONFIG_LIBDIR variable has a huge advantage over PKG_CONFIG_PATH in that it has precedence. I think it makes more sense to use the PKG_CONFIG_LIBDIR environment variable for AutotoolsToolchain and pkg_config_libdir option for Meson to point to Conan's generated pkg-config files. This is important to ensure that the provided Conan packages are always preferred over external pkg-config files.

This still leaves some gaps though, such as the fact that PKG_CONFIG_LIBDIR overwrites the the system pkg-config directories. This will break consumers relying on external pkg-config files. However, I think this is advantageous since it would require explicitly opting-in to using pkg-config files not provided by Conan which should improve reproducibility when creating Conan packages. To accommodate consumers and recipes wishing to use system packages, it would make sense to allow overriding the default behavior. For consumers, this would most likely be done via a conf value that allows adding paths to PKG_CONFIG_PATH. Again, it's important that Conan packages are always preferred and using platform_requires will ensure that Conan doesn't generate any pkg-config files for a given recipe, thus allowing the external pkg-config files to be picked up. System pkg-config search paths could also be added to PKG_CONFIG_PATH so that consumers are not required to add this path explicitly.

I'm pretty sure there are recipes that use system packages implicitly. If system include directories were to not be added to PKG_CONFIG_PATH, then recipes would probably want a convenience method to opt in to having the system pkg-config search paths added to PKG_CONFIG_PATH.

The PKG_CONFIG_SYSROOT_DIR is prepended to every path defined in PKG_CONFIG_PATH, so handling PKG_CONFIG_SYSROOT_DIR with Conan will automatically correct system include directories, like /usr/include, in PKG_CONFIG_PATH for pkg-config.

I think that better handling of these variables can significantly improve reproducibility while reducing the barrier of entry for those cross-compiling packages.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[jcar87]

Have managed to reproduce some issues related to this - to see how we can best accommodate:

• a sysroot
• that may or may not have .pc files to stand-in for system dependencies
• .pc files generated by Conan

There are a few challenges, namely:

• all the documentation I can find about pkg-config in cross-compilation environments is to have either a separate pkg-config or a wrapper script that would have it locating things only inside the sysroot (https://autotools.info/pkgconfig/cross-compiling.html) - this is good for locating .pc files inside the sysroot (e.g. system dependencies) - but not so good to locate the .pc files generated by Conan (which are outside the sysroot)
• if the cross-building g++ and linker are correctly configured for the target platform, pkg-config should not report the sysroot paths in the flags (i.e if libfoo.so is in the sysroot and is not meant to be provided by conan, the only flag should be -lfoo , without any-I and no -L - I'm not currently observing this. This can be a problem because if the sysroot contains things that are also provided by Conan, we would expect conan to take precedence, but pkg-config may end up altering this order
• unsure what happens when one needs to locate .pc files in the build machine for executables - we can have different behaviour for the host and build profiles

I have this working more or less but meson seems to be doing some additional path prefixing that makes this not work at all - investigating

other comments:

Apparently, PKG_CONFIG_LIBDIR is for setting the primary search paths while PKG_CONFIG_PATH is for setting the secondary search paths. In the case of libselinux, it appears that setting PKG_CONFIG_LIBDIR fixes issues with system directories being picked up for the build when cross-compiling.

However, the PKG_CONFIG_LIBDIR variable has a huge advantage over PKG_CONFIG_PATH in that it has precedence.

My understanding is that PKG_CONFIG_PATH is searched first, and then it falls back to PKG_CONFIG_LIBDIR, which, if undefined, uses the values pkg-config was compiled with.
See https://bugs.freedesktop.org/show_bug.cgi?id=88992

This resulted in pcre2 being picked up from my sysroot and compiling without error, although it still should have been using the pcre2 package from Conan.
This is indeed the biggest challenge at the moment

pkg-config locates the system's pcre2 pkg-config file and adds -I/usr/include to the compiler flags, breaking cross-compilation.

depending on how it is configured pkg-config may locate .pc files in the host - but I'm susprised it would emit -I/usr/include - pkg-config erases default system paths for good reasons - these are consisdered "system directories", in the sense that the compiler and linker will already look there and additional flags are not necessary: https://gitlab.freedesktop.org/pkg-config/pkg-config/-/blob/master/pkg.c?ref_type=heads#L817-838

[jwillikers]

Probably worth noting that pkgconf and pkg-config aren't 1:1 in how they handle everything. Also, I think issue #16468 is related, particularly with sysroot handling in Meson. I'm linking to it for additional context.