[question] Conan 2 lockfiles: How-to lock pkgid and prev?

[SzBosch]

Dear Conan team,

earlier with Conan 1 a lockfile contained the full reference incl. pkgid and prev.

With Conan 2.0.x this information is not anymore in the lockfile by default. But you provided a parameter "--lockfile-packages" which brought back this information.

Now with newer Conan versions (tried 2.4.1 and 2.6.0) this parameter is gone from the documentation, but can be still given on the command line, but giving an (non-breaking) error, e.g.:

conan lock create --requires="mypkg/1.0.0" --lockfile-packages
ERROR: The --lockfile-packages arg is private and shouldn't be used
...

What is the correct way to get a full lockfile incl. pkgid and prev?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @SzBosch

Thanks for your question.

Lockfiles in Conan 2 do not lock the package_id and package-revisions for 2 reasons:

• the package_id is now a mathematical and exact function of the inputs. If the exact versions and recipe revisions are locked (which they are in a lockfile) and the same input profiles are given, it will always result in the same package_id for the packages. Note that one of the goals of lockfiles in Conan 2 is that the same lockfile can be used for multiple binary configurations/profiles. So it doesn't make much sense to lock a specific package_id in these lockfiles.
• Every recipe-revision+package_id combination should have one and only one package-revision. Package revisions haven't been fully removed from the model to allow tracking of possible errors, but having more than one package revision for the same package-id is considered in Conan 2 a model or process error, meaning that a binary that was already existing was incorrectly re-built from source and overwritten. While having recipe-revisions is very natural, because there can be changes to source code including recipes conanfile.py that doesn't bump the version, re-building from source a binary that already exist shouldn't be done.

The --lockfile-packages was just an internal experiment to check the potential extensibility of the current model, just in case, but this has finally not been necessary.

Please let me know if this clarifies your question.

[SzBosch]

Hi @memsharded,

thank you for the quick answer.

What I basically understood now is earlier (e.g. as explained here: https://blog.conan.io/2019/09/27/package-id-modes.html) the pkgid was only calculated from settings, options and requirements.
The prev was calculated from the content (files)

Now it has changed to the pkgid is calculated from settings, options, requirements AND the content (files), correct?

With which version has this exactly changed?
(with 2.0.x we are still able to create multiple prevs for the same pkgid)

[memsharded]

Now it has changed to the pkgid is calculated from settings, options, requirements AND the content (files), correct?

No, the package_id is still calculated from settings, options and requirements, and the package_revision is calculated from the package contents, this is why it is possible to build more than one package revision, because unfortunately C++ builds are not always deterministic: https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html

But Conan 2 assumes that the configuration is correctly defined. If the source input is the same (same recipe-revision), and the configuration is also exactly the same (same profile inputs, settings, options, conf and dependencies) producing the same package_id, the resulting binary should be the same (modulo the non-determinism). Consequently, there is no reason to re-build a package binary when there already exist a package revision for the same recipe-revision+package_id, and as a result, a correct model and process will never have more than one package revision for that same combination.

[SzBosch]

Unfortunately it seems very easy for people (and even in automated CI/CD cases) to screw up this model by specifying a version in the conanfile.py, executing "conan export-pkg" and "conan upload", then modifying a file (which is not recipe relevant), forget to change the version in the conanfile.py and execute again "conan export-pkg" and "conan upload" resulting in an additional prev for the same version, rrev and pkgid.

Is there any possibility with conan to prevent or at least detect this issue?

[memsharded]

Unfortunately it seems very easy for people (and even in automated CI/CD cases) to screw up this model by specifying a version in the conanfile.py, executing "conan export-pkg" and "conan upload", then modifying a file (which is not recipe relevant), forget to change the version in the conanfile.py and execute again "conan export-pkg" and "conan upload" resulting in an additional prev for the same version, rrev and pkgid.

The first important recommendation, as documented in https://docs.conan.io/2/knowledge/guidelines.html, is that developers shouldn't be able to upload directly to production repos. Of course they can upload to "playground"-like repos, but not to production repos.

Then, for CI, it is way more unlikely to screw it up. In the first place, they are not locally modifying files at all, not using local flow with conan export-pkg, not exporting and uploading several times. When using conan create it is as simple as specifying conan create . --build=missing and it will avoid building new prevs for already existing binaries. Also, it is not that usual that the modified files are not related to the recipe at all. It might happen that some changes are done in the README, for example, but if using revision_mode = "scm", or using the scm capture mode (https://docs.conan.io/2/examples/tools/scm/git/capture_scm/git_capture_scm.html) it will still use the commit information, which would be a different commit anyway even if the changes are not "exported" with the recipe.

So we haven't seen issues with this in practice, and basically all occurrences of this situation happened because the process was "over-building" with some --build="*" or conan create without --build=missing and without any analysis of the changes.

It is important to note that one of the reason this was dropped in Conan 2 is because even if this an "inefficiency" and users might be unnecessarily rebuilding from source and wasting resources, the resulting binaries or the new package-revisions are basically the same binaries, just with some minor differences like embedded timestamps that results in a new package-revision. So defaulting to always using the latest package-revision is actually very safe, all the previous package-revisions were actually the same binary, so no need to lock it.

Is there any possibility with conan to prevent or at least detect this issue?

Detecting these inefficiencies is relatively easy, doing a conan list pkg/version:*#* --format=json > pkg_revisions.json will result in a json file that can be checked for multiple package revisions.

[johan-boule]

Thanks for the questions and explanations. This thread would make a good documentation. This is all clear.
It seems there's a confusion when one says the package id doesn't contain the package contents.
I guess for many people, the term "package" includes the source code, while in Conan's terminology it may sometimes refer exclusively to the produced binaries. Maybe also the absence of actual source code from Conan center repo, invites the Conan team to lean towards that terminology.

[memsharded]

Thanks for the feedback @johan-boule

If @SzBosch also thinks it is more clear now, I can move it to the docs repo to add there some further explanations.

[d3matt]

@memsharded (sorry to resurrect a months old issue) if multiple binary packages are really to be avoided... Could conan upload default to not upload binary packages that match something already uploaded? I'm nervous that there's a bit of a hole here for people who have not setup CI for uploading conan packages but still want to have dependency level reproducibility of a build from a conan lock file. With the v1 lock files, I had a guarantee that I would get the exact same binaries every time for my dependencies.

[memsharded]

Hi @d3matt and all,

Yes, sorry this has fallen through the cracks, sometimes it happens when things are too busy.
Yes, we have been wanting to add something to improve the "protection" to avoid this situation, but we couldn't prioritize it yet. Thanks for your ping, I hope we can have a look to that soon.