RFC: add an experimental bindeps backend

[cognifloyd]

Would anyone be interested in a bindeps backend that can analyze binaries to list which shared libs it is linked with?

The bindeps backend

The bindeps backend would be functionally similar to dpkg-shlibdeps in debian, and rpmdeps+elfdeps in EL distros. But, it would not require the native packaging tooling to run it. I imagine it would be most useful when using the nfpm backend (which wraps nFPM) to create native system packages (.deb, .rpm, ...) because nfpm does not analyze included binaries like the native deb/rpm tooling does. But it could also be useful to audit/lint generated binaries before distributing them.

What I have so far:

So far, I have code that can

• take a pex_binary,
• extract its wheels (using pex-tools of course), and
• use elfdeps 📦 (the python lib, not the native rpmbuild binary) to get the list of SONAMEs used by any binaries in the package.

I also have code that can take the list of SONAMEs and map them to deb package names using the debian package search API (for debian or ubuntu) instead of querying via native packaging tool(s) like dpkg -S. This is important because deb depends expects package names, not SONAMEs. This package lookup is not needed for rpm where requires takes a list of SONAMEs in addition to package names.

For the rules I've written so far, the only python-specific bits handle extracting wheels from a pex_binary and using zipfile to make the wheel contents available for elfdeps to inspect. The actual elf dependency inspection logic and the deb package search are not python-specific and could easily be used to inspect binaries from other pants package targets

Note

I looked at a variety of python libs before settling on elfdeps. For example, lddcollect, uses dpkk -S to map SONAME to deb package name. But that would require running the tool on every target distro version (in a docker container or similar) instead of just running on whatever host pants happens to be running on. I also looked at using parts of auditwheel, but that is too python-specific and it abstracts away some of the underlying elf metadata (from pyelftools 📦) I needed to generate package dependency metadata. auditwheel might still be very useful (see below), but not for imitating dpkg-shlibdeps or rpmdeps.

Potential scope of the bindeps backend:

I'm calling this a bindeps backend, because it does not need to be specific to elf binaries/libs. In figuring out how best to analyze linux wheels, I also found these projects which would fit nicely under a bindeps backend (though I haven't written the code to make them part of a pants backend) - perhaps as check or fix tools that take packaged binaries as input:

• These 3 can analyze a wheel's included python extensions (binaries / shared libs) and "repair" the wheel by copying the shared lib(s) into a copy of the wheel so that no system deps are required.

  • delocate 📦: this is MacOS-specific, so it works with dynamic libraries. It can also combine an arm64 wheel with an x86_64 wheel to make a universal2 wheel that supports both architectures.
  • auditwheel 📦: this is linux-specific, so it works with ELF binaries / .so files. It can also audits the wheels' manylinux tag, if any, to ensure that the wheel follows the system dep requirements to use that manylinux tag.
  • delvewheel 📦: this is Windows-specific, so it works with DLLs.

• repairwheel 📦: This lib combines the 3 above libs, (so it is cross-platform) focusing on using pure python libs instead of system binaries to modify any libs in the wheel. It also promises to be more host-agnostic so that it can hermetically create the same changes on any host platform.

• abi3audit 📦: Another wheel audit tool. This one audits a wheel's abi3 tag to see if python extensions in the wheel only use python symbols stabalized in the target python version or earlier.

Request for feedback

1. Does this sound generally useful? If so, which part(s) sound useful?
2. Does the bindeps name make sense for a backend of this scope? At first, I thought I would use elfdeps, but that doesn't leave room for cross-platform expansion in the future.
3. Is there any interest in the creation of pants.backend.experimental.bindeps?

[cognifloyd]

Targets that a backend like bindeps could potentially analyze (organized by language/ecosystem):

• misc:

  • archive
  • file || files
  • makeself_archive
  • resource || resources
  • system_binary for giggles? probably not very helpful

• go:

  • go_binary

• java (& friends):

  • deploy_jar
  • jvm_war

• javascript:

  • node_package
  • npm_distribution
  • package_json

• python:

  • pex_binary || pex_binaries
  • python_distribution

I suppose it could also do third party requirements (go_third_party_package, jvm_artifact, node_third_party_package, pipenv_|poetry_|python_|uv_requirements), but I suspect that would involve more logic to (trigger a rule to) download it before analyzing it.

[cburroughs]

I think this sounds interesting!

Some possible use cases:

• Tell me which native libraries are required by a pex_binary that are not bundled with an included wheel.
• ^^, but fail some goal if any native libraries outside some filter list are required.
• Maybe, some sort of repair-all-and-repackage on a pex_binary?

I think I would use the first two, but skeptical I would actually do anything so magical as the last one

Does the bindeps name make sense for a backend of this scope?

It might be a little terse for something on the esoteric side but native_shared_library_deps is a bit long?

[cognifloyd]

I think this sounds interesting!

Oh good. I was afraid I might be the only interested person in the pants community.

Some possible use cases:

• Tell me which native libraries are required by a pex_binary that are not bundled with an included wheel.

Hmm. This sounds like a new goal, probably with shared rules that can also be invoked by the check goal.

• ^^, but fail some goal if any native libraries outside some filter list are required.

Ooh. This sounds similar to __dependencies_rules__. Maybe __packaging_rules__? Other backends that wrap security scanners could probably tie into such a thing too.

• Maybe, some sort of repair-all-and-repackage on a pex_binary?

I think I would use the first two, but skeptical I would actually do anything so magical as the last one

I was shocked to discover tools that auto-add libs in modified wheels like auditwheel, delocate, delvewheel, and repairwheel. Apparently that kind of thing is popular in some circles like data science.

Like you, I hesitate to use these auto-repair features, at least in an open source project, because then you're embedding the lib, not just using it, which could cause licensing conflicts. If it's only for internal deployment not external distribution, there probably aren't (as many?) licensing issues though.

Does the bindeps name make sense for a backend of this scope?

It might be a little terse for something on the esoteric side but native_shared_library_deps is a bit long?

Yeah. I try for shorter names, and bindeps might be too short. But native_shared_library_deps feels a bit long. Hmm. Gotta love naming things.

[cognifloyd]

• Tell me which native libraries are required by a pex_binary that are not bundled with an included wheel.

Hmm. This sounds like a new goal, probably with shared rules that can also be invoked by the check goal.

Oh! I misunderstood your suggestion. That could include a new goal, but it could also just be an additional source of binaries/libraries to analyze when generating package dependencies. So far, I use pex-tools to extract the wheels from a pex. How can I extract anything from a pex that is not a wheel? Build a venv out of it (can pex build a venv for a foreign platform to facilitate static analysis)?

edit: I've got support for this now. Sadly, I have to unzip the pex to get the other files. The zip format and folder structure in the pex feels like implementation details that I should not be relying on.