some python packages include unintended directories in site-packages/

[smoser]

In trying to build mult-version version of py3-proto-plus I found that it could not be installed along side of py3.12-google-resumable-media, because they both had usr/lib/python3.12/site-packages/testing/constraints-3.10.txt.

That didn't seem right.

I used https://gist.github.com/smoser/0a11e2643b884960c1e5349d4dc0b8c7#file-get-archive-info to get .flist files (tar -tvf output) of all the files in the wolfi to see what other packages had that problem.

#!/usr/bin/gawk
# print any the top level entries in usr/lib/python3.XX/site-packages
# that do not match dist-info (python module info) or __pycache__
BEGINFILE { delete tld }
$6 ~ /usr.lib.python3[.][0-9]+.site-packages\/[^/]*$/ &&
   $6 !~ /dist-info/ &&
   $6 !~ "__pycache__" {
    split($6, r, /\//)
    tld[r[5]]=1
}
ENDFILE { 
  if (length(tld)) {
      buf = ""
      for (key in tld)  {
          buf = buf " " key
      }
      printf("%02d %s%s\n", length(tld), FILENAME, buf)
  }
}

Using the above awk, here are some interesting bits:

$ awk -f my.awk py3.*.flist  | grep testing
03 py3.12-google-resumable-media-2.7.2-r1.flist google build testing
04 py3.12-proto-plus-1.25.0-r0.flist docs build testing proto

$ awk -f my.awk py3*.flist  | grep docs
04 py3.12-google-auth-oauthlib-1.2.1-r1.flist scripts docs build google_auth_oauthlib
04 py3.12-pipenv-2024.3.1-r0.flist docs build examples pipenv
04 py3.12-proto-plus-1.25.0-r0.flist docs build testing proto

$ awk -f my.awk py3*.flist  | grep -w build
01 py3.12-build-1.2.2-r1.flist build
02 py3.12-codespell-2.3.0-r1.flist build codespell_lib
02 py3.12-google-auth-2.36.0-r0.flist google build
04 py3.12-google-auth-oauthlib-1.2.1-r1.flist scripts docs build google_auth_oauthlib
03 py3.12-google-resumable-media-2.7.2-r1.flist google build testing
02 py3.12-googleapis-common-protos-1.65.0-r1.flist google build
04 py3.12-pipenv-2024.3.1-r0.flist docs build examples pipenv
04 py3.12-proto-plus-1.25.0-r0.flist docs build testing proto
02 py3.12-python-gitlab-5.0.0-r0.flist build gitlab
01 py3.12-scikit-build-0.18.1-r1.flist skbuild
01 py3.12-scikit-build-core-0.10.7-r1.flist scikit_build_core
02 py3.12-tqdm-4.66.6-r0.flist build tqdm

So we can see there that some things are providing top level python modules that they surely did not intend to.

I opened googleapis/proto-plus-python#503 , but haven't gotten to the root cause of why those directories are getting pulled in to the installed package. The 'build' directory is output of a previous 'pip build' run, and that can be cleaned out with an 'rm' in the pipeline, but the other directories should not be getting in either.

[smoser]

I'm attaching a modified version of the script I attached googleapis/proto-plus-python#503 .
do-build.sh.txt.

When building with:

git clone https://github.com/googleapis/proto-plus-python.git
cd proto-plus-python
rm -Rf *
git checkout .

sh ../do-build.sh.txt 

If I build in a clean repo (rm -Rf *; git checkout .), and include setuptools-scm in the packages list then I get the docs/ and testing/ dirs in the .whl. If I exclude it, then I will not get that.

[smoser]

Maybe related to pypa/setuptools-scm#561

[smoser]

OK. @pnasrat I'm interested in your thoughts on what we can do here.
It does seem like setuptools_scm is changing the behavior and resulting in the 'docs' and 'testing/' packages getting added.

The 'build/' is easy enough, we can just explicitly clean that dir out in py/pip-build-install, and I've verified that does work, but the non-'build' directories we'll have to handle another way.

[smoser]

#33397 is up with one solution. It works , but is admittedly kind of hacky.
@pnasrat suggested that we should remove setuptools-scm from the build base. We absolutely can do that.

Some options we have are:

1. remove setuptools-scm - a pretty big change and we would have to add some checks to verify that version was getting set correctly or we would likely miss some packages that needed setuptools-scm in the root.
2. some solution like seen in some python packages include unintended directories in site-packages/ #33397 - either way, the 'build/' cleanup bit there does seem like we should land.
3. get a fix into setuptools-scm

[xnox]

I think it is universally known that python packages have or create top-level modules test/ tests/ testing/ and so on, and many of them ship those by default.

This is why pybuild in Debian tries to never run setup.py / build / install from the top-level src dir which is often polluted with those and prefers to create a new directory and run build / install steps there. Such that, usually, it is without the extraflurious modules which shouldn't be installed as public ones.

Blanket banning them is probably the right way forward as all of them periodically grow top-level tests module.

[smoser]

I think it is universally known that python packages have or create top-level modules test/ tests/ testing/ and so on, and many of them ship those by default.

well, we have 580 packages that do this same dance. only 11 of them had a problem. And those are fixable by removing packages from their build environment that they did not expect. So maybe your info is a bit dated.

We can absolutely change how the pipeline works, but it has to work for all of those packages. imo, It has worked extremely well as a single "just do the right thing".

I'm also all for testing that things do not include such top level modules. We have mmelange SCA that warns about multiple top level modules but no one reads warnings. We could make that fail and require package info to allow it for the few cases where it is needed.

[pnasrat]

I suspect those 11 are older setuptools which does have default exclusions for the flat file finder see

https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#flat-layout

'tests', 'tests.*', 'unit_test', 'unit_test.*', 'unit_tests', 'unit_tests.*',

[pnasrat]

@smoser which 11 packages? I suspect there is something wrong with our invocation and adding custom prevent-inclusion feels like a work around not getting to the root cause