[BUG] pip install python-augeas fails with setuptools 80.7.0 or later

[mgedmin]

setuptools version

80.9.0

Python version

3.10.12

OS

Ubuntu 22.04 LTS

Additional environment information

No response

Description

I have a daily Jenkins job that tests my Ansible playbooks. It started failing on May 15 by failing to pip install certbot certbot-augeas on an Ubuntu 22.04 LTS VM. I've reduced the failing test case to this command:

pip cache purge && rm -fr /tmp/env && virtualenv /tmp/env && /tmp/env/bin/pip install -U setuptools && /tmp/env/bin/pip install python-augeas

May 14 is the release date of setuptools 80.7.0. I've verified that if I pip install 'setuptools < 80.7.0', the failure goes away, but if I pip install setuptools==80.7.0, the failure reappears.

The setup.py in python-augeas looks a bit old-fashioned, but doesn't seem to be using any fancy features? It uses the deprecated test_suite, but I don't think that could cause python setup.py egg_info to fail.

For some reason the failure only shows up on Ubuntu 22.04, while the same Ansible playbook (and my reproducer one-liner) runs fine against an Ubuntu 24.04 LTS VM (with Python 3.12.3).

Expected behavior

I expect pip install certbot-apache to succeed.

How to Reproduce

virtualenv env && env/bin/pip install -U setuptools && env/bin/pip install python-augeas

Output

ansible@jammy:/tmp$ virtualenv env && env/bin/pip install -U setuptools && env/bin/pip install python-augeas
created virtual environment CPython3.10.12.final.0-64 in 105ms
  creator CPython3Posix(dest=/tmp/env, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ansible/.local/share/virtualenv)
    added seed packages: pip==22.0.2, setuptools==59.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Requirement already satisfied: setuptools in ./env/lib/python3.10/site-packages (59.6.0)
Collecting setuptools
  Downloading setuptools-80.9.0-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 7.3 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Uninstalling setuptools-59.6.0:
      Successfully uninstalled setuptools-59.6.0
Successfully installed setuptools-80.9.0
Collecting python-augeas
  Downloading python-augeas-1.2.0.tar.gz (99 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.4/99.4 KB 1.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [52 lines of output]
      /tmp/env/lib/python3.10/site-packages/setuptools/__init__.py:92: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
      !!
      
              ********************************************************************************
              Requirements should be satisfied by a PEP 517 installer.
              If you are using pip, you can try `pip install --use-pep517`.
      
              By 2025-Oct-31, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
              ********************************************************************************
      
      !!
        dist.fetch_build_eggs(dist.setup_requires)
      /tmp/env/lib/python3.10/site-packages/setuptools/_distutils/dist.py:289: UserWarning: Unknown distribution option: 'test_suite'
        warnings.warn(msg)
      Traceback (most recent call last):
        File "/tmp/pip-install-n6pg4cb9/python-augeas_40492a9571144f39af4a1ca1a1ffafcb/.eggs/cffi-1.17.1-py3.10-linux-x86_64.egg/cffi/cparser.py", line 5, in <module>
          from . import _pycparser as pycparser
      ImportError: cannot import name '_pycparser' from 'cffi' (/tmp/pip-install-n6pg4cb9/python-augeas_40492a9571144f39af4a1ca1a1ffafcb/.eggs/cffi-1.17.1-py3.10-linux-x86_64.egg/cffi/__init__.py)
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-n6pg4cb9/python-augeas_40492a9571144f39af4a1ca1a1ffafcb/setup.py", line 16, in <module>
          setup(name=name,
        File "/tmp/env/lib/python3.10/site-packages/setuptools/__init__.py", line 115, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/env/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
          _setup_distribution = dist = klass(attrs)
        File "/tmp/env/lib/python3.10/site-packages/setuptools/dist.py", line 321, in __init__
          _Distribution.__init__(self, dist_attrs)
        File "/tmp/env/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 309, in __init__
          self.finalize_options()
        File "/tmp/env/lib/python3.10/site-packages/setuptools/dist.py", line 784, in finalize_options
          ep(self)
        File "/tmp/env/lib/python3.10/site-packages/setuptools/dist.py", line 804, in _finalize_setup_keywords
          ep.load()(self, ep.name, value)
        File "/tmp/pip-install-n6pg4cb9/python-augeas_40492a9571144f39af4a1ca1a1ffafcb/.eggs/cffi-1.17.1-py3.10-linux-x86_64.egg/cffi/setuptools_ext.py", line 216, in cffi_modules
          add_cffi_module(dist, cffi_module)
        File "/tmp/pip-install-n6pg4cb9/python-augeas_40492a9571144f39af4a1ca1a1ffafcb/.eggs/cffi-1.17.1-py3.10-linux-x86_64.egg/cffi/setuptools_ext.py", line 49, in add_cffi_module
          execfile(build_file_name, mod_vars)
        File "/tmp/pip-install-n6pg4cb9/python-augeas_40492a9571144f39af4a1ca1a1ffafcb/.eggs/cffi-1.17.1-py3.10-linux-x86_64.egg/cffi/setuptools_ext.py", line 25, in execfile
          exec(code, glob, glob)
        File "augeas/ffi.py", line 19, in <module>
          ffi = FFI()
        File "/tmp/pip-install-n6pg4cb9/python-augeas_40492a9571144f39af4a1ca1a1ffafcb/.eggs/cffi-1.17.1-py3.10-linux-x86_64.egg/cffi/api.py", line 65, in __init__
          from . import cparser
        File "/tmp/pip-install-n6pg4cb9/python-augeas_40492a9571144f39af4a1ca1a1ffafcb/.eggs/cffi-1.17.1-py3.10-linux-x86_64.egg/cffi/cparser.py", line 7, in <module>
          import pycparser
      ModuleNotFoundError: No module named 'pycparser'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[abravalheri]

Based on the trace it seems that the problem does not happen exactly when installing python-augeas but more specifically when installing (or not installing) the build-time dependency on cffi>=1.0.0.

Could you please try to install that package first to verify if it is really the source of the problems?

python-augeas does not seem to define a pyproject.toml so that may be triggering pip to use the (deprecated) setup.py install instead of simply calling the relevant up-to-date APIs. This is a complicated path because the visibility about reproducible steps is not great: pip bases the choice of the procedure on the state of the environment. So I would also recommend running with pip install --use-pep517 to improve reproducibility. This is also indicated in the warning notice.

[mgedmin]

$ pip cache purge && rm -fr /tmp/env && virtualenv /tmp/env && /tmp/env/bin/pip install -U setuptools cffi && /tmp/env/bin/pip install python-augeas

succeeds so yes, this is something about setup_requires failing to install cffi correctly.

pip cache purge && rm -fr /tmp/env && virtualenv /tmp/env && /tmp/env/bin/pip install -U setuptools && /tmp/env/bin/pip install --use-pep517 python-augeas

also succeeds, and is probably a better workaround than my current pinning of an older setuptools version.

The warning notice spoke about "setuptools.installer" and "fetch_build_eggs" which are some setuptool-internal names not even mentioned in python-augeas's setup.py. I didn't realize it meant this applied to setup_requires.. In retrospect it was stupid of me to ignore the entire notice because of that.

TBH I have no idea what "Use PEP 517 for building source distributions" means in practical terms. I foresee a lot of documentation reading in my future.

So if I understand everything correctly, the ball is now in python-augeas's court to switch from setup_requires to a pyproject.toml? I should try to file a bug.

[abravalheri]

PEP 517 introduced a series of "build APIs" and formalized a process/workflow on how they should be invoked to create a wheel/sdist. This was meant as an interoperability standard for Python packages (at least the ones installable via pip).

This replaced the pre-2015 era of running python setup.py ....
As a result Setuptools deprecated the CLI-driven workflow (to avoid maintaining 2 separated interfaces).

---

The warning notice spoke about "setuptools.installer" and "fetch_build_eggs" which are some setuptool-internal names not even mentioned in python-augeas's setup.py.

setup_requires is kind of relic of the old times when Setuptools was trying to be an installer. It's initial implementation relied on these internal names, as well as easy_install and eggs.

v80.7.0 started the process of removal of installer and some internal pieces because they have been troublesome for a while (and there was a couple of instances that social engineering could be used to architect vulnerabilities, if I remember correctly).

What Setuptools did to keep setup_requires working in the modern context was to forward the list of dependencies to the actual installer during one of the calls to the Build APIs (as specified in the PEP 517 workflow). This side stepped the problems with installer/easy_install and allowed setup_requires to keep working.

---

The problem arises when packages are installed using the pre-2015 workflow of running python setup.py ... as a CLI, instead of the standard build API. Now this may happen without the user knowing because pip uses heuristics to try to guess if a package is a "legacy package"¹ or if it can be installed using the new APIs. This process depends on the state of the environment and it is not very transparent and to the end user how to make it more reproducible.

2 things that will force pip to use the standard build APIs are:

• --use-pep517
• The presence of a pyproject.toml file at the root of the package.

Note that the use of build APIs is supposed to become the default behaviour, but pip has been postponing this transition for a while to avoid disruption in the community. So user intervention may be needed for some edge packages that don't use pyproject.toml yet.

---

In general there is no need to fully switch configurations from setup.py to pyproject.toml. But the mere presence of the file will make pip behave more consistently and always use the build APIs instead of the legacy workflow.

Footnotes

1. Or more precisely, a package whose build dependencies are not declared and that implicitly rely on the state of the currently activated environment and the packages that have already been installed. ↩