Invoke python -m venv from a different working directory (#1698)

When `python -m venv` is run, after creating the virtual environment it
invokes `ensurepip` to install pip.

The `ensurepip` module in older Python versions didn't correctly run its
Python subprocesses in isolated mode, meaning that the working directory
is added to `sys.path`. This can cause issues if the app's build
directory contains files/directories that shadow expected package names
(such as a `brotli` directory).

For example:

```
-----> Installing Poetry 1.8.4
Error: Command '['/tmp/codon/tmp/cache/.heroku/python-poetry/venv/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

 !     Internal Error: Unable to create virtual environment for Poetry.
 !
 !     The 'python -m venv' command to create a virtual environment did
 !     not exit successfully.
 !
 !     See the log output above for more information.
```

The best fix is for apps to upgrade to newer Python patch versions
(3.8.14+, 3.9.14+, 3.10.6+), since they include the upstream
`ensurepip` fix as well as many other bug and security fixes.

However, to ensure venv creation still works on these older Python
versions, as a workaround we can run the `python -m venv` command from a
different working directory (as an alternative to isolated mode).

Fixes #1697.
GUS-W-17215816.

Author: edmorley

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## [Unreleased]
 
 - Deprecated using Pipenv without a lockfile (`Pipfile.lock`). ([#1695](https://github.com/heroku/heroku-buildpack-python/pull/1695))
+- Fixed Poetry venv creation when using an outdated Python version whose `ensurepip` module doesn't enable isolated mode, and the app's build directory contents shadows a package imported by pip (such as `brotli`). ([#1698](https://github.com/heroku/heroku-buildpack-python/pull/1698))
 
 ## [v266] - 2024-11-08
 

lib/poetry.sh

@@ -37,6 +37,7 @@ function poetry::install_poetry() {
 
 		# The Poetry directory will already exist in the relocated cache case mentioned above.
 		rm -rf "${poetry_root}"
+		mkdir -p "${poetry_root}"
 
 		# We can't use the pip wheel bundled within Python's standard library to install Poetry
 		# (which would allow us to use `--without-pip` here to skip the pip install), since it
@@ -45,7 +46,10 @@ function poetry::install_poetry() {
 		# are still using outdated patch releases of those Python versions, whose bundled pip
 		# can be older (for example Python 3.9.0 ships with pip v20.2.1). Once Python 3.10 EOLs
 		# we can switch back to the previous approach since Python 3.11.0 ships with pip v22.3.
-		if ! python -m venv "${poetry_venv_dir}"; then
+		# Changing the working directory away from the build dir is required to work around an
+		# `ensurepip` bug in older Python versions, where it doesn't run Python in isolated mode:
+		# https://github.com/heroku/heroku-buildpack-python/issues/1697
+		if ! (cd "${poetry_root}" && python -m venv "${poetry_venv_dir}"); then
 			output::error <<-EOF
 				Internal Error: Unable to create virtual environment for Poetry.
 

spec/fixtures/poetry_oldest_python/brotli/.gitkeep

@@ -0,0 +1,2 @@
+The brotli directory tests the workaround for an `ensurepip` bug in older Python versions:
+https://github.com/heroku/heroku-buildpack-python/issues/1697

spec/hatchet/poetry_spec.rb

@@ -185,8 +185,10 @@
     end
   end
 
-  # This checks that the Poetry bootstrap works even with older bundled pip, and that
-  # our chosen Poetry version also supports our oldest supported Python version.
+  # This checks that the Poetry bootstrap works even with older bundled pip, and that our
+  # chosen Poetry version also supports our oldest supported Python version. The fixture
+  # also includes a `brotli` directory to test the workaround for an `ensurepip` bug in
+  # older Python versions: https://github.com/heroku/heroku-buildpack-python/issues/1697
   context 'when using the oldest supported Python version' do
     let(:app) { Hatchet::Runner.new('spec/fixtures/poetry_oldest_python') }