virtualenvs.prefer-active-python is not respected

[haizaar]

Description

Good day,
I'm trying to make poetry use the currently active python provide by nix-shell but it still picks the one it was built with despite virtualenvs.prefer-active-python = true:

❯ poetry config --list
cache-dir = "/home/haizaar/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = true
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/haizaar/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = true
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true
❯ which python
/nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env/bin/python
❯ rm -rf .venv
❯ poetry shell
Creating virtualenv mediapipe-custom-hand-guesture-recogniser in /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv
Spawning shell within /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv
. /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/activate
❯ which python
/home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/python
❯ ls -lah `which python`
Permissions Size User    Group Date Modified Name
lrwxrwxrwx     - haizaar users  6 Apr 14:43  /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/python -> /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11

You can see that the virtual env uses python /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11 while the active python was /nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env/bin/python.

/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11 is the Python poetry was built with:

❯ poetry debug info

Poetry
Version: 1.8.2
Python:  3.11.8

Virtualenv
Python:         3.11.8
Implementation: CPython
Path:           /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv
Executable:     /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/python
Valid:          True

Base
Platform:   linux
OS:         posix
Python:     3.11.8
Path:       /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8
Executable: /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11

Running poetry -vvv shell shows that it clearly fails to recognize currently active python properly:

❯ poetry -vvv shell
Loading configuration file /home/haizaar/.config/pypoetry/config.toml
Loading configuration file /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/poetry.toml
Trying to detect current active python executable as specified in the config.
Found: /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python

Workarounds

None I could find unfortunately.

Poetry Installation Method

System package manager

Operating System

Nixos 23.12

Poetry Version

1.8.2

Poetry Configuration

cache-dir = "/home/haizaar/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = true
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/haizaar/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = true
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

Platform: "linux-x86_64"
Python version: "3.11"
Current installation scheme: "posix_prefix"

Paths: 
        data = "/nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env"
        include = "/nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env/include/python3.11"
        platinclude = "/nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env/include/python3.11"
        platlib = "/nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env/lib/python3.11/site-packages"
        platstdlib = "/nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env/lib/python3.11"
        purelib = "/nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env/lib/python3.11/site-packages"
        scripts = "/nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env/bin"
        stdlib = "/nix/store/kbgznx7akvi869qnnmhks820rhmgn66j-python3-3.11.8-env/lib/python3.11"

...

Example pyproject.toml

[tool.poetry]
name = "mediapipe-custom-hand-guesture-recogniser"
version = "0.1.0"
description = "Custom hand guesture recognition using mediapipe"
authors = ["Kazuhito Takahashi", "Zaar Hai <hello@zaar.me>"]
license = "Apache 2.0"
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

❯ poetry -vvv shell
Loading configuration file /home/haizaar/.config/pypoetry/config.toml
Loading configuration file /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/poetry.toml
Trying to detect current active python executable as specified in the config.
Found: /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python
Creating virtualenv mediapipe-custom-hand-guesture-recogniser in /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv
[virtualenv] find interpreter for spec PythonSpec(path=/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11)
[virtualenv] proposed PythonInfo(spec=CPython3.11.8.final.0-64, exe=/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11, platform=linux, version='3.11.8 (main, Feb  6 2024, 21:21:21) [GCC 13.2.0]', encoding_fs_io=utf-8-utf-8)
[virtualenv] accepted PythonInfo(spec=CPython3.11.8.final.0-64, exe=/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11, platform=linux, version='3.11.8 (main, Feb  6 2024, 21:21:21) [GCC 13.2.0]', encoding_fs_io=utf-8-utf-8)
[virtualenv] filesystem is case-sensitive
[virtualenv] create virtual environment via CPython3Posix(dest=/home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv, clear=False, no_vcs_ignore=False, global=True)
[virtualenv] create folder /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin
[virtualenv] create folder /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages
[virtualenv] write /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/pyvenv.cfg
[virtualenv]    home = /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin
[virtualenv]    implementation = CPython
[virtualenv]    version_info = 3.11.8.final.0
[virtualenv]    virtualenv = 20.25.1
[virtualenv]    include-system-site-packages = true
[virtualenv]    base-prefix = /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8
[virtualenv]    base-exec-prefix = /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8
[virtualenv]    base-executable = /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11
[virtualenv] symlink /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11 to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/python
[virtualenv] create virtualenv import hook file /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/_virtualenv.pth
[virtualenv] create /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/_virtualenv.py
[virtualenv] ============================== target debug ==============================
[virtualenv] debug via /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/python /nix/store/vmx0wi2cbfn69vnfffssxkqv5pgdx1yx-python3.11-virtualenv-20.25.1/lib/python3.11/site-packages/virtualenv/create/debug.py
[virtualenv] {
[virtualenv]   "sys": {
[virtualenv]     "executable": "/home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/python",
[virtualenv]     "_base_executable": "/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11",
[virtualenv]     "prefix": "/home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv",
[virtualenv]     "base_prefix": "/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8",
[virtualenv]     "real_prefix": null,
[virtualenv]     "exec_prefix": "/home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv",
[virtualenv]     "base_exec_prefix": "/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8",
[virtualenv]     "path": [
[virtualenv]       "/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python311.zip",
[virtualenv]       "/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11",
[virtualenv]       "/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/lib-dynload",
[virtualenv]       "/home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages",
[virtualenv]       "/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/site-packages"
[virtualenv]     ],
[virtualenv]     "meta_path": [
[virtualenv]       "<class '_virtualenv._Finder'>",
[virtualenv]       "<class '_frozen_importlib.BuiltinImporter'>",
[virtualenv]       "<class '_frozen_importlib.FrozenImporter'>",
[virtualenv]       "<class '_frozen_importlib_external.PathFinder'>"
[virtualenv]     ],
[virtualenv]     "fs_encoding": "utf-8",
[virtualenv]     "io_encoding": "utf-8"
[virtualenv]   },
[virtualenv]   "version": "3.11.8 (main, Feb  6 2024, 21:21:21) [GCC 13.2.0]",
[virtualenv]   "makefile_filename": "/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/config-3.11-x86_64-linux-gnu/Makefile",
[virtualenv]   "os": "<module 'os' (frozen)>",
[virtualenv]   "site": "<module 'site' (frozen)>",
[virtualenv]   "datetime": "<module 'datetime' from '/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/datetime.py'>",
[virtualenv]   "math": "<module 'math' from '/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so'>",
[virtualenv]   "json": "<module 'json' from '/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/json/__init__.py'>"
[virtualenv] }
[virtualenv] add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, via=copy, app_data_dir=/home/haizaar/.local/share/virtualenv)
[virtualenv] install pip from wheel /nix/store/vmx0wi2cbfn69vnfffssxkqv5pgdx1yx-python3.11-virtualenv-20.25.1/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-24.0-py3-none-any.whl via CopyPipInstall
[virtualenv] install setuptools from wheel /nix/store/vmx0wi2cbfn69vnfffssxkqv5pgdx1yx-python3.11-virtualenv-20.25.1/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/setuptools-69.1.0-py3-none-any.whl via CopyPipInstall
[filelock:filelock] Attempting to acquire lock 140567286838928 on /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any.lock
[filelock:filelock] Attempting to acquire lock 140567286838736 on /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any.lock
[filelock:filelock] Lock 140567286838928 acquired on /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any.lock
[filelock:filelock] Attempting to release lock 140567286838928 on /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any.lock
[filelock:filelock] Lock 140567286838736 acquired on /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any.lock
[filelock:filelock] Lock 140567286838928 released on /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any.lock
[filelock:filelock] Attempting to release lock 140567286838736 on /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any.lock
[filelock:filelock] Lock 140567286838736 released on /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any.lock
[virtualenv] copy /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/distutils-precedence.pth to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/distutils-precedence.pth
[virtualenv] copy /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any/pip-24.0.virtualenv to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/pip-24.0.virtualenv
[virtualenv] copy directory /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/pkg_resources to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/pkg_resources
[virtualenv] copy directory /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any/pip to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/pip
[virtualenv] copy directory /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/setuptools-69.1.0.dist-info to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/setuptools-69.1.0.dist-info
[virtualenv] copy directory /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/setuptools to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/setuptools
[virtualenv] copy directory /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/_distutils_hack to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/_distutils_hack
[virtualenv] copy /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-69.1.0-py3-none-any/setuptools-69.1.0.virtualenv to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/setuptools-69.1.0.virtualenv
[virtualenv] generated console scripts 
[virtualenv] copy directory /home/haizaar/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-24.0-py3-none-any/pip-24.0.dist-info to /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/lib/python3.11/site-packages/pip-24.0.dist-info
[distlib:distlib.util] changing mode of /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/pip3 to 755
[distlib:distlib.util] changing mode of /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/pip-3.11 to 755
[distlib:distlib.util] changing mode of /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/pip3.11 to 755
[distlib:distlib.util] changing mode of /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/pip to 755
[virtualenv] generated console scripts pip3.11 pip pip-3.11 pip3
[virtualenv] add activators for Bash, CShell, Fish, Nushell, PowerShell, Python
[virtualenv] write /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/pyvenv.cfg
[virtualenv]    home = /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin
[virtualenv]    implementation = CPython
[virtualenv]    version_info = 3.11.8.final.0
[virtualenv]    virtualenv = 20.25.1
[virtualenv]    include-system-site-packages = true
[virtualenv]    base-prefix = /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8
[virtualenv]    base-exec-prefix = /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8
[virtualenv]    base-executable = /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11
[virtualenv]    prompt = mediapipe-custom-hand-guesture-recogniser-py3.11
Using virtualenv: /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv
Spawning shell within /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv
. /home/haizaar/dev/mediapipe-custom-hand-guesture-recogniser/.venv/bin/activate

[dimbleby]

Code is here

Hard to see that going wrong, suggest put in some prints to see what is happening and submit a pull request when you figure it out!

[hexmode]

Similar issue on debian:

$ rm -rf ~/.cache/pypoetry/

$ poetry config --list
cache-dir = "~/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = false
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # ~/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = true
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

$ poetry debug info

Poetry
Version: 1.8.3
Python:  3.12.3

Virtualenv
Python:         3.12.3
Implementation: CPython
Path:           NA
Executable:     NA

Base
Platform:   linux
OS:         posix
Python:     3.12.3
Path:       ~/.asdf/installs/python/3.12.3
Executable: ~/.asdf/installs/python/3.12.3/bin/python3.12

$ poetry -vvv shell
Loading configuration file ~/.config/pypoetry/config.toml
Trying to detect current active python executable as specified in the config.
Found: ~/.asdf/installs/python/3.12.3/bin/python
Creating virtualenv devops-fabric-MXLzDq5t-py3.12 in ~/.cache/pypoetry/virtualenvs
[virtualenv] find interpreter for spec PythonSpec(path=/usr/bin/python3.11)

Note everything up until the last line says python3.12 and it is python3.11 after that.

[hexmode]

In my testing the problem is in virtualenv.cli_run in build_venv. Before that line:

• Executable_str: ~/.asdf/installs/python/3.12.3/bin/python3.12
• sys: ~/.local/pipx/venvs/poetry/bin/python
• args: ['--no-download', '--no-periodic-update', '--python', '.../.asdf/installs/python/3.12.3/bin/python3.12', '--prompt', 'devops-fabric-py3.12', '--no-wheel', '.../.cache/pypoetry/virtualenvs/devops-fabric-MXLzDq5t-py3.12']

This indicates that venv is being passed a path for python, but, somehow, is ignoring it.

[hexmode]

It looks like there is no --python parameter for current virtualenv. Instead, --try-first-with should be used.

[hexmode]

Well, there is a --python but it looks like this needs --try-first-with.

[dimbleby]

But --python is documented - and on a quick read of virtualenv code I don't see why it would not work. If that is not working, it probably wants reporting to virtualenv.

Are we passing an absolute path? Hard to tell above, looks like perhaps you have redacted the full path?

If we are not, perhaps the better fix is to pass an absolute path as --python

[hexmode]

1. An absolute path is provided.
2. --python is documented, but so is --try-first-with.

Even though --try-first-with was originally created for Windows where the wrong python interpreter was being found, it seems to fit this use case as well.

[dimbleby]

If you have a reproducible example where --python should work but doesn't, please report that to virtualenv. Let's fix this for all virtualenv users, not just find a workaround in poetry only.

[dimbleby]

Perhaps you are seeing some version of pypa/virtualenv#2285?

It is indicated there that a pull request would be welcome.

I'd add that virtualenv releases considerably more often than poetry does, so if you are keen to see a fix hit the wild - it'll almost surely happen faster there.

[hexmode]

Since you mentioned the absolute path being passed to --python, it looks like this is closer to this issue where the args are --try-first-with=/usr/bin/python3.10 -p /usr/bin/python3.8 and the resulting environment is 3.10.