Running pytest in a sub-interpreter

[bluetech]

Python 3.14 added a new module concurrent.interpreters which allows creating isolated sub-interpreters with their own GIL. I like the concept and experimented it with it a bit in pytest-xdist/execnet, trying to use sub-interperters instead of processes. I immediately ran into some problems.

But before getting into xdist and such (which share some issues with #13768, particularly around stdio capturing), the most basic thing is just running pytest in a single sub-interpreter:

from concurrent import interpreters

interp = interpreters.create()

interp.exec("""
import sys
sys.path.insert(0, "src")
import pytest
pytest.main()
""")

Here are the problems this runs into:

• faulthandler cannot be imported in a sub-interpreter. This may be a fundamental issue as faulthandler is a process-wide concept (Isolate the faulthandler Module Between Interpreters python/cpython#101509). I am not sure yet. To make this work we may want to disable auto-loading the faulthandler plugin and skip its tests (or run them in a subprocess) when running under a sub-interpreter.

• multiprocessing.Pool used by a couple of our tests testing/acceptance_test.py::TestGeneralUsage::test_config_error and testing/_py/test_local.py::TestLocalPath::test_make_numbered_dir_multiprocess_safe is incompatible multiprocessing.Pool is incompatible with concurrent.interpreters python/cpython#140057

• readline imported by test test_libedit_workaround is incompatible

[RonnyPfannschmidt]

that one looks fun

we'll need a special execnet mode and we'll need stdio capture revisited, in particular execnet coordinated stdio capture