[subinterpreters] PEP 554 implementation: add interpreters module #76785
Comments
|
In the interest of getting something landed for 3.7, so we can start using it in tests, I'm putting up a patch for a low-level interpreters module. In some ways this is a precursor for issue bpo-30439, which will add a proper public stdlib module in 3.8. The module I'm adding draws from the ideas in PEP-554 (particularly for channels). Consequently, this will also give us an opportunity to try out some of the semantics from the PEP to give us better ideas for 3.8. I expect to have some follow-on patches to facilitate simpler use in tests. This patch is big enough already. :) |
ericsnowcurrently
added
3.7 (EOL)
|
@ned, it may be a little tight to land this given the time left before beta 1. However, this is meant as a tool for us to use in the test suite (particularly to test the subinterpreter C-API). So I'm arguing that, if necessary, it would still be okay to land this after the feature freeze. (I'm still hoping to get this in before the cutoff.) What do you think? |
|
FYI, there are a few things I need to clean up in the PR. However, I expect that those changes will be minor relative to the the whole patch, so I wanted to get the ball rolling on a review. :) |
|
@eric, given the breadth of change introduced in the PR (including adding a new extension), I think it would be best if at all possible to get it in for beta 1 if we can resolve the review comments in time. If necessary and if there are no objections from other core developers, I would be willing to consider making an exception and allowing it into beta 2 as long as it remains a private interface. If it looks like it won't be in releasable shape by then, I think you should hold off for 3.8; doing otherwise would be unfair to others and to our downstream beta users / testers, for example, even if it is private, adding a new extension and setup.py changes potentially affect downstream packagers. |
|
Sounds good, Ned. Thanks for taking a look. I should have everything finished up by Friday, so I'm hopeful for landing the change before the deadline. I may have a few minor tweaks to make after that, but I'll discuss that with you before making any changes if that happens. |
|
I've merged the patch without Windows support, which shouldn't be a problem given the purpose of the extension module. I've also added a PR for get the module building under Windows. I'd like to get that resolved ASAP. |
|
Eric, looks like some buildbots are unhappy, for instance: |
|
Yeah, I'm looking into it. Also, I noticed some refleaks that I'll be sorting out. |
|
On 4 of the buildbots: ====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 890, in test_drop_multiple_times
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 848, in test_drop_single_user
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 957, in test_drop_used_multiple_times_by_single_user
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 899, in test_drop_with_unused_items
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3) |
|
On the PPC64 AIX 3.x buildbot: ====================================================================== Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test__xxsubinterpreters.py", line 784, in test_repr
self.assertEqual(repr(cid), 'ChannelID(10)')
AssertionError: 'ChannelID(0)' != 'ChannelID(10)'
- ChannelID(0)
+ ChannelID(10)
? + |
|
I just put up a PR that should fix the 4 buildbots. |
|
The buildbots should be happier now. I'll keep an eye on them. |
|
A couple defects reported by coverity: ** CID 1428758: Integer handling issues (CONSTANT_EXPRESSION_RESULT) /Modules/_xxsubinterpretersmodule.c: 45 in _coerce_id() 1217 } |
|
I've landed a PR that fixes all the memory leaks in the module. It also fixes the 2 defects reported by coverity. The only thing left here is to get the module building under Windows. |
|
FYI, out of 2389 source lines in the C extension, 1563 are the channel-related code. That means the non-channel code is 826 lines (about a third). That non-channel code does not depend on the channel code at all and I considered splitting the source out, but figured there wasn't enough benefit. However, I might revisit the matter later when I circle back to PEP-554. :) |
|
Eric, it looks like your recent commit introduced a refleak. We need to fix it before beta2. ~/d/p/cpython (master $) » ./python.exe -m test -R3:3 test_multiprocessing_fork 1 test failed: And just before it: ~/d/p/cpython ((bd09335…) $) » ./python.exe -m test -R3:3 test_multiprocessing_fork Total duration: 9 min 12 sec |
…110248) This is a temporary fix. The full fix may involve serializing the traceback in some form.
…ythongh-110248)" (pythongh-110314) This reverts commit 6bc889a. That commit is causing some buildbot failures.
…110322) This is a temporary solution. The full fix may involve serializing the traceback in some form. (FYI, I merged this yesterday and the reverted it due to buildbot failures. See pythongh-110248.)
…thongh-110251) This specifically refers to `test.support.interpreters.Interpreter.run()`.
(This is still a test module.)
…h-110607) This restores their shareability, which was disabled by pythongh-110318 due to ref leaks.
…ongh-111502) This is partly to clear this stuff out of pystate.c, but also in preparation for moving some code out of _xxsubinterpretersmodule.c. This change also moves this stuff to the internal API (new: Include/internal/pycore_crossinterp.h). @vstinner did this previously and I undid it. Now I'm re-doing it. :/
This moves several general internal APIs out of _xxsubinterpretersmodule.c and into the new Python/crossinterp.c (and the corresponding internal headers). Specifically: * _Py_excinfo, etc.: the initial implementation for non-object exception snapshots (in pycore_pyerrors.h and Python/errors.c) * _PyXI_exception_info, etc.: helpers for passing an exception beween interpreters (wraps _Py_excinfo) * _PyXI_namespace, etc.: helpers for copying a dict of attrs between interpreters * _PyXI_Enter(), _PyXI_Exit(): functions that abstract out the transitions between one interpreter and a second that will do some work temporarily Again, these were all abstracted out of _xxsubinterpretersmodule.c as generalizations. I plan on proposing these as public API at some point.
There were a few corner cases I didn't handle properly in pythongh-111530, which I've noticed while working on a follow-up PR. This fixes those cases.
…ythongh-111715) I added _Py_excinfo to the internal API (and added its functions in Python/errors.c) in pythongh-111530 (9322ce9). Since then I've had a nagging sense that I should have added the type and functions in its own PR. While I do plan on using _Py_excinfo outside crossinterp.c very soon (see pythongh-111572/pythongh-111573), I'd still feel more comfortable if the _Py_excinfo stuff went in as its own PR. Hence, here we are. (FWIW, I may combine that with pythongh-111572, which I may, in turn, combine with pythongh-111573. We'll see.)
…thongh-112323) The new function corresponds to the existing (public) PyType_GetName() and PyType_GetQualName().
This involves a number of changes for PEP 734.
…ongh-113014) pythongh-112982 broke test_threading on one of the s390 buildbots (Fedora Clang Installed). Apparently ImportError is raised (rather than ModuleNotFoundError) for the name part of "from" imports. This fixes that by catching ImportError in test_threading.py.
…3012) This brings the module (along with the associated extension modules) mostly in sync with PEP 734. There are only a few small things to wrap up.
This is one of the last pieces to get test.support.interpreters in sync with PEP 734.
…ions (pythongh-113034) When an exception is uncaught in Interpreter.exec_sync(), it helps to show that exception's error display if uncaught in the calling interpreter. We do so here by generating a TracebackException in the subinterpreter and passing it between interpreters using pickle.
…terpreter Exceptions (pythongh-113036) We need the TracebackException of uncaught exceptions for a single purpose: the error display. Thus we only need to pass the formatted error display between interpreters. Passing a pickled TracebackException is overkill.
In pythongh-112982 I made some changes to .github/CODEOWNERS. Later, @ezio-melotti pointed out that some of those changes were unnecessary.
The primary objective here is to allow some later changes to be cleaner. Mostly this involves renaming things and moving a few things around. * CrossInterpreterData -> XIData * crossinterpdatafunc -> xidatafunc * split out pycore_crossinterp_data_registry.h * add _PyXIData_lookup_t
…h-126602) This change makes it easier to backport the _interpreters, _interpqueues, and _interpchannels modules to Python 3.12.
…126704) These changes makes it easier to backport the _interpreters, _interpqueues, and _interpchannels modules to Python 3.12. This involves the following: * rename several structs and typedefs * add several typedefs * stop using the PyThreadState.state field directly in parking_lot.c
…126707) These changes makes it easier to backport the _interpreters, _interpqueues, and _interpchannels modules to Python 3.12. This involves the following: * add the _PyXI_GET_STATE() and _PyXI_GET_GLOBAL_STATE() macros * add _PyXIData_lookup_context_t and _PyXIData_GetLookupContext() * add _Py_xi_state_init() and _Py_xi_state_fini()
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
The text was updated successfully, but these errors were encountered: