-
-
Notifications
You must be signed in to change notification settings - Fork 89
Description
Hi there, thanks for the nice code!
I'm encountering a problem where my scripts will hang indefinitely on close if an exception is raised after a ProcessPool has been created (even if the exception itself is handled in the body of the program, and even if the pool is not being used at the time of the exception). The script can only be stopped by using ctrl+c, which gives this monster stacktrace:
Stack Trace
... <-- script does work here
^C <-- deliberately interrupted here
Process ForkPoolWorker-10:
script exiting <-- exit message from the script
Process ForkPoolWorker-7:
Process ForkPoolWorker-3:
Process ForkPoolWorker-13:
Process ForkPoolWorker-4:
Process ForkPoolWorker-8:
Process ForkPoolWorker-12:
Process ForkPoolWorker-5:
Process ForkPoolWorker-9:
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Process ForkPoolWorker-6:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
Process ForkPoolWorker-11:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 368, in get
res = self._reader.recv_bytes()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 224, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 422, in _recv_bytes
buf = self._recv(4)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 387, in _recv
chunk = read(handle, remaining)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 368, in get
res = self._reader.recv_bytes()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 224, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 422, in _recv_bytes
buf = self._recv(4)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 387, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
<-- Hang indefinitely
^C <-- Get tired of waiting and raise crtl-c
Process ForkPoolWorker-14:
Exception ignored in atexit callback: <function _exit_function at 0x7f889017ef80>
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/util.py", line 334, in _exit_function
_run_finalizers(0)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/util.py", line 300, in _run_finalizers
finalizer()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 695, in _terminate_pool
cls._help_stuff_finish(inqueue, task_handler, len(pool))
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 675, in _help_stuff_finish
inqueue._rlock.acquire()
KeyboardInterrupt:
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 368, in get
res = self._reader.recv_bytes()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 224, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 422, in _recv_bytes
buf = self._recv(4)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 387, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
The problem is challenging to recreate exactly, but it can be at least partially recreated by doing the following at the console:
Stack Trace
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathos.pools
>>> pool = pathos.pools.ProcessPool(20)
>>> Process ForkPoolWorker-17: <-- Raise ctrl-c here
Process ForkPoolWorker-13:
Process ForkPoolWorker-20:
Process ForkPoolWorker-18:
Process ForkPoolWorker-16:
Process ForkPoolWorker-19:
Process ForkPoolWorker-9:
Process ForkPoolWorker-7:
Process ForkPoolWorker-11:
Process ForkPoolWorker-5:
Process ForkPoolWorker-15:
KeyboardInterrupt
>>> Process ForkPoolWorker-10:
Traceback (most recent call last):
Traceback (most recent call last):
Process ForkPoolWorker-6:
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
Process ForkPoolWorker-14:
Process ForkPoolWorker-4:
Process ForkPoolWorker-3:
Traceback (most recent call last):
Process ForkPoolWorker-8:
Process ForkPoolWorker-2:
Process ForkPoolWorker-1:
Process ForkPoolWorker-12:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
Traceback (most recent call last):
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
Traceback (most recent call last):
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
KeyboardInterrupt
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 368, in get
res = self._reader.recv_bytes()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 224, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 422, in _recv_bytes
buf = self._recv(4)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/connection.py", line 387, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
>>> <-- Hangs here
<-- Ctrl-d (EOF) to to exit, hangs again
<-- Ctrl-c again to exit
^CProcess ForkPoolWorker-36:
Process ForkPoolWorker-35:
Process ForkPoolWorker-32:
Process ForkPoolWorker-33:
Process ForkPoolWorker-31:
Process ForkPoolWorker-34:
Process ForkPoolWorker-39:
Process ForkPoolWorker-37:
Process ForkPoolWorker-29:
Process ForkPoolWorker-27:
Process ForkPoolWorker-38:
Process ForkPoolWorker-26:
Process ForkPoolWorker-30:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Exception ignored in atexit callback: <function _exit_function at 0x7f10820e55a0>
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/util.py", line 334, in _exit_function
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
Process ForkPoolWorker-23:
Process ForkPoolWorker-25:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
Process ForkPoolWorker-24:
KeyboardInterrupt
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
Process ForkPoolWorker-40:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Process ForkPoolWorker-28:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
Process ForkPoolWorker-21:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
Traceback (most recent call last):
KeyboardInterrupt
Traceback (most recent call last):
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
_run_finalizers(0)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/util.py", line 300, in _run_finalizers
Process ForkPoolWorker-22:
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
finalizer()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 695, in _terminate_pool
Traceback (most recent call last):
cls._help_stuff_finish(inqueue, task_handler, len(pool))
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 675, in _help_stuff_finish
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
Traceback (most recent call last):
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
inqueue._rlock.acquire()
KeyboardInterrupt:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/queues.py", line 367, in get
with self._rlock:
File "/home/oliver/.local/lib/python3.10/site-packages/multiprocess/synchronize.py", line 101, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
In this latter example you'll notice that the pool has not even been given any work to do yet. My machine does not have 20 CPUs available, but the problem appears to be at least partially race-dependent, so the more processes that are started, the more likely for the bug to arise.
The problem can at least partially be avoided by calling terminate()
on the pool before close:
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathos.pools
>>> pool = pathos.pools.ProcessPool(20)
>>> pool.terminate()
>>>
KeyboardInterrupt
>>>
But using a context manager has no effect:
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathos.pools
>>> with pathos.pools.ProcessPool(20) as pool:
... pass
...
>>> Process ForkPoolWorker-13:
(stack trace continues as above)