Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: python process worker #138

Merged
merged 6 commits into from
Jan 5, 2024
Merged

fix: python process worker #138

merged 6 commits into from
Jan 5, 2024

Conversation

ClemDoum
Copy link
Collaborator

@ClemDoum ClemDoum commented Dec 21, 2023

Description

Python multiprocessing is not trivial to handle consistently access on the different OS + the pyinstaller executable. Using the wrong setup trigger problems such as workers not launching or hanging at application shutdown.

This PRs aims at fixing these issues.

Changes

neo4j-app

Fixed

  • switch to python >= 3.10.8 in order to avoid: benefit from cpython fix
  • switched back from ProcessPoolExecutor to multiprocessing.Pool using the spawn context + maxtasksperchild=1. Using maxtasksperchild=1 allows the child process to be destroyed when it receives SIGTERM or SIGKILL signals. Otherwise probably due to this cpython bug the pool hangs and fails to kill child processes. Since we know that Python workers will only accomplish a single task which is work_forever, using maxtasksperchild=1 is an acceptable hack

Changed

  • use pyinstaller==6.3.0 for builds
  • updates dependencies

@ClemDoum ClemDoum self-assigned this Dec 21, 2023
@ClemDoum ClemDoum force-pushed the fix/multiprocessing-context branch 6 times, most recently from fda39b1 to 4c937a6 Compare January 3, 2024 12:28
@ClemDoum ClemDoum force-pushed the fix/multiprocessing-context branch 4 times, most recently from c51d8a7 to c02664a Compare January 5, 2024 09:00
@ClemDoum ClemDoum changed the title fix: set multiprocessing context according to OS fix: python process worker Jan 5, 2024
@ClemDoum ClemDoum merged commit 92a3af7 into main Jan 5, 2024
8 checks passed
@ClemDoum ClemDoum deleted the fix/multiprocessing-context branch January 5, 2024 09:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant