-
Notifications
You must be signed in to change notification settings - Fork 13
Implement concurrent parsing #743
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: alpha-test-dev
Are you sure you want to change the base?
Conversation
So final test that is failing is due to not having check mode supported. |
I have narrowed the issue down to unpickling I have boiled it down: import montepy
import pickle
error = montepy.errors.ParsingError("", "", [])
pickle.loads(pickle.dumps(error)) This causes: Traceback (most recent call last):
File "/home/mgale/mambaforge/lib/python3.13/site-packages/IPython/core/interactiveshell.py", line 3667, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<ipython-input-23-8c9e5f955a19>", line 1, in <module>
pickle.loads(pickled)
~~~~~~~~~~~~^^^^^^^^^
TypeError: ParsingError.__init__() missing 2 required positional arguments: 'message' and 'error_queue' |
Maybe this is our solution? https://bugs.python.org/issue32696 |
It seems to be due to the fact that we are inheriting from Update: It doesn't seem like
|
The issues with windows are independent of serial/multi-process mode. This is wild. These issues should be abstracted away by python. |
Fixed broken test cases with Windows
Update: I was able to effective fix #745. Now to get this ready I need to:
|
@tjlaboss I think this is finally ready. |
I have gotten some preliminary data from @dodu94, that this actually slows down the parsing of large models. I think we need to pause this until we can figure out that issue. |
TIL: |
Pull Request Checklist for MontePy
Description
This PR adds concurrent parsing using
multiprocessing
viaconcurrent.futures
. This allows large models to be parsed faster, though it is not perfectly scalable. This option is off by default due to windows requiring aif __name__ == "__main__":
guard.I am merging this onto
alpha-test-dev
because this needs to be a minor release. I am not sure when we want to do a minor release yet. Probably afterjit_parsing
is implemented as well.Implementation.
In general the process of parsing can be broken into three main tasks:
How this works in the multi-processing mode is best viewed through the processes.
concurrent.futures
). Once it is ready it reads the resulting objects out from another queue, and appends then to the problem. Once all objects appended the sub processes are terminated and the objects are linked togetherperformance
I haven't effectively collected data for this yet. This can be done at a later date. From the preliminary data I have gotten there appears to now be a floor on load times of about a second due to the added overhead.
Fixes #742, #745
General Checklist
black
version 25.Documentation Checklist
First-Time Contributor Checklist
pyproject.toml
if you wish to do so.Additional Notes for Reviewers
Ensure that:
📚 Documentation preview 📚: https://montepy--743.org.readthedocs.build/en/743/