Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is parallel mode desired? #81

Closed
CodyCBakerPhD opened this issue Sep 24, 2024 · 2 comments · Fixed by #90
Closed

Is parallel mode desired? #81

CodyCBakerPhD opened this issue Sep 24, 2024 · 2 comments · Fixed by #90
Assignees

Comments

@CodyCBakerPhD
Copy link
Member

A question for @grg2rsr as I clean up and update the various conversion scripts

For any script dealing with iteration over multiple sessions, do you still want that to be parallelized?

@grg2rsr
Copy link
Collaborator

grg2rsr commented Sep 27, 2024

no I don't think so. Do you think it will have any performance gains over submitting many jobs to the cluster and letting the job scheduler take care of it? Especially since this is going to be a "one time run" (or few times), I don't think speed is super critical

@CodyCBakerPhD
Copy link
Member Author

Do you think it will have any performance gains over submitting many jobs to the cluster and letting the job scheduler take care of it?

That is effectively the same thing, though probably not maximizing potential I/O usage per 'job'

Should be even faster overall though if the cluster has a large number of nodes

So I will let you handle it by designing bash scripts that run the conversion on a particular job ID and remove the current parallel scripts

@CodyCBakerPhD CodyCBakerPhD linked a pull request Sep 27, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants