Skip to content

Commit

Permalink
use max_cores in taskQueue instead of system cores (#2038)
Browse files Browse the repository at this point in the history
The class TascQueue accepts an argument `thread_count` that is used for the max size of a queue.

In the `MultithreadedJobExecutor `class there is a variable defined (`max_cores`) that is getting its value from the available cores of the machine. Further down in the same class when TaskQueue is called instead of using the `max_cores` it is using `psutil.cpu_count()`. 

I suggest to use  the self.max_cores in the call of TaskQueue in the file executor.py instead of psutil.cpu_count()

Use case:
when a job executor is setup as MultithreadedJobExecutor one can override the max_cores after the initialization of the object and limit the use to the specified cores.

Additional enhancement would be to include an argument to allow the use to provide the number of cores available for use

Co-authored-by: Michael R. Crusoe <[email protected]>
  • Loading branch information
kmavrommatis and mr-c authored Nov 11, 2024
1 parent 0b64935 commit 048eb55
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion cwltool/executors.py
Original file line number Diff line number Diff line change
Expand Up @@ -429,7 +429,7 @@ def run_jobs(
logger: logging.Logger,
runtime_context: RuntimeContext,
) -> None:
self.taskqueue: TaskQueue = TaskQueue(threading.Lock(), psutil.cpu_count() or 1)
self.taskqueue: TaskQueue = TaskQueue(threading.Lock(), int(math.ceil(self.max_cores)))
try:
jobiter = process.job(job_order_object, self.output_callback, runtime_context)

Expand Down

0 comments on commit 048eb55

Please sign in to comment.