Track the number of tasks executing on each worker, and submit tasks to workers with minimum number of tasks #273
Description
Alternative to the current round robin approach, A possible way of scheduling is to track the number of tasks running on each worker in a scheduler class.
This could be done by having a dictionary of tasks submitted to each worker stored as a data member of the scheduler class(Current RoundRobin class would be replaced by a class named Scheduler ).
The task tracking would be done with 2 steps.
1)When a task is submitted to a worker, the tasks submitted counter would be incremented for the particular worker in the scheduler's dictionary.
2)In the monitor process when a task is completed this information is send to the root node, A background thread in the root process would poll for tasks completed with a incremental backoff as in MPI4py. Thus when task is completed on a worker the tasks submitted counter would be decremented for the particular worker in the scheduler's dictionary.
As scheduler can be aware of tasks per worker, it can submit new tasks to the workers with minimum number of tasks