Skip to content

Stop using multiprocessing for fitting dispersion models earlier #142

Open
@le-ander

Description

@le-ander

From experience, I can say that the multiprocessing overhead when fitting dispersion models seems to be a lot larger than the code is currently written for.

ie. fitting the last 50 or so models still takes a long time and as soon as multiprocessing is switched off for the last models, things become a lot faster. maybe multiprocessing could be only used when there are more than 10x as many genes left than processors here:

if nproc > 1 and len(idx_update) > nproc:

So something like:
if nproc > 1 and len(idx_update) > 10 * nproc:

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions