Peak Memory Usage

[landroutsosAIE]

Is there a way to find out what was the peak of memory (ram) usage of a pipeline?

[bentsherman]

The execution report will give you the peak mem usage (peak_rss) for each task in the pipeline

[landroutsosAIE]

Please correct me if i am missing something

[bentsherman]

As long as there is enough memory for the largest task, the pipeline will work. But if you want to know how much memory you need to run the pipeline with the highest level of parallelism, that is a much more complicated question that depends on the pipeline structure. I would render the pipeline DAG and inspect it manually to get a sense for where the "widest" part of your pipeline is in terms of memory usage.

[landroutsosAIE]

Thats a good suggestion, but the DAG for the rnaseq pipeline that I am running is massive. I will try. Thanks

[bentsherman]

Yes, it might not be feasible if the workflow is too large. There are some config options to control the level of detail for the DAG which might help a little.

But the bigger problem I see is, for a sufficiently large workflow, even if you find the maximum possible amount of memory that could be used at once, it could vary greatly based on the order in which tasks are executed, and it might not ever even reach that theoretical maximum. So you might end up over-provisioning by taking this analytical approach. You might be better off inspecting the waterfall for a few small runs and finding some heuristic.

People usually get around this problem with auto-scaling in the cloud. But I assume you are preparing some fixed allocation of compute nodes.

[landroutsosAIE]

Yes, the order is very important for this problem and the maximum cant be 100% correct.
We didnt really think of auto-scaling in the cloud, since we arent really sure of the Azure Batch costs in this case. We have allowed pool creation depending on the label of each process (low, medium, high etc) and auto-scale for each independent pool (it has a fixed vmType).
I will try some things and get back to our conversation.