Issues running AXO trigger study pipeline on coffea.casa

[ekauffma]

This is a follow up to a presentation at the coffea users meeting (thank you all for your help already):

https://indico.cern.ch/event/1496981/contributions/6305632/attachments/3000153/5286709/axo_coffeausersmeeting_250121.pdf

I created a public repo containing the same script we showed during the meeting and a json with a filepath for a file containing 100 events for those who want to test it out: https://github.com/ekauffma/axo_studies_public (I think only CMS members will have access to the file so please contact me at [email protected] if you have issues)

To recap what people were saying during the meeting, the dask dashboard screenshot looked alarming due to the 90k hist-on-block tasks..

I am in the process of trying out adding trigger as category axis as suggested by @lgray and I will update here with results!

[ekauffma]

I ran two different trials.

For the first, I processed 9 1d histograms for 9 triggers (total of 81 histograms). This took 111.1 seconds.

For the second, I take Lindsey's suggestion of making an axis for the trigger (total of 9 histograms). This took 45.5 seconds.

The task graph looks the same in both cases (obtained by to_compute[1].get('Scouting_2024I').visualize(filename="dask_coffea_graph_separateTriggers", format="png")). Please let me know if there is a way to get a more detailed task graph!

[lgray]

@ekauffma can you do: dask.optimize(to_compute) followed by that string of function calls to visualize? This will let us see everything you're calculating and also the compute graph that's actually getting executed. visualize on its own doesn't optimize the graph and looking at an individual histogram will only show you its graph, rather than the fully optimized and fused graph.

[lgray]

also what you're visualizing upstairs appears to be the report, rather than the histograms! you probably need to check out to_compute[0], for instance?

[ekauffma]

Here are the two task graphs for the trials I described above:

Trial 1:

Trial 2:

These seem very large to me so I repeated the trials except for 1 histogram per trigger and 3 triggers enabled:

Trial 1:

Trial 2:

[lgray]

Is this with or without an optimized graph?

[ekauffma]

This is optimized I believe

[ikrommyd]

These are the graphs I got yesterday from running over 2 partitions of the one root file you sent
optimized.pdf
unoptimized.pdf

Also I think pdf files are much better for graphs IMO

[ikrommyd]

Whatever was already in the script you sent during the meeting.

[ekauffma]

Is there anything about these task graphs that seem concerning? I am not sure what to look for

[ikrommyd]

Yes, the way they explode to A LOT of hist-on-block nodes even though we're starting only from two input nodes.

[ekauffma]

Do you have any idea why this might happen? I took Lindsey's suggestion a step further and also added an axis for different objects (muons, electrons, etc.) for the applicable histograms. Running on one file, I should be creating 39 histogram objects. I see 111 hist-on-block nodes here.

dask_coffea_graph_combinedTriggers (1).pdf

[jpearkes]

Hi @ekauffma,

I think the reason that the number of hist-on-block nodes is larger than the number of actual histograms is because coffea is splitting the file up into 3 processing steps and 39*3=117. If we increase the step size to 200_000 the preprocessing only splits things into 1 step, and then the total number of histograms in 39 as expected.

E.g. number of hist-on-block nodes = (number of steps from preprocessing)*(number of histograms)

With step size of 50_000

With step size of 200_000

This doesn't help with the crashes much, but at least it explains why the number of hist-on-block calls is so large.