Hanging during initialisation with EB

[ThomasHowarth]

This bug is very specific, but wondering if anyone has any thoughts on it, or have seen anything similar.

When I run on some systems (this error is not consistent across different HPC systems) with EB, the code hangs on initialisation while resetting the covered masks when I run with no AMR. I can trick it out of this by setting amr.max_level=1 and peleLM.refine_EB_max_level = 0. I don't think this would effect code performance too much since it presumably would just have an empty second entry in the vectors of MultiFabs, but it seems like a strange thing to happen. It also seems very strange that it does not happen consistently across different systems.

[marchdf]

I haven't seen this... seems super annoying. Is there any sort of consistency to the brokenness when testing different compilers?

[esclapez]

So you're running single level with EB right ?

Once thing that could be slow, but hopefully not make things hang, is the computation of the signed distance function to control derefining regions near the EB. It is done during the init, where the distance function is computed on the base level only, starting from the one AMReX compute internally during the EB generation and extending it because we need it further away from the EB.
It is interpolated on finer levels if need be because distance function can be expensive.
There is a test to only trigger the computation of the distance function if you require your EB refined at a level lower than the amr.max_level. And it is basically OFF if you only have a single level.

But there might be something I'm missing or some weird conjunction of oddities leading to the stall. It might be worth forcing this check off to see if that helps.