Capability to switch number of OpenMP threads for different sub-apps

[lewisgross1296]

Reason

I recently ran into trouble when trying to do a coupled simulation where I was having inability to converge with any number of OpenMP threads greater than a certain number. This effectively limits my --n-threads argument to that number (which was 32). However, I really would like to be able to run OpenMC with the max threads per node (112 on Windriver). This thread behavior is likely a bug, but until it gets solved, I'll need to run my MOOSE Heat Conduction with 32 threads. In theory, you can just bump up the MPI procs to account for less threads, however, using multiple MPI processes per node has multiplicative memory requirements and my OpenMC models are large.

It appears, that there's currently no way to accomplish something like this, see libmesh/libmesh#4185.

Design

A few things would be required

• A function to change the number of threads being used, likely from the OpenMC side.
• Code in OpenMCCellAverageProblem to switch the number of threads back and forth before/after each OpenMC run. Likely some MOOSE data structures rely on the initial threading configuration, so we'd want to make sure not to break it
• Some member variable that stores the number of threads for each (sub)app

Impact

This would be pretty useful for me in the case that we can't debug my threading issue quickly. It could also be useful for others wishing to use different numbers of threads for different applications, which might be a generally useful feature, for simulation and/or debugging.

Additional context
Add any other context or screenshots about the feature request here.
Tagging @pshriwise in case I missed anything from our discussion

[lewisgross1296]

According to @nuclearkevin, the issue I was seeing is an older bug related to HYPRE. His workaround involves using MOOSE's default (block Jacobi) preconditioner instead of the hypre boomeramg preconditioner.

I'll re-run my model to confirm, but perhaps Kevin can explain further if needed/others are curious. I am also wondering if HYPRE plans to fix this issue, but that is probably out of our scope to worry about.

[lewisgross1296]

Reporitng back on Kevin's suggestion to use block Jacobi as the preconditioner over HYPRE boomeramg. It seems like he was for sure right about the bug being a HYPER problem.

When I submitted the same job with

  petsc_options_value = ''
  petsc_options_iname = ''

instead of

  petsc_options_value = 'hypre boomeramg'
  petsc_options_iname = '-pc_type -pc_hypre_type'

it was able to "work" with 112 threads, whereas before it couldn't even do a single heat conduction solve.

However, a few performance hits. I noticed initialization was a bit longer for the THM sub-apps

Still Setting Up..........thm000: 
thm000: 
thm000: *** Info ***
thm000: The following information occurred in the AuxKernel 'channel:area_aux' of type ProjectionAux.
thm000: 
thm000: Projection lowers order, please expect a loss of accuracy
  Finished Instantiating Sub-Apps                                                        [638.17 s] [70348 MB]
Still Setting Up..............................................................................................................................................................................................................................................................
  Initializing
    Updating Because Mesh Changed
      Updating Mesh                                                                      [ 14.83 s] [75902 MB]
    Finished Updating Because Mesh Changed                                               [ 15.30 s] [76027 MB]
    Currently Initializing Equation Systems.....                                         [ 32.75 s] [80820 MB]
  Finished Initializing                                                                  [ 48.05 s] [80820 MB]
Finished Setting Up                                                                      [2121.53 s] [80820 MB]
...

Parallelism:
  Num Processors:          32
  Num Threads:             112

whereas before this took

Still Setting Up...thm000: 
thm000: 
thm000: *** Info ***
thm000: The following information occurred in the AuxKernel 'channel:area_aux' of type ProjectionAux.
thm000: 
thm000: Projection lowers order, please expect a loss of accuracy
  Finished Instantiating Sub-Apps                                                        [ 76.66 s] [11498 MB]
Still Setting Up................................
  Initializing
    Updating Because Mesh Changed
      Updating Mesh                                                                      [ 17.56 s] [11837 MB]
    Finished Updating Because Mesh Changed                                               [ 18.02 s] [11963 MB]
    Currently Initializing Equation Systems......                                        [ 38.30 s] [14652 MB]
  Finished Initializing                                                                  [ 56.32 s] [14652 MB]
Finished Setting Up                                                                      [447.64 s] [14652 MB]
...

Parallelism:
  Num Processors:          32
  Num Threads:             32

In terms of the actual solves, and perhaps to be expected, the preconditioning was not as good as HYPRE boomeramg (with 32 threads instead of 112).

Using the default block Jacobi required near 300 linear iterations per NL iteration. With boomeramg, it required 10. Looking at the job wall clock times, it seems like the boomeramg job w 32 threads took 22m09s where the block Jacobi with 112 threads took 47m58s. Maybe the slowdown for heat conduction + THM is worth it for the 3.5x threads advantage in OpenMC when we add neutronics?

[nuclearkevin]

I'd say getting a (anticipated) 350% speedup for OpenMC (by using threading) vs a 50% slowdown for heat conduction is likely a worthwhile tradeoff, especially once you add more nuclides to your materials when depleting. This is a hunch on my part though; it's hard to estimate what the performance difference will actually look like until you run both threading scenarios (with OpenMC) and compare the difference.

[lewisgross1296]

I think the OpenMP threads only help during transport since it's MPI doing depletion in parallel, but even still you're probably right about the trade-off being net positive. From previous experience, Monte Carlo neutron transport is the most expensive part of my simulation, so 3.5x goes a long way there, whereas heat conduction is going to be marginally slower considering actual wall clock time.

On the other hand, who can we bother to fix the HYPRE error because boomeramg + 112 threads would be very useful. I was unsure how to make a MWE but it sounds like it's a known past issue so maybe they don't need an MWE from my example? (which would be a maximal working example 😂)