Simulation stalls after "Setting invalid value" warning (BoutException) — should we abort earlier?

[malamast]

Hi all,

In some runs the solver prints a warning indicating that an invalid value was set for a field, and shortly afterward the simulation appears to stall (time/steps stop making any progress and the run effectively hangs).

Example log snippet:
^M2.328e+06 198 4.67e+00 79.9 0.0 1.6 4.6 13.9
/ Step 2673 of 7228. Elapsed 3:49:58.8 ETA 5:54:14.9^MTime: 2333164.8061423623, timestep: 4173.610483127789, nl iter: 2, lin iter: 34, reason: 3
WARNING: BoutException thrown: Setting invalid value for 'e:energy_source'

The warning originates from the “invalid value” guard in component.hxx:
https://github.com/boutproject/hermes-3/blob/master/include/component.hxx#L255

Would it be better / acceptable to terminate the run immediately when this happens (i.e., treat it as a fatal error) rather than continuing after issuing a warning?

I use the SNES backward euler solver for these 2D transport simulations with neutrals (no drifts).

[mikekryjak]

This sounds similar to what Arun has seen recently with the PETSc TS solver.
As I understand from our discussion with @bendudson, the idea is that if there is an exception of this kind, then SNES would just try again. There's a try/except block for that here:

https://github.com/boutproject/BOUT-dev/blob/21a336ee2da7e1fe9ad564714e3f859e3994b9d8/src/solver/impls/snes/snes.cxx#L1209-L1221

But your experience clearly shows something's going wrong here. I don't know what it could be because I never dug deep into snes.cxx.

[malamast]

Oh I see. Not sure what's going on. I think, this exception "Setting invalid value for ..." occur when the solver becomes too aggressive and the timestep is too large. For now, I just set:
max_timestep = 1000
in the input file but we might need to further check why the simulation stalls instead of reducing the timestep.

[malamast]

The error occurs in snes_function here:
https://github.com/boutproject/BOUT-dev/blob/1ddf5b8cff07e67ea47ffc30f83570190aafd390/src/solver/impls/snes/snes.cxx#L1212
and for my test case, this happens when linear is true. This means that the rhs function was call for the construction of the jacobian here:
https://github.com/boutproject/BOUT-dev/blob/1ddf5b8cff07e67ea47ffc30f83570190aafd390/src/solver/impls/snes/snes.cxx#L77

I added SNESSetJacobianDomainError(snes); in the snes_function and SNESSetCheckJacobianDomainError(snes, PETSC_TRUE); in the constructor of the class but it didn't solve the problem. The solver stalls after throwing the warning.