EICRecon crashes due to topological clustering algorithm being too slow

[marrbnl]

When reconstructing 10x100 eAu events after applying beam effects using "ip6_eau_110x10", the EICReco crashes with the following error message:

14:51:29.515  [info] Processing paused.
14:51:29.516 [fatal] Timeout in worker thread 0 while executing PhysicsEventMap2 on event #0
14:51:29.516  [info] Worker thread signalled; waiting for backtrace capture.
----------------------------------------------------------
JException
  Message:  Timeout in worker thread
  Backtrace:

   0: std::_Rb_tree_increment(std::_Rb_tree_node_base const*)
      /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/gcc-runtime-12.2.0-c4pnjmu2myklfcam34jsztp6cbuvyqnz/lib/libstdc++.so.6:bfcc7

   1: void eicrecon::ImagingTopoCluster::bfs_group<eicrecon::ImagingTopoCluster::process(std::tuple<gsl::not_null<edm4eic::CalorimeterHitCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::ProtoClusterCollection*> > const&) const::{lambda(auto:1 const&, auto:2 const&)#1}>(edm4eic::CalorimeterHitCollection const&, std::set<unsigned long, eicrecon::ImagingTopoCluster::process(std::tuple<gsl::not_null<edm4eic::CalorimeterHitCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::ProtoClusterCollection*> > const&) const::{lambda(auto:1 const&, auto:2 const&)#1}, std::allocator<unsigned long> >&, std::__cxx11::list<unsigned long, std::allocator<unsigned long> >&, unsigned long) const
      /opt/local/lib/EICrecon/plugins/BEMC.so:cdce5

Dmitry and Alex pointed out that this is due to topological clustering algorithm is too slow as the afterburner would allow for neutrons to hit the ZDC, so a Au nucleus breaking up and depositing several neutrons into the ZDC, while many hadrons potentially hit the imaging EMCAL.

[wdconinc]

We can also just disable the timeout everywhere. Some events just take a while to analyze.

[wdconinc]

Interesting that this happens, because we already tried to fix this once for the ZDC in #1581 (went from O(n^3) to O(n^2), if I recall correctly). I thought that took care of the low-hanging fruit (unfortunately).

[marrbnl]

I also observed that the time it takes for npsim to run increases by almost a factor of 5 by including beam effects for 10x100 eAu events, which is probably also related to ZDC.

[wdconinc]

Yes. And be happy if it only increases by a factor 5 :-) @ajentsch can tell you of the times when it took 5 minutes per event into the ZDC... Improvements welcome!

[marrbnl]

I tested the new xml file without ZDC. Here is the comparison of the time taken by npsim using 10x100 eAu events:

• With ZDC and without afterburner: ~ 120 s per event
• With ZDC and with afterburner: ~ 600 s per event
• Without ZDC and with afterburner: ~ 20 s per event