Refactoring the new Async mode additions (apt-sim#321)

The goal of this PR is to adapt the new Asynchronous AdePT components to
the way the library is structured. This involves keeping a clear
separation between AdePT and the integration layer.

Main changes:

- Remove `AdePTConfiguration` from the integration layer.
This is an utility class which holds the AdePT configuration for a run,
it is part of AdePT and independent from the integration layer. However
the new `AdePTConfiguration::CreateAdePTInstance` is compiled as part of
`AdePTG4Integration`. In practice, this isn't a change from the previous
implementation, where the integration layer has the responsability of
initializing `AdePTConfiguration`, and also of passing the values in
`AdePTConfiguration` to `AdePTTransport`.
This will be changed in this PR, by introducing a new `AdePTTransport`
constructor which takes an `AdePTConfiguration` instance as an argument.
Now the integration layer is only responsible of initializing
`AdePTConfiguration`.
- `AdePTConfiguration` will now also store the number of threads, which
is used to compute the number of tracks and step slots allocated to each
thread in synchronous mode. By doing this, we will also remove the need
to store the number of slots per thread as `static inline` variables in
`AdePTTransport`. This was done because in Geant4 we can only get the
correct number of threads from the master, as the workers will report 1,
thus we can only compute the space per thread from the master. However
this is a G4-specific requirement that is not related to AdePT, so now,
instead of doing that, we store the number of threads as a `static
inline` variable in `AdePTTrackingManager`, on the integration side,
which each thread then passes on to `AdePTConfiguration`.

- Remove the `AdePTTransportFactory` method in favour of using the
`AdePTTransport` constructor.
This method is currently implemented as a free function that is part of
the integration layer. It is called from
`AdePTConfiguration::CreateAdePTInstance`.
In principle, it wouldn't be an issue to keep one factory per transport
backed, per integration layer, however at least for the synchronous
backend there seems to be little reason to use a factory method instead
the `AdePTTransport` constructor.
In order to 

- Implement a constructor in `AdePTTransport` which takes an
`AdePTConfiguration` instance as an argument.
In the current implementation, the integration layer is supposed to
initialize `AdePTConfiguration` and also to pass those values on to
`AdePTTransport`. With this change, the integration layer only needs to
initialize the configuration.

- Move the instantiation of `ShowerGPU` back to to
`AdePTTrackingManager.cu`.
We need to instantiate `ShowerGPU` in a `.cu` file so that this function
is compiled by `nvcc`. This is a necessary evil, and needs to be done in
the integration layer since AdePT is a header-only library. This
function has currently been moved to `AdePTTransport.cu`, which lives on
the integration layer. As with the `AdePTConfiguration` change, in
practice the integration layer is still responsible of doing this
instantiation, it just lives on a different file. Since
`AdePTTransport.cu` is not actually related to `AdePTTransport`, and it
is `AdePTTrackingManager` that calls this function, I believe it is less
misleading to keep this in `AdePTTrackingManager.cu`.

- Remove the functionality to keep tracks returned by AdePT on host.
There are two main reasons for this:
- This functionality is meant to keep loopers returned from AdePT on
host, however it doesn't distinguish between loopers and particles that
simply stepped outside of the GPU region. This may be implemented in the
future in a similar manner, by having a special ID for loopers if
needed.
- However in the future we may need to keep unique IDs for GPU tracks,
in order to be able to identify them on host, so we can't use the ID
field for this.

- Remove the call to the user stepping action from the tracking manager.
  - `G4SteppingManager::Stepping` already calls the stepping action

Author: JuanGonzalezCaminero

CMakeLists.txt

@@ -164,11 +164,11 @@ endif()
 #----------------------------------------------------------------------------#
 set(ADEPT_G4_INTEGRATION_SRCS
   src/AdePTTrackingManager.cc
+  src/AdePTTrackingManager.cu
   src/AdePTPhysics.cc
   src/HepEMPhysics.cc
   src/AdePTGeant4Integration.cpp
   src/AdePTConfigurationMessenger.cc
-  src/AdePTConfiguration.cc
 )
 
 add_library(CopCore INTERFACE)
@@ -180,8 +180,6 @@ target_include_directories(CopCore
 
 add_library(AdePT_G4_integration SHARED
   ${ADEPT_G4_INTEGRATION_SRCS}
-  src/AdePTTransport.cc
-  src/AdePTTransport.cu
 )
 target_include_directories(AdePT_G4_integration
   PUBLIC

include/AdePT/core/AdePTConfiguration.hh

@@ -25,8 +25,9 @@ std::shared_ptr<AdePTTransportInterface> AdePTTransportFactory(unsigned int nThr
 ///
 class AdePTConfiguration {
 public:
-  AdePTConfiguration();
-  ~AdePTConfiguration();
+  AdePTConfiguration() : fAdePTConfigurationMessenger{new AdePTConfigurationMessenger(this)} {}
+  ~AdePTConfiguration() {}
+  void SetNumThreads(int numThreads) { fNumThreads = numThreads; }
   void SetTrackInAllRegions(bool trackInAllRegions) { fTrackInAllRegions = trackInAllRegions; }
   void AddGPURegionName(std::string name) { fGPURegionNames.push_back(name); }
   void SetAdePTActivation(bool activateAdePT) { fAdePTActivated = activateAdePT; }
@@ -42,6 +43,7 @@ public:
 
   bool GetTrackInAllRegions() { return fTrackInAllRegions; }
   bool IsAdePTActivated() { return fAdePTActivated; }
+  int GetNumThreads() { return fNumThreads; };
   int GetVerbosity() { return fVerbosity; };
   int GetTransportBufferThreshold() { return fTransportBufferThreshold; }
   int GetCUDAStackLimit() { return fCUDAStackLimit; }
@@ -50,14 +52,13 @@ public:
   double GetMillionsOfHitSlots() { return fMillionsOfHitSlots; }
   std::vector<std::string> *GetGPURegionNames() { return &fGPURegionNames; }
 
-  std::shared_ptr<AdePTTransportInterface> CreateAdePTInstance(unsigned int nThread);
-
   // Temporary
   std::string GetVecGeomGDML() { return fVecGeomGDML; }
 
 private:
   bool fTrackInAllRegions{false};
   bool fAdePTActivated{true};
+  int fNumThreads;
   int fVerbosity{0};
   int fTransportBufferThreshold{200};
   int fCUDAStackLimit{0};

include/AdePT/core/AdePTTransport.h

@@ -19,6 +19,7 @@
 #include "CommonStruct.h"
 #include <AdePT/core/AdePTScoringTemplate.cuh>
 #include <AdePT/core/HostScoringStruct.cuh>
+#include <AdePT/core/AdePTConfiguration.hh>
 
 class G4Region;
 struct GPUstate;
@@ -32,7 +33,7 @@ class AdePTTransport : public AdePTTransportInterface {
   using TrackBuffer                = adeptint::TrackBuffer;
   using VolAuxArray                = adeptint::VolAuxArray;
 
-  AdePTTransport() = default;
+  AdePTTransport(AdePTConfiguration &configuration);
 
   ~AdePTTransport() { delete fScoring; }
 
@@ -76,8 +77,8 @@ class AdePTTransport : public AdePTTransportInterface {
 
 private:
   static inline G4HepEmState *fg4hepem_state{nullptr}; ///< The HepEm state singleton
-  static inline int fCapacity{1024 * 1024};            ///< Track container capacity on GPU
-  static inline int fHitBufferCapacity{1024 * 1024};   ///< Capacity of hit buffers
+  int fCapacity{1024 * 1024};                          ///< Track container capacity on GPU
+  int fHitBufferCapacity{1024 * 1024};                 ///< Capacity of hit buffers
   int fNthreads{0};                                    ///< Number of cpu threads
   int fMaxBatch{0};                                    ///< Max batch size for allocating GPU memory
   int fNumVolumes{0};                                  ///< Total number of active logical volumes

include/AdePT/core/AdePTTransport.icc

@@ -38,6 +38,22 @@ void ShowerGPU(IntegrationLayer &integration, int event, TrackBuffer &buffer, GP
 
 } // namespace adept_impl
 
+template <typename IntegrationLayer>
+AdePTTransport<IntegrationLayer>::AdePTTransport(AdePTConfiguration &configuration)
+{
+  fDebugLevel = 0;
+  fBufferThreshold = configuration.GetTransportBufferThreshold();
+  fMaxBatch = 2 * configuration.GetTransportBufferThreshold();
+  fTrackInAllRegions = configuration.GetTrackInAllRegions();
+  fGPURegionNames = configuration.GetGPURegionNames();
+  fCUDAStackLimit = configuration.GetCUDAStackLimit();
+  fCapacity = 1024 * 1024 * configuration.GetMillionsOfTrackSlots() / configuration.GetNumThreads();
+  fHitBufferCapacity = 1024 * 1024 * configuration.GetMillionsOfHitSlots() / configuration.GetNumThreads();
+  
+  printf( "AdePT Allocated track capacity: %d tracks\n", fCapacity);
+  printf( "AdePT Allocated step buffer capacity: %d tracks\n", fHitBufferCapacity);
+}
+
 template <typename IntegrationLayer>
 bool AdePTTransport<IntegrationLayer>::InitializeField(double bz)
 {

include/AdePT/integration/AdePTTrackingManager.hh

@@ -9,6 +9,7 @@
 
 #include "globals.hh"
 #include <AdePT/core/AdePTTransportInterface.hh>
+#include <AdePT/core/AdePTTransport.h>
 #include "AdePT/copcore/SystemOfUnits.h"
 #include <AdePT/integration/AdePTGeant4Integration.hh>
 #include <AdePT/core/AdePTConfiguration.hh>
@@ -49,9 +50,9 @@ private:
   /// @brief Steps a track using the Generic G4TrackingManager until it enters a GPU region or stops
   void StepInHostRegion(G4Track *aTrack);
 
+  static inline int fNumThreads{0};
   std::set<G4Region const *> fGPURegions{};
   int fVerbosity{0};
-
   std::shared_ptr<AdePTTransportInterface> fAdeptTransport;
   AdePTConfiguration *fAdePTConfiguration;
   unsigned int fTrackCounter{0};

src/AdePTConfiguration.cc

@@ -513,7 +513,6 @@ void AdePTGeant4Integration::ReturnTrack(adeptint::TrackData const &track, unsig
   G4Track *secondary = new G4Track(dynamique, track.globalTime, posi);
   secondary->SetLocalTime(track.localTime);
   secondary->SetProperTime(track.properTime);
-  secondary->SetTrackID(kAdePTTrackID);
   secondary->SetParentID(track.parentID);
 
   G4EventManager::GetEventManager()->GetStackManager()->PushOneTrack(secondary);

src/AdePTGeant4Integration.cpp

@@ -28,21 +28,25 @@ AdePTTrackingManager::~AdePTTrackingManager()
 
 void AdePTTrackingManager::InitializeAdePT()
 {
-  const auto num_threads = G4RunManager::GetRunManager()->GetNumberOfThreads();
-
   // Check if this is a sequential run
   G4RunManager::RMType rmType = G4RunManager::GetRunManager()->GetRunManagerType();
   bool sequential             = (rmType == G4RunManager::sequentialRM);
 
   // One thread initializes common elements
   auto tid = G4Threading::G4GetThreadId();
   if (tid < 0) {
+    // Only the master thread knows the actual number of threads, the worker threads will return "1"
+    // This value is stored here by the master in a static variable, and used by each thread to pass the 
+    // correct number to their AdePTConfiguration instance
+    fNumThreads = G4RunManager::GetRunManager()->GetNumberOfThreads();
+    fAdePTConfiguration->SetNumThreads(fNumThreads);
+
     // Load the VecGeom world in memory
     AdePTGeant4Integration::CreateVecGeomWorld(fAdePTConfiguration->GetVecGeomGDML());
 
     // Create an instance of an AdePT transport engine. This can either be one engine per thread or a shared engine for
     // all threads.
-    fAdeptTransport = fAdePTConfiguration->CreateAdePTInstance(num_threads);
+    fAdeptTransport = std::make_shared<AdePTTransport<AdePTGeant4Integration>>(*fAdePTConfiguration);
 
     // Initialize common data:
     // G4HepEM, Upload VecGeom geometry to GPU, Geometry check, Create volume auxiliary data
@@ -54,7 +58,8 @@ void AdePTTrackingManager::InitializeAdePT()
   } else {
     // Create an instance of an AdePT transport engine. This can either be one engine per thread or a shared engine for
     // all threads.
-    fAdeptTransport = fAdePTConfiguration->CreateAdePTInstance(num_threads);
+    fAdePTConfiguration->SetNumThreads(fNumThreads);
+    fAdeptTransport = std::make_shared<AdePTTransport<AdePTGeant4Integration>>(*fAdePTConfiguration);
     // Initialize per-thread data
     fAdeptTransport->Initialize();
   }
@@ -190,10 +195,8 @@ void AdePTTrackingManager::ProcessTrack(G4Track *aTrack)
     // Check if the particle is in a GPU region
     const bool isGPURegion = trackInAllRegions || fGPURegions.find(region) != fGPURegions.end();
 
-    if (isGPURegion &&
-        aTrack->GetTrackID() != AdePTGeant4Integration::kAdePTTrackID /*track didn't come back from AdePT*/) {
+    if (isGPURegion) {
       // If the track is in a GPU region, hand it over to AdePT
-
       auto particlePosition  = aTrack->GetPosition();
       auto particleDirection = aTrack->GetMomentumDirection();
       G4double energy        = aTrack->GetKineticEnergy();
@@ -240,16 +243,11 @@ void AdePTTrackingManager::StepInHostRegion(G4Track *aTrack)
   G4TrackingManager *trackManager    = eventManager->GetTrackingManager();
   G4SteppingManager *steppingManager = trackManager->GetSteppingManager();
   G4Region const *previousRegion       = aTrack->GetVolume()->GetLogicalVolume()->GetRegion();
-  G4UserSteppingAction *steppingAction = steppingManager->GetUserAction();
-  const bool stayOnHost                = aTrack->GetParentID() == AdePTGeant4Integration::kAdePTTrackID;
 
   // Track the particle Step-by-Step while it is alive and outside of a GPU region
   while ((aTrack->GetTrackStatus() == fAlive || aTrack->GetTrackStatus() == fStopButAlive)) {
     aTrack->IncrementCurrentStepNumber();
     steppingManager->Stepping();
-    if (steppingAction) steppingAction->UserSteppingAction(aTrack->GetStep());
-
-    if (stayOnHost) continue;
 
     if (aTrack->GetTrackStatus() != fStopAndKill) {
       // Switch the touchable to update the volume, which is checked in the