Event level parallelism in Geant4 Version 10.0

• The new run-manager class G4MTRunManager (that inherits from G4RunManager) implements the master model. It uses the mandatory class G4MTRunManagerKernel the multi-threaded equivalent of G4RunManagerKernel.
• The new run-manager class G4WorkerRunManager (that inherits from G4WorkerRunManager) implements the worker model. It uses the mandatory class G4WorkerRunManagerKernel the worker-model equivalent of G4RunManagerKernel
• The new user-initialization class G4VUserActionInitialization is responsible for the instantiation of thread-local user actions
• The new user-initialization class G4UserWorkerInitialization is responsible for the initialization of worker threads

In this page we will concentrate on aspects that are important for kernel developers, in particular we will discuss the most critical aspect for Geant4 Version 10.0 multi-threading: memory handling, split-classes and thread-local storage.
A beginner guide to multi-threading targeted to Geant4 developers has been presented during the 18th Collaboration Meeting: https://indico.cern.ch/getFile.py/access?contribId=3&sessionId=7&resId=0&materialId=slides&confId=250021

Memory handling in Geant4 Version 10.0

Introduction

In a few cases classes exist which are split - part of their state is constant, and part is per-worker. A simple example of this is a particle definitions, such as G4Electron, which holds both data (which is constant) and a pointer to the G4ProcessManager object for electrons - which must be different for each worker (thread).

We handle these 'split' classes specially, to enable data members and methods which correspond to the per-thread state to give a different result on each worker thread. The implementation of this requires an array for each worker (thread) and an additional indirection - which imposes a cost for each time the method is called. However this overhead is small and has been measured to be about 1%.

Thread safety and sharing of objects

Let us consider the simplest possible class G4Class that consists of a single data member:

class G4Class {
    [static] G4double fValue; //static keyword is optional
};

A classical way to solve this problem is to protect concurrent access to a shared memory location using a lock or a mutex (see section Threading model utilities and functions. However this technique can reduce overall performances because only one thread at a time is allowed to be executed. In Geant4 Version 10.0 we have achieved thread safety via the use of thread local storage. For an explanation of what is thread local storage several sources exists, for a basic introduction adequate for our discussion, web resources give enough details (e.g. wikipedia).

We define an instance of a variable thread-local (or thread-private) if each thread owns a copy of the variable. A thread-shared variable is an instance of a variable that is shared among the threads (i.e. all thread access to the same memory location that holds the value of the variable). In addition, if we need to share the same fValue between several instances of G4Class we call the data field instance-shared otherwise it is instance-local.

It is clear that, for the case of thread-shared variables, all thread needs synchronization to avoid race condition (it is worth to remind that there are no race conditions if the variable is accessed only to be read, for example in the case the variable is marked as const).

One or more instances of G4Class can exist in our application. These instances can be thread-local (e.g. G4VProcess) or thread-shared (e.g. G4LogicalVolume). In addition the class data field fValue can be by itself thread-local or thread-shared. The actions to be taken to transform the code depend on three key aspects:

• Do we need to make the instance(s) of G4Class thread-local or thread-shared?
• Do we need to make the data field fValue thread-local or thread-shared?
• In case more than one instance of G4Class exits at the same time, do we need fValue to be instance-local or instance-shared?

Case A: thread-local class instance(s), thread-shared and instance-shared data field

class G4Class {
    static const G4double fValue;
};

Case B: thread-local class instance(s), thread-local and instance-shared data field.

#include "G4Types.hh"
class G4Class {
    static G4ThreadLocal G4double fValue;
};

Case C: thread-local class instance(s), thread-shared and instance-local data field

Case D: thread-local class instance(s), thread-local and instance-local data field

Case E: thread-shared class instance(s), thread-shared and instance-shared data field

Case F: thread-shared class instance(s), thread-local and instance-shared data field

Case G: thread-shared class instance(s), thread-shared and instance-shared data field

Case H: thread-sahred class instance(s), thread-local and instance-local data field

• Use the split-class mechanism. This requires some deep understanding of Geant4 multi-threading and coordination with the kernel developers. Split-classes result in thread-safe code with good CPU performances, however they also require modification in other aspects of kernel category (in particular they require changes in run category). The idea behind the split-class mechanism is that each thread-shared instance of G4Class initializes the thread-local data fields copying the initial status from the master thread, guaranteed to be fully configured. Additional details on split classes are available here.
• If performances are not a concern a simpler solution is available. This is a simplified version of the split-class mechanism that does not copy the initial status of the thread-local data field from the master thread. A typical example is a cache variable that reduces CPU usage keeping in memory the value of a CPU intensive calculation. In such a case the G4Cache utility class can be employed (see here).

Details on the split classes mechanism

class G4Class
{
 private:
    G4double fValue;
 public:
    G4Class() {  }
    void SetMyData( G4double aValue ) { fValue = aValue; }
    G4double GetMyData() const { return fValue; }
};

At first we can add to the declaration of fValue the TLS keyword G4ThreadLocal. Unfortunately there are several constraints on what can be specified as TLS (G4ThreadLocal in a POSIX system is a typedef to __thread), in particular the data member has to be declared static:

#include "tls.hh"
class G4Class 
{
 private:
    static G4ThreadLocal G4double fValue;
 public:
    G4Class()  {  }
    void SetMyData( G4double aValue ) { fValue = aValue; }
    G4double GetMyData() const { return fValue; }
};
G4ThreadLocal G4double G4Class::fValue = -1; 

We can modify the code as follow:

class G4ClassData {
public:
   G4double fValue;
   void intialize() {
           fValue = -1;
   }
};

typedef G4Splitter<G4ClassData> G4ClassManager;
typedef G4ClassManager G4ClassSubInstanceManager;

#define G4MT_fValue ((subInstanceManager.offset[gClassInstanceId]).fValue)
class G4Class {
private: 
    G4int gClassInstanceId;
    static G4ClassSubInstanceManager subInstanceManager;
public:
    G4Class()
   { 
      gClassInstanceId = subInstanceManager.CreateSubInstance();
    }
    void SetMyData( G4double aValue ) { G4MT_fValue = aValue; }
    G4double GetMyData() const { return G4MT_fValue; }
};

G4ClassSubInstanceManager G4Class::subInstanceManager;
template <class G4ClassData> G4ThreadLocal G4int G4Splitter<G4ClassData>::workertotalspace = 0;
template <class G4ClassData> G4ThreadLocal G4int G4Splitter<G4ClassData>::offset = 0;

As one can see the use of the value of fValue variable is very similar to how we use it in the original sequential mode, all the handling of the TLS is done in the template class G4Splitter that can be implemented as:

template <class T>
class G4Splitter
{
  private:
    G4int totalobj;
  public:
    static G4ThreadLocal G4int workertotalspace;
    static G4ThreadLocal T* offset;
  public:
    G4Splitter() : totalobj(0) {}
    G4int CreateSubInstance() 
    {
       totalobj++;
       if ( totalobj > workertotalspace ) { NewSubInstances(); }
       return (totalobj-1); 
    }
    void NewSubInstances()
    {
         if ( workertotalspace >=totalobj ) { return; }
         G4int originaltotalspace = workertotalspace;
         workertotalspace = totalobj + 512;
         offset = (T*) realloc( offset , workertotalspace * sizeof(T) );
         if ( offset == 0 ) 
         { 
            G4Excepetion( "G4Splitter::NewSubInstances","OutOfMemory",FatalException,"Cannot malloc space!");
         }
        for ( G4int i = originaltotalspace; i < workertotalspace ; i++)
        {
            offset[i].intialize();
        }
    }
   void FreeWorker()
   {
     if ( offset == 0 ) { return; }
     delete offset;
   }
};

Let's Imagine now a function that can be called concurrently by more than one thread:

#include "G4Class.hh"
//Variables at global scope
G4Class a; 
G4Class b;

void foo() 
{
     a.SetMyData(0.1); //First instance
     b.SetMyData(0.2); //Second instance
   G4cout<< a.GetMyData() << " "<< b.GetMyData() << G4endl;
}

totalobj = 0
TLS workertotalspace = 0 
TLS offset = NULL (remember this is a pointer!)

subInstanceManager.offset[0].fValue = 0.1;

Since offset and workertotalspace are G4ThreadLocal this mechanism allows each thread to have its own copy of fValue. The function foo() can be called from different threads and they will use the thread-shared a and b to access a thread-local fValue data field..

A realistic example of this can be seen in the class G4ParticleDefinition. An additional complication is when the initialization of the thread-local part is not trivial and we want to "copy" some values from the corresponding values of the master thread (in our example, how to initialize fValue to a default value at run time?). How to obtain this is shown in G4LogicalVolume, but it should be noted that the worker thread need to call a special function to initialize correctly the worker thread memory space. For this reason Geant4 kernel code needs to be adapted every time a new class is declared split.

The following diagram shows the chain of calls in G4ParticleDefinition when a thread needs to access a process pointer:

List of split-classes

• For geometry related split classes the class G4GeomSplitter implements the split-class mechanism. These are the geometry related split-classes:

1. G4LogicalVolume
2. G4PhysicalVolume
3. G4PVReplica
4. G4Region
5. G4PloyconeSide
6. G4PolyhedraSide

• For Physics related split-classes the classes G4PDefSplitter and G!4VUPLSplitter implement the split-class mechanism. These are the physics related split-classes:

1. G4ParticleDefinition
2. G4VUserPhysicsList
3. G4VModularPhysicsList
4. G4VPhysicsConstructor

Explicit memory handling

G4Cache template class

In many cases this is not needed and what we really want are independent thread-local and instance-local data field in thread-shared instances of G4Class. For example a class representing a cross-section is made shared because of its memory footprint. However it requires a data field to act as a cache to store a value of a CPU intensive calculation. Since different thread share this instance we need to transform the code in a manner similar to what we do for split-classes mechanism. The helper class G4Cache can be used for this purpose.

This is a template class that implements a light-weight split-classes mechanism. Being a template it allows for storing any user-defined type. The public API of this class is very simple and it provides two methods

T& G4Cache<T>::Get() const;
void G4Cache<T>::Put(const T& val) const;

#include "G4Cache.hh"
class G4Class {
    G4Cache<G4double> fValue;
    void foo() {
          // Store a thread-local value
          G4double val = someHeavyCalc();
          fValue.Put( val );
    }
    void bar() {
         //Get a thread-local value:
         G4double local = fValue.Get();
    }
};

void G4Class::bar() {
    //Get a reference to the thread-local value:
   G4double& local = fValue.Get();
   // Use local as in the original sequential code, cache is updated, without the need to use Put
  local++;
}

#include "G4Cache.hh"
class G4Class {
    G4Cache<G4Something*> fValue;
    void bar() {
         //Get a thread-local value:
         G4Something* local = fValue.Get();
         if ( local == 0 ) {
                   local = new G4Something( …. );
                   //warning this may cause a memory leak. Use of G4AutoDelete can help, see later
         }
    }
};

The use of G4Cache implies some CPU penalty, it is a good practice to try to minimize the use of G4Cache. For example, do not use them in several data field separately, but use a helper structure and use this structure as template parameter:

class G4Class {
   struct {
       G4double fValue1;
       G4Something* fValue2;
   } ToBeCached_t;
  G4Cache&ltToBeCached_t&gt fCache;
};

Finally two specialized versions of G4Cache exist that implement the semantics of std::vector and std::map:

• G4VectorCache<T> implements thread-local std::vector<T> = with methods: =Push_back(…) , operator[], Begin(), End(), Clear(), Size(), Pop_back()
• G4MapCache<K,V> implements thread-local std::map<K,V> with methods: Insert(…), Begin(), End(), Find(…), Size(), Get(…), Erase(…), operator[] and introduces the method Has(…)

G4AutoDelete namespace

#include "G4AutoDelete.hh"
void G4Class::bar() {
    //Get a thread-local value:
    G4Something* local = fValue.Get();
    if ( local == 0 ) {
            local = new G4Something( …. );
            G4AutoDelete::Register( local ); //All thread instances will be delete automatically
    }
}

This technique will delete all instances of the registered objects at the end of the program after the main function has returned (as they would have been declared static).

This method requires some attention and has several limitations: 1 Registered objects will be deleted only at the end of the program 1 The order in which objects of different type will be deleted is not specified 1 Once an object is registered it cannot be deleted anymore explicitly by user 1 The objects that are registered with this method cannot contain G4ThreadLocal data and cannot be split-classes 1 Registered object cannot make use of G4Allocator In particular since the objects will be deleted after the main program exit in a non-specified order their destructors should be simple and should not depend on other objects.

Thread Private singleton

class G4Singleton {
public:
   G4Singleton* GetInstance() {
        static G4Singleton anInstance;
        return &anInstance;
   }   
};

With multi-threading many managers and singletons are required to become thread-local. For this reason they have been transformed to:

class G4Singleton {
private:
   static G4ThreadLocal* instance;
public:
   G4Singleton* GetInstance() {
        if ( instance == 0 ) instance = new G4Singleton;
        return instance;
   }   
};

This causes a memory leak since it is not possible to delete thread-local instances of the singletons. The class G4ThreadLocalSingleton can be used to solve this problem. This template class has a single public method T* G4ThreadLocalSingleton<T>::Instance() that returns a pointer to a thread-local instance of T.

The example code can be transformed to:

#include "G4ThreadLocalSingleton.hh"
class G4Singleton {
   friend class G4ThreadLocalSingleton<G4Singleton>;
public:
   G4Singleton* GetInstance() {
       static G4ThreadLocalSingleton<G4Singleton> theInstance; 
       return theInstance.Instance();
   }   
};

Threading model utilities and functions

Thread related types and functions.

Example:

//Define a thread-function using G4 types
G4ThreadFunReturnType myfunc(  G4ThreadFunArgType val) {
  double value = *(double*)val;
  MESSAGE("value is:"<<value);
  return /*(G4ThreadFunReturnType)*/NULL;
}

//Example: spawn 10 threads that execute myfunc
int main(int,char**) {
  MESSAGE( "Starting program ");
  int nthreads = 10;
  G4Thread* tid = new G4Thread[nthreads];
  double *valss = new double[nthreads];
  for ( int idx = 0 ; idx < nthreads ; ++idx ) {
    valss[idx] = (double)idx;
    G4THREADCREATE( &(tid[idx]) , myfunc, &(valss[idx]) );
  }
  for ( int idx = 0 ; idx < nthreads ; ++idx ) {
    G4THREADJOIN( (tid[idx]) );
  }
  MESSAGE( "Program ended ");
  return 0;
}

Mutex/conditions related types and functions

Example:

//Create a global mutex
G4Mutex mutex = G4MUTEX_INITIALIZER; //Alternatively, call G4MUTEXINIT(mutex);

//A shared resource (e.g. manipulated by all threads)
G4int aValue = 1;

G4ThreadFunReturnType myfunc(  G4ThreadFunArgType ) {
  //Explicit lock/unlock
  G4MUTEXLOCK( &mutex );
  ++aValue;
  G4MUTEXUNLOCK( &mutex );
  //The following should be used instead of the previous because it guarantees automatic
  //unlock of mutex.
  //When variable l goes out of scope, G4MUTEXUNLOCK is automatically called
  G4AtuoLock l(&mutex);
  --aValue;
  //Explicit lock/unlock. Note that lock/unlock is not tried if already locked/unlock
  l.lock();
  l.lock();//No problem here
  ++aValue;
  l.unlock();
  l.lock();
  --aValue;
  return /*(G4ThreadFunReturnType)*/NULL;
}

Conditions are also available via the G4Condition type, G4CONDITION_INITIALIZER macro and the two functions G4CONDITIONWAIT and G4CONDITIONBORADCAST. The use of condition allows to implement the barrier mechanism (e.g. synchronization point for threads). A detailed example on the use of condition and how to implement correctly a barrier is discussed in G4MTRunManager code (at the end of file source/run/src/G4MTRunManager.cc).

Additional material

18th Collaboration Meeting

Plenary Session 3 - Geant4 version 10 (part 1)

Parallel session 7B - Hadronics issues related to MT

Developments for multi-threading: work-spaces

Status of the planned developments: coding guidelines, MT migration, g4tools migration, code review

G4MT CP on MIC Architecture

• X. Dong et al, Creating and Improving Multi-Threaded Geant4 , Journal of Physics: Conference Series 396, no. 5, p. 052029.
• X. Dong et al, Multithreaded Geant4: Semi-automatic Transformation into Scalable Thread-Parallel Software, Euro-Par 2010 - Parallel Pro- cessing (2010), vol. 6272, pp. 287-303.
• S. Ahn et al, Geant4-MT: bringing multi-threaded Geant4 into production, to be published in SAN&MC2013 proceeding