Event level parallelism in Geant4 Version 10.0

The threading model of Geant4 and several aspects and the new classes are explained in detail here: Geant4MTAdvandedTopicsForApplicationDevelopers
In this page we will concentrate on aspects that are important for kernel developers. In particular we will discuss the most important aspect for Geant4 Version 10.0 multi-threading model, memory handling, split-classes and thread-local storage.
A beginner guide to multi-threading targeted to Geant4 developers can be found in the 18th Collaboration Meeting: https://indico.cern.ch/getFile.py/access?contribId=3&sessionId=7&resId=0&materialId=slides&confId=250021

Memory handling in Geant4 Version 10.0


In Geant4 we distinguish two broad types of classes overall: ones whose instances are separate for each thread (such as a physics process, which has a state), and ones whose instances are shared between threads (e.g. an element G4Element which holds constant data ).

In a few cases classes exist which are split - part of their state is constant, and part is per-worker. A simple example of this is a particle definitions, such as G4Electron, which holds both data (which is constant) and a pointer to the G4ProcessManager object for electrons - which must be different for each worker (thread).

We handle these 'split' classes specially, to enable data members and methods which correspond to the per-thread state to give a different result on each worker thread. The implementation of this requires an array for each worker (thread) and an additional indirection - which imposes a cost for each time the method is called.

Thread safety and sharing of objects

To better understand how memory is handled in multi-threaded Geant4 it is better to proceed with a simplified example:
Let us consider a class G4Class that, for simplicity, contains a single data member:
class G4Class {
    [static] G4double fValue; //static keyword is optional
Our goal is to transfer the code of G4Class to make it thread-safe. We define as thread-safe if more than one thread can simultaneously operate on the class data member of methods without interfering with each other in an unpredictable way. A classical way to proceed is to protect concurrent access to a shared memory location by thread using a lock on a mutex. However this technique substantially reduce performances because only one thread at a time is allowed to be executed. In Geant4 Version 10.0 we have achieved thread safety via the use of thread local storage. For an explanation of what is thread local storage several sources exists, for a basic introduction adequate for our discussion, web resources give enough details (e.g. wikipedia).
We define an instance of a variable thread-local (or thread-private) if each thread owns a copy of the variable. On the other side a thread-shared variable is an instance of a variable that is shared among the threads (i.e. all thread have access to the same memory location that holds the value of the variable). In addition, if we need to share fValue between several instances of G4Class we call the data field instance-shared otherwise it is instance-local.
It is easy understandable that in the case of thread-shared variables thread needs synchronization to avoid race condition (it is worth to remind that there are no race conditions if the variable is accessed only to be read, for example in the case the variable is marked as const).
One or more instances of G4Class can exist in our application. These instances can be thread-local (e.g. a G4VProcess) or thread-shared (e.g. a G4LogicalVolume). In addition the class data field fValue can be by itself thread-local or thread-shared. The action to be taken to transform the code to be thread-safe depends on three key aspects:
  • Do we need to make the instance(s) of G4Class thread-local or thread-shared?
  • Do we need to make the data field fValue thread-local or thread-shared?
  • In case more than one instance of G4Class exits, do we need fValue to be instance-local or instance-shared?
This gives rise to 8 different possible combinations, summarized in the following figures:

It is worth to remind that in Geant4 only few well identified class instances are thread-shared: geometry, particle-definition, materials, EM data-tables.

Case A: thread-local class instance(s), thread-shared and instance-shared data field

In this case each thread has its own instance(s) of G4Class. We need to share fValue both among threads and among instances. As for a sequential application we can simply add the static keyword to the declaration of fValue. This technique is very common in Geant4 but has the disadvantage that the result code is not thread-safe (unless locks are used) unless the shared variable is also declared as const. Trying to add const (or modify its value, with the use of a lock) outside of the event loop is the best solution:
class G4Class {
    static const G4double fValue;

Case B: thread-local class instance(s), thread-local and instance-shared data field.

This scenario is common in Geant4: we need to share a variable (e.g. a data table) between instances of the same class. However it is impractical to share among threads this data field. To make the code thread-safe we can make the data field thread-local declaring the variable thread-local-storage:
#include "G4Types.hh"
class G4Class {
    static G4ThreadLocal G4double fValue;
It should be noted that only simple data types can be declared G4ThreadLocal. More information and the procedure to make an object instance G4ThreadLocal is explained in Geant4MTTipsAndTricks#4_Why_I_cannot_simply_add_G4Thre

Case C: thread-local class instance(s), thread-shared and instance-local data field

This case is probably the less frequent. A possible use-case is the reduction of application memory footprint, providing a thread-shared component to the thread-private instances of G4Class (e.g. a cross-section data table). Since this scenario strongly depends on the implementation details it is not possible to define a common strategy to guarantee thread-safety. The best one being to try to make this shared component const.

Case D: thread-local class instance(s), thread-local and instance-local data field

This case is the simplest, nothing has to be changed in the original code.

Case E: thread-shared class instance(s), thread-shared and instance-shared data field

This case is equivalent to Case A, and the same recommendations and comments are valid.

Case F: thread-shared class instance(s), thread-local and instance-shared data field

This case is equivalent to Case B, and the same recommendations and comments are valid.

Case G: thread-shared class instance(s), thread-shared and instance-shared data field

Since the class instances are shared among threads the data field are automatically thread-shared. No action is needed, however access to data field is not, in general thread safe, and the same comments as for case A holds.

Case H: thread-sahred class instance(s), thread-local and instance-local data field

This is the most complex case and is quite common in Geant4 Version 10.0. For example G4ParticleDefinition instances are shared among the threads, but the G4ProcessManager needs to be thread and instance local. To obtain this there are two possible solutions:
  • Use the split-class mechanism. This requires some deep understanding of Geant4 MT and coordination with kernel developers. This code is recognized as critical and while provides a thread-safe code with good CPU performances it also requires modification in other aspects of kernel category (in particular the introduction of a new split-class requires changes in run category). The idea behind the split-class mechanism is that each thread-shared instance instance of G4Class initializes the thread-local data fields copying the initial status from the master thread that is guaranteed to be fully configured. Additional details on split classes are available in this chapter.
  • If performances are not a concern a simple solution is available. It is a simplified version of the split-class mechanism that does not copy the initial status of the thread-local data field from the master thread. A typical example is a cache variable that reduces CPU usage keeping in memory a value of a CPU intensive calculation that depends on the current event. In such a case the G4Cache utility class can be employed:
#include "G4Cache.hh"
class G4Class {
    G4Cache<G4double> fValue;
    void foo() {
          // Store a thread-local value
          G4double val = someHeavyCalc();
          fValue.Put( val );
    void bar() {
         //Get a thread-local value:
         G4double local = fValue.Get();

Details on the split classes mechanism

Copy from Geant4MTTipsAndTricks#10_Splitting_of_classes.

List of split-classes

In Geant4 Version 10.0 the following are split-classes:
  • For geometry related split classes the class G4GeomSplitter implements the split-class mechanism. These are the geometry related split-classes:
  1. G4LogicalVolume
  2. G4PhysicalVolume
  3. G4PVReplica
  4. G4Region
  5. G4PloyconeSide
  6. G4PolyhedraSide
  • For Physics related split-classes the classes G4PDefSplitter and G!4VUPLSplitter implement the split-class mechanism. These are the physics related split-classes:
  1. G4ParticleDefinition
  2. G4VUserPhysicsList
  3. G4VModularPhysicsList
  4. G4VPhysicsConstructor

Explicit memory handling in Geant4 Version 10.0

Thread Private singleton

G4AutoDelete namespace

G4Cache template class

Topic attachments
I Attachment History Action Size Date Who Comment
JPEGjpg SplitClasses-MemHandling.001.jpg r1 manage 178.5 K 2013-12-05 - 00:37 AndreaDotti  
JPEGjpg SplitClasses-MemHandling.002.jpg r1 manage 193.8 K 2013-12-05 - 00:37 AndreaDotti  
Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r6 - 2013-12-05 - AndreaDotti
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Geant4 All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback