OscarMTProducer: Multithreaded version of OscarProducer


This page aims to document the multithreading aspects of the OscarMTProducer.

For other information, please see SWGuideOscarProducer


Pull requests


The challenge

The Geant4 10 release adds support for multithreading. It is implemented with POSIX threads, and is compatible with CMSSW threading model (TBB). However, there are certain differences in the threading models of Geant4 and CMSSW that make the integration of Geant4 10 MT to CMSSW rather delicate business.

Geant4 10 MT has a master thread (in practice the main thread of the process) that does global initialization, and must not do any work. It spawns worker threads for simulating events in their own event loops. In CMSSW, the work is organized in tasks, and there is no guarantee on which thread the tasks get run.

Overall design

  • Implement OscarMTProducer as a "stream"-type module (see FWMultithreadedFrameworkModuleTypes)
  • Spawn std::thread to act as the G4MT "master thread"
  • Use the threads where TBB tasks are run (spawned by CMSSW framework/TBB) as G4MT "worker threads"



  • Is an edm::stream::EDProducer module


  • Is a "GlobalCache" of OscarMTProducer
  • Encapsulates the "master thread" (creation/communication/joining)
  • The std::thread is started from OscarMTProducer::initializeGlobalCache(), joined in OscarMTProducer::globalEndJob()
  • Begin and end of run are signalled via an enumeration and std::mutex + std::condition variable

The master thread is started in the constructor of OscarMTMasterThread, the function to be executed is given as a lambda with the variables shared by it and the CMSSW threads passed by reference via the lambda capture. The shared variables used are the mutable members of OscarMTMasterThread. The access to these variables from the multiple CMSSW threads is protected with m_protectMutex, that must be locked before accessing the variables. The master thread does not (need to) lock this mutex, since it is already locked by some CMSSW thread when master thread executes (and as discussed next, the execution of CMSSW thread(s) and the master thread is serialized). The other mutex, m_threadMutex, and the condition variables m_notifyMasterCv and m_notifyMainCv are used to signal the master or CMSSW threads, respectively, from the other. They need to be serialized, because most of the time master thread just sleeps, and while it does some work (= initialization or destruction of Geant4), there is no work CMSSW thread could do.

In order to technically support multiple runs in a job, the "master thread function" is implemented as a state loop. In the beginning of the loop, the master thread signals the CMSSW thread and goes to sleep. When signalled, the master thread checks the content of the state variable m_masterThreadState and acts accordingly.

In the Geant4 initialization, an important subtle detail is that the necessary ESProducts (geometry, magnetic field, particle data table) need to be read from EventSetup by the CMSSW thread, and passed by variables shared by CMSSW and master threads to the master thread. Reading from EventSetup in non-CMSSW thread is not supposed to work (will throw an exception).


  • Global Geant4 initialization, run in the master thread (OscarMTMasterThread)
  • Roughly corresponds Geant4's G4MTRunManager


  • Runs the event simulation, run in CMSSW/TBB threads ("worker threads")
  • Roughly corresponds Geant4's G4WorkerRunManager

Geant4 global initialization

The following describes how we do the "global" initialization of Geant4 (i.e. what is done in the master thread).

  1. The master thread is spawned from OscarMTProducer::initializeGlobalCache() by constructing OscarMTMasterThread
  2. At this point, only the (if it is not there yet; see here and here, and the master thread goes to sleep
  3. In the beginning of a run, the master thread is awaken (by a condition variable signal from a CMSSW thread running OscarMTProducer::globalBeginRun())
  4. The Geant4 initialization is run in the master thread (see here and here CMSSW thread is sleeping)
  5. After completing the initialization, master thread signals CMSSW thread and goes to back sleep
  6. All events of a run are processed
  7. At the end of a run, the master thread is again awaken (from OscarMTProducer::globalEndRun())
  8. Geant4 is destructed in the master thread (see here and here; CMSSW thread is sleeping)
  9. After destruction, master thread signals CMSSW thread and goes back to sleep
  10. If there are more runs to process, continue from 3 (in practice we typically process only 1 run per job)
  11. After everything is processed, the master thread is awaken and joined (from OscarMTProducer::globalEndJob())
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2019-05-10 - MattiKortelainen
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback