Transitions Seen by Modules in the Threaded Framework

Complete: 3


New Transitions

In addition to the transitions used by the single threaded framework (a begin or end of File, Run, LuminosityBlock, a new Event or an EventSetup IOV change) the multi-threaded framework adds two additional concepts: Global and Stream.


Global transitions are ones that apply to the framework as a whole. For example, the global begin Run and begin LuminosityBlock transitions are only seen when the source first reads a new Run or LuminosityBlock. Similarly, the global end Run and end LuminosityBlock transitions are only seen once all processing of data (i.e. Events) for that Run or LuminosityBlock has finished. One important item to remember about global transitions is it is possible that multiple global transitions can be running concurrently. For example, two or more begin or end Runs can be running simultaneously. Also a new global begin transition can occur while a global end transition is still running. Therefore, code that deals with global transitions must be internally thread safe. The only transition that is not seen globally is an Event transition.


A Stream handles the processing of Events. A Stream enforces the processing of transitions based on the unscheduled execution model and guarantees that a module is called at most once per transition. That means EDFilters, EDAnalyzers and OutputModules are called in the order specified by the Paths or EndPaths to which they are associated while EDProducers are called the first time their data is requested by another module. Transitions are processed serially within a Stream. The ordering of transitions are as expected: begin Run, then begin LuminosityBlock followed by multiple Events, and ending with end LuminosityBlock and finally end Run. No modules in a stream will begin a new transition until all modules in that Stream have finished the previous transition. Each Stream in a job is independent and runs concurrently with all other Streams. Each Event is seen by only one Stream therefore a Stream only sees a subset of Events coming from the Source. Therefore the single threaded version of cmsRun can be thought of as only containing one Stream.

It is possible to have one instance of a module shared by all Streams. Such modules are referred to as Global modules (since they will see every Global and Stream transition sent by the framework). However, sometimes it is more convenient to have each Stream own its own copy of a module. Such modules are referred to as Stream modules.



Shown above are the relationships between the Global and Stream transitions. Dependencies between transitions which cause the system to wait are shown as lines. The three different colors correspond to the transitions related to three different LuminosityBlocks and the Events contained within those LuminosityBlocks. Time is represented along the horizontal axis. The length of each box represents the amount of time needed to processes a transition. In the diagram, Event 2 takes substantially longer than any other Event in the job.

The following are the minimal dependencies between transitions as seen by a module

  • Begin Stream transitions will not run until the begin Job transition has finished.
  • A Stream's begin Run will not run until the corresponding Global begin Run transition has ended.
  • A Stream's begin LuminosityBlock will not run until the corresponding Global begin LuminosityBlock transition has ended.
  • The Global end LuminosityBlock will not run until all the corresponding Stream end LuminosityBlock transitions have ended.
  • The Global end Run will not run until all the corresponding Stream end Runs have finished.
  • The Global end Job will not run until all end Stream transitions have finished.

Having all Stream begin transitions wait until the corresponding Global transition has finished guarantees that any work done in the Global transition will be available at the time of the Stream transition. Similarly, having all Global end transitions wait until all the corresponding Stream end transitions have finished guarantees that any summary information accumulated by all the Streams will be available to the Global end transition.

Items to note in the above diagram is a Global begin LuminosityBlock transition can happen before the Global end LuminosityBlock of a previous LuminosityBlock has been called. The same holds for a begin Run and an end Run for two different Runs. In contrast, Stream transitions always occur in the 'expected' order. Also, a Stream will only see a LuminosityBlock based transitions for LuminosityBlocks corresponding to Events that a Stream processes. If a Stream never processes a LuminosityBlock if it does not also process at least one Event for that LuminosityBlock. The only exception is the case where the Source delivers a LuminosityBlock that contains no Events. In that particular case, only one Stream will process that LuminosityBlock. Similarly, a Stream will only see Run transitions if that same Stream will also see LuminosityBlock contained in that Run. The exception once again is for a Run that contains no LuminosityBlocks (and therefore no Events) will be see by just one Stream.


There are cases where Global transitions must be serialized. For example, the present DQM design makes uses of a universally accessible DQMStore, which works as a shared memory access point. The DQMStore holds histograms for only one Run and one LuminosityBlock at a time and then reuses those same histograms if the system switches to a new Run or LuminosityBlock. Clearly this requires that the system enforce that only one Run or LuminosityBlock is globally active in a job that is running the DQM. Therefore if a Source, module or Service in the system explicitly states it can only handle on instance of a transition in the system at a time, they framework will enforce that rule.

We want to minimize such serialization since serialization puts an upper limit on the speed gains obtained from parallelization. In the case of the framework, such serialization is very bad when

  • There are large variations between the time it takes to process the various events. In such a case when we reach the end of a LuminosityBlock we can't proceed to the next LuminosityBlock because of the serialization requirement. Therefore all Streams are stuck waiting for the slowest Stream to finish processing its Event for the previous LuminosityBlock.
  • If there are very few events in a LuminosityBlock or Run and we are only allowed one LuminosityBlock or Run active in the job at a time.

Review status

Reviewer/Editor and Date Comments
Last reviewed by: ChrisDJones - 13-Nov-2012 Created the page

Responsible: ChrisDJones
Last reviewed by:

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Async_global_module_transitions.png r1 manage 57.1 K 2012-11-14 - 17:29 ChrisDJones Diagram showing Global and Stream transitions
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2012-11-26 - ChrisDJones
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback