Difference: RawEventJuggler (1 vs. 13)

Revision 132014-03-31 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, moving/modifying the contents of the raw event.

Line: 267 to 267
  %ENDSYNTAX%
Added:
>
>

- Move around events into "Pit" format for Brunel/Moore/L0App/Swimming

Same as the above answer for moving into Stripping20 format, but swap the Input and Output. Input = "Stripping20", Output =0.0 .

 

- Custom move one bank somewhere

Starting from the most basic script:

Revision 122014-03-24 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, moving/modifying the contents of the raw event.

Line: 61 to 61
 
  • RawEventFormat: SVN a DBASE package which holds a very simple dictionary of which RawBank can be found in which RawEvent in which processing version. Since it is a simple python dictionary it has very few requirements and can be used at all levels of our software. Since it is stored as a DBASE package, adding new raw banks or new divisions of software needs to be carefully considered in terms of backwards compatibility. The latest version will be picked up by all software old and new, so additional entries are encouraged, modifying existing entries is discouraged..
  • RawEventCompat: SVN is a package living in LHCb which finds and uses the RawEventFormat dictionaries. In here live several different configurables and functions useful for different things.
Added:
>
>

Related documentation:

 

Python modules and help

Aside from the actual decoders themselves, the framework lives entirely in python. This implies that:

Revision 112014-03-21 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, moving/modifying the contents of the raw event.

Line: 179 to 179
  %SYNTAX{ syntax="python"}% from Configurables import DecodeRawEvent, RawEventFormatConf, RawEventJuggler, RecombineRawEvent
Changed:
<
<
ApplicationMgr().TopAlg+=GaudiSequencer(“Spam”)
>
>
ApplicationMgr().TopAlg+=[GaudiSequencer(“Spam”)]
 %ENDSYNTAX%

- Moving around banks so that multiple TCKs can be stored.

Revision 102014-02-26 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, moving/modifying the contents of the raw event.

Line: 327 to 327
  There are several steps to take to add a new raw bank.
  1. Add the bank type to the enumerated list of possible banks and to the list of how to translate this into a string RawBank.h
Changed:
<
<
  1. Create the class which actually is the bak itself, ideally using GOD XML
>
>
  1. Create the class which actually is the bank itself, ideally using GOD XML
 
  1. Create the encoder/decoder for this bank, which should be similar to an existing decoder, ideally, with configurable input and output locations. Ideally the input locations should be a list of possibilities.
  2. Create a new versioned RawEventFormat entry which adds your new bank into the RawEventFormat database. Do not edit existing entries, if this bank is from a new detector, you will need to at least create two new versions, one which describes where this bank is when it comes out of the pit, the input to L0App, Tesla, Moore, and Brunel, the second which describes where it should be put after Brunel, into which of the possible RawEvents.
  3. Add the configurations for your decoder into the DecoderDB, copy an existing entry. It should be rather straight forward given all the examples in the package.

Revision 92014-02-18 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, moving/modifying the contents of the raw event.

Line: 140 to 140
  , "KillInputBanksAfter" : None #Regex of banks to kill AFTER copying around, from input locations , "KillExtraBanks" : False #Bool, whether or not to remove banks which don't exist in target format , "KillExtraNodes" : False #Bool, whether to remove nodes which don't exist in target format
Changed:
<
<
, "KillExtraDirectories" : False #Kill not only the RawEvent, but also unlink the parent directories . Requires KillExtraNodes.
>
>
, "KillExtraDirectories" : False #Kill not only the RawEvent, but also unlink the parent directories. Requires KillExtraNodes.
  , "WriterOptItemList" : None #append a writer here to add output locations as OptItemList , "WriterItemList" : None #append a writer here to add output locations as ItemList , "Sequencer" : None #send in a sequencer to add the juggling into

Revision 82014-02-18 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, moving/modifying the contents of the raw event.

Line: 333 to 333
 
  1. Create a new versioned RawEventFormat entry which adds your new bank into the RawEventFormat database. Do not edit existing entries, if this bank is from a new detector, you will need to at least create two new versions, one which describes where this bank is when it comes out of the pit, the input to L0App, Tesla, Moore, and Brunel, the second which describes where it should be put after Brunel, into which of the possible RawEvents.
  2. Add the configurations for your decoder into the DecoderDB, copy an existing entry. It should be rather straight forward given all the examples in the package.
Added:
>
>

- Example of workflow in stripping20 and juggling that would work for swimming the same stripping:

Stripping workflow

Moore swimming workflow

DaVinci swimming workflow

 -- RobLambert - 21 Jan 2014

META FILEATTACHMENT attachment="Juggler1.png" attr="" comment="Object diagram for the juggler" date="1390397728" name="Juggler1.png" path="Juggler1.png" size="155569" user="rlambert" version="1"
META FILEATTACHMENT attachment="DAQ1.png" attr="" comment="OD for DAQSys" date="1390398031" name="DAQ1.png" path="DAQ1.png" size="166792" user="rlambert" version="1"
Added:
>
>
META FILEATTACHMENT attachment="Swim1.png" attr="" comment="Stripping20 raw event locations" date="1392721626" name="Swim1.png" path="Swim1.png" size="93774" user="rlambert" version="1"
META FILEATTACHMENT attachment="Swim2.png" attr="" comment="Swimming Moore for S20" date="1392721649" name="Swim2.png" path="Swim2.png" size="140292" user="rlambert" version="1"
META FILEATTACHMENT attachment="Swim3.png" attr="" comment="Swimming DaVinci" date="1392721674" name="Swim3.png" path="Swim3.png" size="145288" user="rlambert" version="1"

Revision 72014-02-05 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, moving/modifying the contents of the raw event.

Line: 96 to 96
 
  • Running two versions of the same decoder to/from different locations for your personal use
  • Splitting up the raw event differently from one of the "standard" versions for your personal use
Added:
>
>

How to interpret RawEventFormat

  • RawEventFormat: SVN a DBASE package which holds a very simple dictionary of which RawBank can be found in which RawEvent in which processing version. Since it is a simple python dictionary it has very few requirements and can be used at all levels of our software. Since it is stored as a DBASE package, adding new raw banks or new divisions of software needs to be carefully considered in terms of backwards compatibility. The latest version will be picked up by all software old and new, so additional entries are encouraged, modifying existing entries is discouraged..

  • The __init__.py module already holds the dictionaries, so that you can from RawEventFormat import ...
  • Raw_format_db: a dictionary of version : configuration
  • version : an arbitrary number designation for a certain placement of raw banks used somewhere in the past
  • configuration : a dictionary of RawBank : [ possible locations ]
  • [ possible locations ] : an ordered list of the most likely place to find the raw bank if it exists.
  • Which RawBanks?: There is a very large number of RawBanks Doxygen, a lot of which are obsolete but kept in the RawBank.h definition in order to interpret arbitrary files. Those which are obsolete will not appear on every file, and some banks are created at different stages of processing, so a raw file straight from the pit will not necessarily contain every bank. However, this dictionary stores where it will be if it exists, for every defined raw bank. All raw banks known at the time of the processing of these data appear in this list. Even if they do not appear on the file you are looking at, at least this should tell you where they logically should be put if they do exist on this file.
  • Reco_dict_db: a dictionary of logical name string : version. This dictionary contains logical names associated to different versions, for example it allows us to book-keep version 2.0 as "the version used for stripping 20". This is a 1:1 mapping and should not be changed later, since older versions may pick up the changes you make. It can be added to, though, e.g. you could add "Brunel after v44r8" if that had a different format, or "Reco 99" if that has a different format. Since this is a 1:1 mapping, the juggling tools accept either a float corresponding to a version number, or a string corresponding to a logical name.
  • logical name string: logical names associated to different versions, for example it allows us to book-keep version 2.0 as "the version used for stripping 20". Juggling tools accept either a float corresponding to a version number, or a string corresponding to a logical name.
 

RawEventJuggler user guide

object diagram for the juggler

Revision 62014-01-23 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"
Changed:
<
<

RawEventJuggler, how to move around the raw event, how to modify the contents of the raw event.

>
>

RawEventJuggler, moving/modifying the contents of the raw event.

  object diagram for DAQSys
Line: 35 to 35
 
  • Missing pieces: Since we need to minimise the footprint of data on the grid, and minimise the bandwidth requirement for user analysis, certain processings re-arrange and drop pieces of the raw event.
  • Moved pieces: Since opening the RawEvent requires parsing the extent of every bank, for use cases where we don't need everything we can gain CPU by breaking the raw event up into pieces in an earlier step. We can store multiple RawEvents, with different potentially overlapping RawBanks elsewhere in the TES.
Changed:
<
<
  • Backwards incompatibility: moving around pieces of the RawEvent is an inherantly backwards-incompatible change, what we needed was a tool which could ensure backwards compatibility.
>
>
  • Backwards incompatibility: moving around pieces of the RawEvent is an inherently backwards-incompatible change, what we needed was a tool which could ensure backwards compatibility.
 
  • Accountability: Before the creation of the framework described here (LHCb < v36r3) we had no way to book-keep the possible location for different raw banks/raw events aside from in the versions of the software used to split it up.

Talks and other information

Line: 57 to 57
 
  • Decoders: Each RawBank nominally has a corresponding decoder which needs to know where the raw bank is supposed to be found. These are each stored in their own package living in the LHCb project, for example see HLTDAQ (SVN)
  • DAQSys: DAQSys (SVN) is the package which holds all the pieces necessary to steer the decoders and configure the decoders themselves. The two parts which you need to know about are:
    • DecoderDB: SVN a python module which makes a very simple python dictionary of how to configure certain decoders if they are needed. It translates the many different methods of decoding into one simple interface to simplify configuration.
Changed:
<
<
    • DecodeRawEvent(): SVN a Configurable which can control the setup of the decoders from the database.Decoding used to be done
>
>
    • DecodeRawEvent(): SVN a Configurable which can control the setup of the decoders from the database. Decoding used to be managed by one options file which configured the DataOnDemand service, but that made it very much static and not very configurable downstream. We replaced this with a configurable which manages the decoders and separated the management for the definition of properties to eset etc, which are put into the DecoderDB.
 
  • RawEventFormat: SVN a DBASE package which holds a very simple dictionary of which RawBank can be found in which RawEvent in which processing version. Since it is a simple python dictionary it has very few requirements and can be used at all levels of our software. Since it is stored as a DBASE package, adding new raw banks or new divisions of software needs to be carefully considered in terms of backwards compatibility. The latest version will be picked up by all software old and new, so additional entries are encouraged, modifying existing entries is discouraged..
  • RawEventCompat: SVN is a package living in LHCb which finds and uses the RawEventFormat dictionaries. In here live several different configurables and functions useful for different things.
Line: 149 to 149
 
Changed:
<
<
>
>
  • HltLine: SVN (HltDecodeRaw.py)
 

- The most basic LHCb script

Line: 304 to 304
 RawEventJuggler().KillExtraNodes=not doIExpectYouToTalk

def noMrBondIExpectYouToDie():

Changed:
<
<
ApplicationMgr().TopAlgs=[slowmovingdeathtrap]+ApplicationMgr().TopAlgs
>
>
ApplicationMgr().TopAlg=[slowmovingdeathtrap]+ApplicationMgr().TopAlg
  if not doIExpectYouToTalk: appendPostConfigAction(noMrBondIExpectYouToDie)

Revision 52014-01-23 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, how to move around the raw event, how to modify the contents of the raw event.

Line: 11 to 11
 

What's the RawEvent?

Changed:
<
<
The RawEvent is a mini event store, which contains heavily packed, optimised data as close as possible to what is actually shipped off of the detector.

These packed data are known as RawBanks for which there may be several per subdetector, and many different types.

These data are concatenated and compressed into a "RawEvent" which is a concatenated set of "RawBanks". A raw event contains typically many different banks. In order to interpret a single bank from a raw event it is necessary to first parse the raw event which determines which banks are present and which sections of the raw event are assigned to which bank.

For further details on raw formats see:

>
>
See https://twiki.cern.ch/twiki/bin/view/LHCb/RawEvent#What_is_the_RawEvent
 

Where is it from, and how do we use it?

Changed:
<
<
  • Real data: the raw event is the concatenated set of banks that came from the real detector in some order or other. They are combined in the Online DAQ system by the "Event Builder" node into a RawEvent, which is then shipped around through the trigger etc. and eventually written to an MDF/Raw file.
  • Simulation: each sub-detector has developed code to simulate the raw data banks produced by their front end electronics, and to decode the raw banks in both real and simulated data. The same raw data structure is simulated by the Boole digitization application which produces a very similar "RawEvent" object containing a collection of banks with the same format.
  • Interpreting = Decoding: RawBanks are interpreted from within the raw event using a "Decoder" for which there is nominally one per bank type, however, each subdetector uses completely different design principles to write the decoder. Most raw banks are translated into a set of Hits or other objects which live in a different location on the TES, and this translation is usally done by a dedicated decoding algorithm within a specific sequence.
>
>
See https://twiki.cern.ch/twiki/bin/view/LHCb/RawEvent#Where_is_it_from_and_how_do_we_u
 

Where can I find the RawEvent?

Revision 42014-01-22 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, how to move around the raw event, how to modify the contents of the raw event.

Added:
>
>
object diagram for DAQSys
 This page describes what to do to move around the raw event, what frameworks we have in place to manage that.
Changed:
<
<

! Contents

>
>

Contents

 

What's the RawEvent?

Changed:
<
<
The RawEvent is a mini event store, which contains heavily packed, optimised data as close as possible to what is actually shipped off of the detector.
>
>
The RawEvent is a mini event store, which contains heavily packed, optimised data as close as possible to what is actually shipped off of the detector.

These packed data are known as RawBanks for which there may be several per subdetector, and many different types.

These data are concatenated and compressed into a "RawEvent" which is a concatenated set of "RawBanks". A raw event contains typically many different banks. In order to interpret a single bank from a raw event it is necessary to first parse the raw event which determines which banks are present and which sections of the raw event are assigned to which bank.

 
Changed:
<
<
These packed data are known as RawBanks for which there may be several per subdetector, and many different types.
>
>
For further details on raw formats see:
 
Changed:
<
<
These data are concatenated and compressed, and a raw event contains typically many different banks. In order to interpret a single bank from a raw event it is necessary to first parse the raw event which determines which banks are present and which sections of the raw event are assigned to which bank.
>
>

Where is it from, and how do we use it?

 
Changed:
<
<

Where is it?

>
>
  • Real data: the raw event is the concatenated set of banks that came from the real detector in some order or other. They are combined in the Online DAQ system by the "Event Builder" node into a RawEvent, which is then shipped around through the trigger etc. and eventually written to an MDF/Raw file.
  • Simulation: each sub-detector has developed code to simulate the raw data banks produced by their front end electronics, and to decode the raw banks in both real and simulated data. The same raw data structure is simulated by the Boole digitization application which produces a very similar "RawEvent" object containing a collection of banks with the same format.
  • Interpreting = Decoding: RawBanks are interpreted from within the raw event using a "Decoder" for which there is nominally one per bank type, however, each subdetector uses completely different design principles to write the decoder. Most raw banks are translated into a set of Hits or other objects which live in a different location on the TES, and this translation is usally done by a dedicated decoding algorithm within a specific sequence.
 
Changed:
<
<
An MDF file, ("RAW" file) is natively a very similar data format to the raw event, it essentially stores back-to-back raw events. Gaudi when running over these events needs to place the data into a Transient Event store, which is a map of a directory string --> DataObject, and in this case the whole RawEvent is one DataObject.
>
>

Where can I find the RawEvent?

 
Changed:
<
<
In the standard form, the raw event appears at DAQ/RawEvent. The object stored here is a RawEvent, which contains within it potentially many different RawBanks.
>
>
An MDF file, ("RAW" file) is natively a very similar data format to the raw event, it essentially stores back-to-back raw events. Gaudi when running over these events needs to place the data into a Transient Event store. Elements in the store are accessed by a string which is sort of like a simple directory path, and returs some sort of DataObject. In this case the whole RawEvent is one DataObject.
 
Changed:
<
<
However, when writing out to a ROOT file (DST) we are free to move/re-arrange this, and store the RawEvent wherever we want to. Potentially we can store several different raw event objects containing different banks. Typically we do split up the original raw event.
>
>
In the standard form, the raw event appears at DAQ/RawEvent. The object stored here is a RawEvent, which contains within it potentially many different RawBanks.

However, when writing out to a ROOT file (DST) we are free to move/re-arrange this, and store the RawEvent wherever we want to. Potentially we can store several different raw event objects containing different banks. Typically we do split up the original raw event. For example, in stripping 20 we had:

  • Deleted DAQ/RawEvent
  • Moved all RICH banks into Rich/RawEvent
  • Moved all Trigger banks, plus ODIN into Trigger/RawEvent
  • Moved all Muon banks into Muon/RawEvent
  • Moved any other existing banks into Other/RawEvent

Since this was done in Brunel deep in the configurable, there was no simple place to really be able to tell which bank sits where, and the Decoders for those banks then have no way of knowing where to look for them in a generic way. Enter the DecoderDB and RawEventFormat.

 

What's the issue?

  • Missing pieces: Since we need to minimise the footprint of data on the grid, and minimise the bandwidth requirement for user analysis, certain processings re-arrange and drop pieces of the raw event.
Changed:
<
<
  • Moved pieces: Since opening the RawEvent requires parsing the extent of every bank, for use cases where we don't need everything we can gain CPU by breaking the raw event up into pieces in an earlier step. We can store multiple RawEvents, with different potentially overlapping RawBanks elsewhere in the TES.
  • Backwards incompatibility: moving around pieces of the RawEvent is an inherantly backwards-incompatible change, what we needed was a tool which could ensure backwards compatibility.
>
>
  • Moved pieces: Since opening the RawEvent requires parsing the extent of every bank, for use cases where we don't need everything we can gain CPU by breaking the raw event up into pieces in an earlier step. We can store multiple RawEvents, with different potentially overlapping RawBanks elsewhere in the TES.
  • Backwards incompatibility: moving around pieces of the RawEvent is an inherantly backwards-incompatible change, what we needed was a tool which could ensure backwards compatibility.
 
  • Accountability: Before the creation of the framework described here (LHCb < v36r3) we had no way to book-keep the possible location for different raw banks/raw events aside from in the versions of the software used to split it up.

Talks and other information

Line: 47 to 70
 

Component Building blocks

This framework comprises several LHCb packages which are inter-dependent.

Changed:
<
<
  • Decoders: Each RawBank nominally has a corresponding decoder which needs to know where the raw bank is supposed to be found. These are each stored in their own package living in the LHCb project, for example see HLTDAQ (SVN)
>
>
  • Decoders: Each RawBank nominally has a corresponding decoder which needs to know where the raw bank is supposed to be found. These are each stored in their own package living in the LHCb project, for example see HLTDAQ (SVN)
 
  • DAQSys: DAQSys (SVN) is the package which holds all the pieces necessary to steer the decoders and configure the decoders themselves. The two parts which you need to know about are:
    • DecoderDB: SVN a python module which makes a very simple python dictionary of how to configure certain decoders if they are needed. It translates the many different methods of decoding into one simple interface to simplify configuration.
    • DecodeRawEvent(): SVN a Configurable which can control the setup of the decoders from the database.Decoding used to be done
Changed:
<
<
  • RawEventFormat: SVN a DBASE package which holds a very simple dictionary of which RawBank can be found in which RawEvent in which processing version. Since it is a simple python dictionary it has very few requirements and can be used at all levels of our software. Since it is stored as a DBASE package, adding new raw banks or new divisions of software needs to be carefully considered in terms of backwards compatibility. The latest version will be picked up by all software old and new, so additional entries are encouraged, modifying existing entries is discouraged..
  • RawEventCompat: SVN is a package living in LHCb which finds and uses the RawEventFormat dictionaries. In here live several different configurables and functions useful for different things.
>
>
  • RawEventFormat: SVN a DBASE package which holds a very simple dictionary of which RawBank can be found in which RawEvent in which processing version. Since it is a simple python dictionary it has very few requirements and can be used at all levels of our software. Since it is stored as a DBASE package, adding new raw banks or new divisions of software needs to be carefully considered in terms of backwards compatibility. The latest version will be picked up by all software old and new, so additional entries are encouraged, modifying existing entries is discouraged..
  • RawEventCompat: SVN is a package living in LHCb which finds and uses the RawEventFormat dictionaries. In here live several different configurables and functions useful for different things.
 

Python modules and help

Line: 89 to 112
 
  • Running two versions of the same decoder to/from different locations for your personal use
  • Splitting up the raw event differently from one of the "standard" versions for your personal use
Added:
>
>

RawEventJuggler user guide

object diagram for the juggler

The RawEventJuggler is a configurable which is smart enough to be able to switch between any two RawEvent formats. It can re-arrange banks into different places, split up and recombine banks, remove banks, remove locations, all sorts of things. Since it is fully integrated into the RawEventFormat system it is quite simple to move around banks and write something out afterwards. The best source of documentation is the code itself:

Common options include:

<!-- SyntaxHighlightingPlugin -->
class RawEventJuggler(ConfigurableUser):
    """
    A complex configurable ...
    Juggle around raw event locations, will split up and/or combine anything from anywhere to anywhere else
using maps.

    Aimed for use during the juggling currently performed in Brunel, and for future use in MC between differ
ent Moore versions

    Can configure writers, sequencers and/or DoD

    """
    __queried_configurables__ = [RawEventFormatConf]

    __slots__ = {
        "Input" : None  #Raw event location to start with. E.g. 2.0
        , "Output" : None #Raw Event location to go to. E.g. 0.0
        , "KillInputBanksBefore" : None #Regex of banks to kill BEFORE copying around, from input locations
        , "KillInputBanksAfter" : None #Regex of banks to kill AFTER copying around, from input locations
        , "KillExtraBanks" : False #Bool, whether or not to remove banks which don't exist in target format
        , "KillExtraNodes" : False #Bool, whether to remove nodes which don't exist in target format
        , "KillExtraDirectories" : False #Kill not only the RawEvent, but also unlink the parent directories
. Requires KillExtraNodes.
        , "WriterOptItemList" : None #append a writer here to add output locations as OptItemList
        , "WriterItemList" : None #append a writer here to add output locations as ItemList
        , "Sequencer" : None #send in a sequencer to add the juggling into
        , "DataOnDemand" : False #Add juggling into DoD?
        , "Depth" : "#1" #what to add to the end of the writer, depth of 1 is fine for RawEvents
        , "TCK" : None #which TCK to use to replace the TCKReplacePattern if encountered
        , "TCKReplacePattern" : "#TCK#"
        , "RootInTES" : "/Event" #I don't think you'll need to set this, but just in case, it's there too
        , "GenericReplacePatterns" : {} #List of additional patterns in locations to replace with other ones, performs location.replace(key,value) on every location
        }
<!-- end SyntaxHighlightingPlugin -->
 

Examples of using/editing this framework

There are many examples in the PPTS talk: 18th November 2013. Here we will cover a few of them.

- Examples in existing code:

Changed:
<
<
  • DstConf: SVN
  • Brunel: SVN
  • Moore:
  • HltLine:
>
>
 

- The most basic LHCb script

Line: 121 to 188
 

- Moving around banks so that multiple TCKs can be stored.

Starting from the most basic script:

Changed:
<
<
  • create %SYNTAX{ syntax="python"}%
>
>
  • create a file where the trigger banks have been copied to a TCK-dependent location %SYNTAX{ syntax="python"}%
 Writer=InputCopyStream(“MyStream”) IOHelper().outStream(“my_copy_file.dst”,writer=Writer)
Line: 134 to 201
 RawEventJuggler().WriterOptItemList=Writer

%ENDSYNTAX%

Changed:
<
<
  • use %SYNTAX{ syntax="python"}%
>
>
  • use that file in a later Gaudi job %SYNTAX{ syntax="python"}%
 # # Either #
Line: 156 to 223
  %ENDSYNTAX%
Changed:
<
<

- Move around events so that Moore or L0App can be re-run

>
>

- Move around events into "Stripping20" format

  Starting from the most basic script:
Changed:
<
<
  • create %SYNTAX{ syntax="python"}%
>
>
  • create a file where the banks have been split up RawEvent living in *DAQ/RawEvent*%SYNTAX{ syntax="python"}%
 Writer=InputCopyStream(“MyStream”) IOHelper().outStream(“my_copy_file.dst”,writer=Writer)
Line: 171 to 238
 RawEventJuggler().WriterOptItemList=Writer

%ENDSYNTAX%

Changed:
<
<
  • use %SYNTAX{ syntax="python"}%
>
>
  • use this file in another later Gaudi Job, e.g. Moore%SYNTAX{ syntax="python"}%
 # # Either #
Line: 239 to 306
  %ENDSYNTAX%
Changed:
<
<

- Custom move one bank somewhere

>
>

- Move and kill banks/locations, insert a sequence as a post-config action

 
Changed:
<
<
Starting from the most basic script:
  • create
    <!-- SyntaxHighlightingPlugin -->

<!-- end SyntaxHighlightingPlugin -->
  • use %SYNTAX{ syntax="python"}%
# # Either #
>
>
Sometimes you will need to move banks around even though there are other sequencers around. If you intend to kill some banks or locations, you can't use DataOnDemand, so, what can you do? You'll need to run the Juggler as a sequence, but insert it first in the list of running algs as a sequence. This might not entirely work either if somethign is trying to access event data before your sequence is run or if someone else has also used a post-config action. So it might be best to re-arrange and write out a juggled file in a pre-processor step.
 
Added:
>
>
Starting from the most basic script: %SYNTAX{ syntax="python"}% doIExpectYouToTalk=False slowmovingdeathtrap=GaudiSequencer("JuggleSeq")
 
Added:
>
>
RawEventJuggler().Sequencer=slowmovingdeathtrap RawEventJuggler().DataOnDemand=doIExpectYouToTalk RawEventJuggler().KillExtraNodes=not doIExpectYouToTalk
 
Changed:
<
<
# # Or #
>
>
def noMrBondIExpectYouToDie(): ApplicationMgr().TopAlgs=[slowmovingdeathtrap]+ApplicationMgr().TopAlgs
 
Added:
>
>
if not doIExpectYouToTalk: appendPostConfigAction(noMrBondIExpectYouToDie)
 %ENDSYNTAX%

- Permanently adding a new raw bank/decoder into the system

There are several steps to take to add a new raw bank.

Changed:
<
<
  1. Add the bank type to the enumerated list and to the list of how to translate this into a string Raw
  2. Create the bank ideally using GOD XML
  3. Create the encoder/decoder for this bank, which should be similar to an existing decoder, ideally.
  4. Create a new versioned RawEventFormat entry which adds your new bank into the RawEventFormat database. Do not edit existing entries, if this bank is from a new detector, you will need to at least create two new versions, one which describes where this bank is when it comes out of the pit, the input to L0App, Tesla, Moore, and Brunel, the second which describes where it should be put after Brunel, into which of the possible RawEvents.
  5. Add the configurations for your decoder into the DecoderDB, copy an existing entry. It should be rather straight forward given all the examples in the package.
>
>
  1. Add the bank type to the enumerated list of possible banks and to the list of how to translate this into a string RawBank.h
  2. Create the class which actually is the bak itself, ideally using GOD XML
  3. Create the encoder/decoder for this bank, which should be similar to an existing decoder, ideally, with configurable input and output locations. Ideally the input locations should be a list of possibilities.
  4. Create a new versioned RawEventFormat entry which adds your new bank into the RawEventFormat database. Do not edit existing entries, if this bank is from a new detector, you will need to at least create two new versions, one which describes where this bank is when it comes out of the pit, the input to L0App, Tesla, Moore, and Brunel, the second which describes where it should be put after Brunel, into which of the possible RawEvents.
  5. Add the configurations for your decoder into the DecoderDB, copy an existing entry. It should be rather straight forward given all the examples in the package.
  -- RobLambert - 21 Jan 2014
Added:
>
>
META FILEATTACHMENT attachment="Juggler1.png" attr="" comment="Object diagram for the juggler" date="1390397728" name="Juggler1.png" path="Juggler1.png" size="155569" user="rlambert" version="1"
META FILEATTACHMENT attachment="DAQ1.png" attr="" comment="OD for DAQSys" date="1390398031" name="DAQ1.png" path="DAQ1.png" size="166792" user="rlambert" version="1"

Revision 32014-01-21 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, how to move around the raw event, how to modify the contents of the raw event.

Line: 89 to 89
 
  • Running two versions of the same decoder to/from different locations for your personal use
  • Splitting up the raw event differently from one of the "standard" versions for your personal use
Added:
>
>

Examples of using/editing this framework

 
Added:
>
>
There are many examples in the PPTS talk: 18th November 2013. Here we will cover a few of them.
 
Changed:
<
<

Examples of using/editing this framework

>
>

- Examples in existing code:

  • DstConf: SVN
  • Brunel: SVN
  • Moore:
  • HltLine:

- The most basic LHCb script

<!-- SyntaxHighlightingPlugin -->
#The simplest possible LHCb script
from Gaudi.Configuration import *
from Configurables import LHCbApp
LHCbApp()
from GaudiConf import IOHelper
IOHelper().inputFiles([“some_file.dst”])
<!-- end SyntaxHighlightingPlugin -->

Let's demonstrate what you need to do to do some juggling with:

<!-- SyntaxHighlightingPlugin -->
from Configurables import DecodeRawEvent, RawEventFormatConf, RawEventJuggler, RecombineRawEvent
ApplicationMgr().TopAlg+=GaudiSequencer(“Spam”)
<!-- end SyntaxHighlightingPlugin -->

- Moving around banks so that multiple TCKs can be stored.

Starting from the most basic script:

  • create
    <!-- SyntaxHighlightingPlugin -->
Writer=InputCopyStream(“MyStream”)
IOHelper().outStream(“my_copy_file.dst”,writer=Writer)

RawEventJuggler().Input=“Pit”
RawEventJuggler().Output=3.0
RawEventJuggler().TCK=“0x990099”
RawEventJuggler().Sequencer=GaudiSequencer(“Spam”)
RawEventJuggler().KillExtraBanks=True
RawEventJuggler().KillExtraNodes=True
RawEventJuggler().WriterOptItemList=Writer
<!-- end SyntaxHighlightingPlugin -->
  • use
    <!-- SyntaxHighlightingPlugin -->
#
# Either
#

#Recreate original locations on demand
RawEventJuggler().Input=3.0
RawEventJuggler().Output=“Stripping20”
RawEventJuggler().TCK=“0x990099”
RawEventJuggler().Sequencer                  =GaudiSequencer(“Spam”)


#
# Or
#

#just use the new version
RawEventFormatConf().TCK=“0x990099”
DecodeRawEvent().OverrideInputs=3.0
<!-- end SyntaxHighlightingPlugin -->
 
Added:
>
>

- Move around events so that Moore or L0App can be re-run

Starting from the most basic script:

  • create
    <!-- SyntaxHighlightingPlugin -->
Writer=InputCopyStream(“MyStream”)
IOHelper().outStream(“my_copy_file.dst”,writer=Writer)

RawEventJuggler().Input=“Pit”
RawEventJuggler().Output=“Stripping20”
RawEventJuggler().Sequencer=GaudiSequencer(“Spam”)
RawEventJuggler().KillExtraBanks=True
RawEventJuggler().KillExtraNodes=True
RawEventJuggler().WriterOptItemList=Writer
<!-- end SyntaxHighlightingPlugin -->
  • use
    <!-- SyntaxHighlightingPlugin -->
#
# Either
#

#recreate the original locations
RawEventJuggler().Input=“Stripping20”
RawEventJuggler().Output=“Moore”
RawEventJuggler().Sequencer                  =GaudiSequencer(“Spam”)



#
# Or
#

#just use the new locations
DecodeRawEvent().OverrideInputs       = “Stripping20”

#
# Or
#

#recreate DAQ/RawEvent
RecombineRawEvent()
<!-- end SyntaxHighlightingPlugin -->
 
Added:
>
>

- Custom move one bank somewhere

Starting from the most basic script:

  • define a new "custom" location
    <!-- SyntaxHighlightingPlugin -->
RawEventFormatConf().forceLoad()
myLocs=RawEventFormatConf().Locations
myLocs[“Custom”]=myLocs[2.0]
myLocs[“Custom”][“TT”]=“My/Custom/Location”
<!-- end SyntaxHighlightingPlugin -->
  • move to this location
    <!-- SyntaxHighlightingPlugin -->
RawEventJuggler().Input=2.0
RawEventJuggler().Output=“Custom”
RawEventJuggler().Sequencer=GaudiSequencer(“Spam”)
RawEventJuggler().KillExtraBanks=True
RawEventJuggler().KillExtraNodes=True
RawEventJuggler().WriterOptItemList=Writer
<!-- end SyntaxHighlightingPlugin -->

- Customize a decoder

  • edit a decoder property
    <!-- SyntaxHighlightingPlugin -->
from DAQSys.Decoders import DecoderDB as ddb

ddb["DecodeVeloRawBuffer/createVeloClusters"].Properties[“OutputLevel”]=DEBUG
<!-- end SyntaxHighlightingPlugin -->

- Run only one decoder

  • add one deoder to a specific sequence
    <!-- SyntaxHighlightingPlugin -->
from DAQSys.Decoders import DecoderDB as ddb
from DAQSys.DecoderClass import decodersForBank

mySeq.Members+=[d.setup() for d in decodersForBank(ddb,“Velo”)]
<!-- end SyntaxHighlightingPlugin -->

- Custom move one bank somewhere

Starting from the most basic script:

  • create
    <!-- SyntaxHighlightingPlugin -->

<!-- end SyntaxHighlightingPlugin -->
  • use
    <!-- SyntaxHighlightingPlugin -->
#
# Either
#



#
# Or
#
<!-- end SyntaxHighlightingPlugin -->
 
Changed:
<
<

- Permanently adding a new raw bank/decoder into the system

>
>

- Permanently adding a new raw bank/decoder into the system

  There are several steps to take to add a new raw bank.
  1. Add the bank type to the enumerated list and to the list of how to translate this into a string Raw

Revision 22014-01-21 - RobLambert

Line: 1 to 1
 
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, how to move around the raw event, how to modify the contents of the raw event.

Line: 39 to 39
  please see the PPTS talk, which covers examples of use
Changed:
<
<
There are also many different tracked tasks which overlap with this effort:
>
>
There are also many different tracked tasks which were identified several years ago, and were delayed until LS1, which correspond to this effort:
 

Component Building blocks

Changed:
<
<
  • Decoders: Each RawBank nominally has a corresponding decoder which needs to know where the raw bank is supposed to be found. These are each stored in their own package living in teh LHCb project, for example see HLTDAQ (SVN)
>
>
This framework comprises several LHCb packages which are inter-dependent.
  • Decoders: Each RawBank nominally has a corresponding decoder which needs to know where the raw bank is supposed to be found. These are each stored in their own package living in the LHCb project, for example see HLTDAQ (SVN)
 
  • DAQSys: DAQSys (SVN) is the package which holds all the pieces necessary to steer the decoders and configure the decoders themselves. The two parts which you need to know about are:
    • DecoderDB: SVN a python module which makes a very simple python dictionary of how to configure certain decoders if they are needed. It translates the many different methods of decoding into one simple interface to simplify configuration.
    • DecodeRawEvent(): SVN a Configurable which can control the setup of the decoders from the database.Decoding used to be done
  • RawEventFormat: SVN a DBASE package which holds a very simple dictionary of which RawBank can be found in which RawEvent in which processing version. Since it is a simple python dictionary it has very few requirements and can be used at all levels of our software. Since it is stored as a DBASE package, adding new raw banks or new divisions of software needs to be carefully considered in terms of backwards compatibility. The latest version will be picked up by all software old and new, so additional entries are encouraged, modifying existing entries is discouraged..
Changed:
<
<
  • RawEventCompat: SVN is a package living in LHCb which finds and uses the RawEventFormat dictionaries. In here lives several different configurables and functions useful for different things.
>
>
  • RawEventCompat: SVN is a package living in LHCb which finds and uses the RawEventFormat dictionaries. In here live several different configurables and functions useful for different things.

Python modules and help

Aside from the actual decoders themselves, the framework lives entirely in python. This implies that:

  • you can browse through the code, and get access to the inbuilt python help on modules.
  • As long as you know python, you'll be very able to comprehend and even edit the content of the packages

<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb --use RawEventFormat 
python 
<!-- end SyntaxHighlightingPlugin -->
<!-- SyntaxHighlightingPlugin -->
from DAQSys import DecoderClass, DecoderDB
help(DecoderClass)
help(DecoderDB)
help(DecoderClass.Decoder)
print DecoderDB.decoders #it's a dictionary!

from RawEventFormat import Raw_location_db
print Raw_location_db #it's a dictionary!
from RawEventCompat import Configuration as R_E_C
help(R_E_C)

from Configurables import RawEventFormatConf, RawEventJuggler, DecodeRawEvent
help(RawEventFormatConf)
help(RawEventJuggler)
help(DecodeRawEvent)
<!-- end SyntaxHighlightingPlugin -->

Feature Discussion

- Transience

The databases used in the configuration are all soft-coded. At any point in your own user code you can extend them or recreate them completely with different content. Examples include:

  • Temporarily changing the properties of a decoder for your specific use
  • Running two versions of the same decoder to/from different locations for your personal use
  • Splitting up the raw event differently from one of the "standard" versions for your personal use

Examples of using/editing this framework

- Permanently adding a new raw bank/decoder into the system

There are several steps to take to add a new raw bank.

  1. Add the bank type to the enumerated list and to the list of how to translate this into a string Raw
  2. Create the bank ideally using GOD XML
  3. Create the encoder/decoder for this bank, which should be similar to an existing decoder, ideally.
  4. Create a new versioned RawEventFormat entry which adds your new bank into the RawEventFormat database. Do not edit existing entries, if this bank is from a new detector, you will need to at least create two new versions, one which describes where this bank is when it comes out of the pit, the input to L0App, Tesla, Moore, and Brunel, the second which describes where it should be put after Brunel, into which of the possible RawEvents.
  5. Add the configurations for your decoder into the DecoderDB, copy an existing entry. It should be rather straight forward given all the examples in the package.
  -- RobLambert - 21 Jan 2014

Revision 12014-01-21 - RobLambert

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LHCbTriggerForUsers"

RawEventJuggler, how to move around the raw event, how to modify the contents of the raw event.

This page describes what to do to move around the raw event, what frameworks we have in place to manage that.

! Contents

What's the RawEvent?

The RawEvent is a mini event store, which contains heavily packed, optimised data as close as possible to what is actually shipped off of the detector.

These packed data are known as RawBanks for which there may be several per subdetector, and many different types.

These data are concatenated and compressed, and a raw event contains typically many different banks. In order to interpret a single bank from a raw event it is necessary to first parse the raw event which determines which banks are present and which sections of the raw event are assigned to which bank.

Where is it?

An MDF file, ("RAW" file) is natively a very similar data format to the raw event, it essentially stores back-to-back raw events. Gaudi when running over these events needs to place the data into a Transient Event store, which is a map of a directory string --> DataObject, and in this case the whole RawEvent is one DataObject.

In the standard form, the raw event appears at DAQ/RawEvent. The object stored here is a RawEvent, which contains within it potentially many different RawBanks.

However, when writing out to a ROOT file (DST) we are free to move/re-arrange this, and store the RawEvent wherever we want to. Potentially we can store several different raw event objects containing different banks. Typically we do split up the original raw event.

What's the issue?

  • Missing pieces: Since we need to minimise the footprint of data on the grid, and minimise the bandwidth requirement for user analysis, certain processings re-arrange and drop pieces of the raw event.
  • Moved pieces: Since opening the RawEvent requires parsing the extent of every bank, for use cases where we don't need everything we can gain CPU by breaking the raw event up into pieces in an earlier step. We can store multiple RawEvents, with different potentially overlapping RawBanks elsewhere in the TES.
  • Backwards incompatibility: moving around pieces of the RawEvent is an inherantly backwards-incompatible change, what we needed was a tool which could ensure backwards compatibility.
  • Accountability: Before the creation of the framework described here (LHCb < v36r3) we had no way to book-keep the possible location for different raw banks/raw events aside from in the versions of the software used to split it up.

Talks and other information

please see the PPTS talk, which covers examples of use

There are also many different tracked tasks which overlap with this effort:

Component Building blocks

  • Decoders: Each RawBank nominally has a corresponding decoder which needs to know where the raw bank is supposed to be found. These are each stored in their own package living in teh LHCb project, for example see HLTDAQ (SVN)
  • DAQSys: DAQSys (SVN) is the package which holds all the pieces necessary to steer the decoders and configure the decoders themselves. The two parts which you need to know about are:
    • DecoderDB: SVN a python module which makes a very simple python dictionary of how to configure certain decoders if they are needed. It translates the many different methods of decoding into one simple interface to simplify configuration.
    • DecodeRawEvent(): SVN a Configurable which can control the setup of the decoders from the database.Decoding used to be done
  • RawEventFormat: SVN a DBASE package which holds a very simple dictionary of which RawBank can be found in which RawEvent in which processing version. Since it is a simple python dictionary it has very few requirements and can be used at all levels of our software. Since it is stored as a DBASE package, adding new raw banks or new divisions of software needs to be carefully considered in terms of backwards compatibility. The latest version will be picked up by all software old and new, so additional entries are encouraged, modifying existing entries is discouraged..
  • RawEventCompat: SVN is a package living in LHCb which finds and uses the RawEventFormat dictionaries. In here lives several different configurables and functions useful for different things.

-- RobLambert - 21 Jan 2014

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback