RawEventJuggler < LHCb

---+!! RawEventJuggler, moving/modifying the contents of the raw event.

<img src="%ATTACHURLPATH%/DAQ1.png" alt="object diagram for DAQSys" width='600' />

This page describes what to do to move around the raw event, what frameworks we have in place to manage that.

---++!! *Contents*

%TOC%

---++ *What's the RawEvent?*
See [[https://twiki.cern.ch/twiki/bin/view/LHCb/RawEvent#What_is_the_RawEvent]]

---++ *Where is it from, and how do we use it?*
See [[https://twiki.cern.ch/twiki/bin/view/LHCb/RawEvent#Where_is_it_from_and_how_do_we_u]]

---++ *Where can I find the !RawEvent?*

An MDF file, ("RAW" file) is natively a very similar data format to the raw event, it essentially stores back-to-back raw events. Gaudi when running over these events needs to place the data into a Transient Event store. Elements in the store are accessed by a string which is sort of like a simple directory path, and returs some sort of !DataObject. In this case the whole !RawEvent is one !DataObject.

In the standard form, the raw event appears at *DAQ/RawEvent*. The object stored here is a !RawEvent, which contains within it potentially many different !RawBanks.

However, when writing out to a ROOT file (DST) we are free to move/re-arrange this, and store the !RawEvent wherever we want to. Potentially we can store several different raw event objects containing different banks. Typically we do split up the original raw event. For example, in stripping 20 we had:
   * Deleted DAQ/RawEvent
   * Moved all RICH banks into Rich/RawEvent
   * Moved all Trigger banks, plus ODIN into Trigger/RawEvent
   * Moved all Muon banks into Muon/RawEvent
   * Moved any other existing banks into Other/RawEvent

Since this was done in Brunel deep in the configurable, there was no simple place to really be able to tell which bank sits where, and the Decoders for those banks then have no way of knowing where to look for them in a generic way. Enter the !DecoderDB and !RawEventFormat.

---++ *What's the issue?*

   * *Missing pieces:* Since we need to minimise the footprint of data on the grid, and minimise the bandwidth requirement for user analysis, certain processings re-arrange and drop pieces of the raw event. 
   * *Moved pieces:* Since opening the !RawEvent requires parsing the extent of every bank, for use cases where we don't need everything we can gain CPU by breaking the raw event up into pieces in an earlier step. We can store multiple !RawEvents, with different potentially overlapping !RawBanks elsewhere in the TES.
   * *Backwards incompatibility:* moving around pieces of the !RawEvent is an inherently backwards-incompatible change, what we needed was a tool which could ensure backwards compatibility.
   * *Accountability:* Before the creation of the framework described here (LHCb &lt; v36r3) we had no way to book-keep the possible location for different raw banks/raw events aside from in the versions of the software used to split it up.

---++ *Talks and other information*

   * Core Software: [[https://indico.cern.ch/contributionDisplay.py?contribId=3&confId=264576][24th July 2013]]: technical details and discussion on implementation
   * Software workshop: [[https://indico.cern.ch/contributionDisplay.py?contribId=6&sessionId=1&confId=256184][7th November 2013]]: soon after and during implementation
   * PPTS: [[https://indico.cern.ch/contributionDisplay.py?contribId=5&confId=262634][18th November 2013]]: user guide for RawEventJuggler etc

*please see the PPTS talk, which covers examples of use*

There are also many different tracked tasks which were identified several years ago, and were delayed until LS1, which correspond to this effort:
   * https://savannah.cern.ch/task/?12590: From 2009: "$STDOPTS/DecodeRawEvent.py does two conceptually different things"
   * https://savannah.cern.ch/task/?19106: from 2011: "Create new configurable for subdetector decoding options"
   * https://savannah.cern.ch/task/?31624: From 2012: "Moore should set options to look for Raw banks only in DAQ/RawEvent"

---++ *Component Building blocks*

This framework comprises several LHCb packages which are inter-dependent.
   * *Decoders:* Each !RawBank nominally has a corresponding decoder which needs to know where the raw bank is supposed to be found. These are each stored in their own package living in the LHCb project, for example see [[https://svnweb.cern.ch/trac/lhcb/browser/LHCb/trunk/Hlt/HltDAQ][HLTDAQ (SVN)]]
   * *DAQSys:* [[https://svnweb.cern.ch/trac/lhcb/browser/LHCb/trunk/DAQ/DAQSys][DAQSys (SVN)]] is the package which holds all the pieces necessary to steer the decoders and configure the decoders themselves. The two parts which you need to know about are:
      * *DecoderDB*: [[https://svnweb.cern.ch/trac/lhcb/browser/LHCb/trunk/DAQ/DAQSys/python/DAQSys/Decoders.py][SVN]] a python module which makes a very simple python dictionary of how to configure certain decoders if they are needed. It translates the many different methods of decoding into one simple interface to simplify configuration.
      * *DecodeRawEvent()*: [[https://svnweb.cern.ch/trac/lhcb/browser/LHCb/trunk/DAQ/DAQSys/python/DAQSys/Configuration.py][SVN]] a Configurable which can control the setup of the decoders from the database. Decoding used to be managed by one options file which configured the !DataOnDemand service, but that made it very much static and not very configurable downstream. We replaced this with a configurable which manages the decoders and separated the management for the definition of properties to eset etc, which are put into the !DecoderDB.
   * *RawEventFormat*: [[https://svnweb.cern.ch/trac/lhcb/browser/DBASE/trunk/RawEventFormat][SVN]] a DBASE package which holds a very simple dictionary of which !RawBank can be found in which !RawEvent in which processing version. Since it is a simple python dictionary it has very few requirements and can be used at all levels of our software. Since it is stored as a DBASE package, adding new raw banks or new divisions of software needs to be carefully considered in terms of backwards compatibility. The latest version will be picked up by all software old and new, so additional entries are encouraged, modifying existing entries is discouraged..
   * *RawEventCompat*: [[https://svnweb.cern.ch/trac/lhcb/browser/LHCb/trunk/DAQ/RawEventCompat][SVN]] is a package living in LHCb which finds and uses the !RawEventFormat dictionaries. In here live several different configurables and functions useful for different things.

---++ *Related documentation:*
   * L0App
   * RunningMooreOnMC

---++ *Python modules and help*

Aside from the actual decoders themselves, the framework lives entirely in python. This implies that:
   * you can browse through the code, and get access to the inbuilt python help on modules.
   * As long as you know python, you'll be very able to comprehend and even edit the content of the packages

%SYNTAX{ syntax="sh"}% 
SetupProject LHCb --use RawEventFormat 
python %ENDSYNTAX% %SYNTAX{ syntax="python"}% 
from DAQSys import DecoderClass, DecoderDB
help(DecoderClass)
help(DecoderDB)
help(DecoderClass.Decoder)
print DecoderDB.decoders #it's a dictionary!

from RawEventFormat import Raw_location_db
print Raw_location_db #it's a dictionary!
from RawEventCompat import Configuration as R_E_C
help(R_E_C)

from Configurables import RawEventFormatConf, RawEventJuggler, DecodeRawEvent
help(RawEventFormatConf)
help(RawEventJuggler)
help(DecodeRawEvent)
%ENDSYNTAX%

---++ *Feature Discussion*

---+++ - Transience

The databases used in the configuration are all soft-coded. At any point in your own user code you can extend them or recreate them completely with different content. Examples include:
   * Temporarily changing the properties of a decoder for your specific use
   * Running two versions of the same decoder to/from different locations for your personal use
   * Splitting up the raw event differently from one of the "standard" versions for your personal use

---++ *How to interpret RawEventFormat*

   * *RawEventFormat*: [[https://svnweb.cern.ch/trac/lhcb/browser/DBASE/trunk/RawEventFormat][SVN]] a DBASE package which holds a very simple dictionary of which !RawBank can be found in which !RawEvent in which processing version. Since it is a simple python dictionary it has very few requirements and can be used at all levels of our software. Since it is stored as a DBASE package, adding new raw banks or new divisions of software needs to be carefully considered in terms of backwards compatibility. The latest version will be picked up by all software old and new, so additional entries are encouraged, modifying existing entries is discouraged..

   * The [[https://svnweb.cern.ch/trac/lhcb/browser/DBASE/trunk/RawEventFormat/python/RawEventFormat/__init__.py][__init__.py]] module already holds the dictionaries, so that you can =from !RawEventFormat import ...=
   * *Raw_format_db*: a dictionary of *version* : *configuration*
   * *version* : an arbitrary number designation for a certain placement of raw banks used somewhere in the past
   * *configuration* : a dictionary of *RawBank* : *[ possible locations ]*
   * *[ possible locations ]* : an ordered list of the most likely place to find the raw bank if it exists. 
   * *Which !RawBanks?*: There is a very large number of !RawBanks  [[http://cern.ch/LHCb-release-area/DOC/lhcb/releases/latest/doxygen/dc/d76/class_l_h_cb_1_1_raw_bank.html][Doxygen]], a lot of which are obsolete but kept in the !RawBank.h definition in order to interpret arbitrary files. Those which are obsolete will not appear on every file, and some banks are created at different stages of processing, so a raw file straight from the pit will not necessarily contain every bank. However, this dictionary stores where it will be *if* it exists, for every defined raw bank. All raw banks known at the time of the processing of these data appear in this list. Even if they do not appear on the file you are looking at, at least this should tell you where they logically should be put if they do exist on this file.
   * *Reco_dict_db*: a dictionary of *logical name string* : *version*. This dictionary contains logical names associated to different versions, for example it allows us to book-keep version 2.0 as "the version used for stripping 20". This is a 1:1 mapping and should not be changed later, since older versions may pick up the changes you make. It can be added to, though, e.g. you could add "Brunel after v44r8" if that had a different format, or "Reco 99" if that has a different format. Since this is a 1:1 mapping, the juggling tools accept either a float corresponding to a version number, or a string corresponding to a logical name.
   * *logical name string*: logical names associated to different versions, for example it allows us to book-keep version 2.0 as "the version used for stripping 20".  Juggling tools accept either a float corresponding to a version number, or a string corresponding to a logical name.


---++ *RawEventJuggler user guide*

<img src="%ATTACHURLPATH%/Juggler1.png" alt="object diagram for the juggler" width='600' />

The RawEventJuggler is a configurable which is smart enough to be able to switch between any two !RawEvent formats. It can re-arrange banks into different places, split up and recombine banks, remove banks, remove locations, all sorts of things. Since it is fully integrated into the !RawEventFormat system it is quite simple to move around banks and write something out afterwards. The best source of documentation is the code itself:
   * [[https://svnweb.cern.ch/trac/lhcb/browser/LHCb/trunk/DAQ/RawEventCompat/python/RawEventCompat/Configuration.py#L289][SVN]]
   * [[http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/lhcb/releases/latest/doxygen/py/d2/d87/class_raw_event_compat_1_1_configuration_1_1_raw_event_juggler.html][DOxygen]].

Common options include:%SYNTAX{ syntax="python"}% 
class RawEventJuggler(ConfigurableUser):
    """
    A complex configurable ...
    Juggle around raw event locations, will split up and/or combine anything from anywhere to anywhere else
using maps.

    Aimed for use during the juggling currently performed in Brunel, and for future use in MC between differ
ent Moore versions

    Can configure writers, sequencers and/or DoD

    """
    __queried_configurables__ = [RawEventFormatConf]

    __slots__ = {
        "Input" : None  #Raw event location to start with. E.g. 2.0
        , "Output" : None #Raw Event location to go to. E.g. 0.0
        , "KillInputBanksBefore" : None #Regex of banks to kill BEFORE copying around, from input locations
        , "KillInputBanksAfter" : None #Regex of banks to kill AFTER copying around, from input locations
        , "KillExtraBanks" : False #Bool, whether or not to remove banks which don't exist in target format
        , "KillExtraNodes" : False #Bool, whether to remove nodes which don't exist in target format
        , "KillExtraDirectories" : False #Kill not only the RawEvent, but also unlink the parent directories. Requires KillExtraNodes.
        , "WriterOptItemList" : None #append a writer here to add output locations as OptItemList
        , "WriterItemList" : None #append a writer here to add output locations as ItemList
        , "Sequencer" : None #send in a sequencer to add the juggling into
        , "DataOnDemand" : False #Add juggling into DoD?
        , "Depth" : "#1" #what to add to the end of the writer, depth of 1 is fine for RawEvents
        , "TCK" : None #which TCK to use to replace the TCKReplacePattern if encountered
        , "TCKReplacePattern" : "#TCK#"
        , "RootInTES" : "/Event" #I don't think you'll need to set this, but just in case, it's there too
        , "GenericReplacePatterns" : {} #List of additional patterns in locations to replace with other ones, performs location.replace(key,value) on every location
        }
%ENDSYNTAX%

---++ *Examples of using/editing this framework*

There are many examples in the PPTS talk: [[https://indico.cern.ch/contributionDisplay.py?contribId=5&confId=262634][18th November 2013]]. Here we will cover a few of them.

---+++ - Examples in existing code:

   * *DstConf*: [[https://svnweb.cern.ch/trac/lhcb/browser/LHCb/trunk/GaudiConf/python/GaudiConf/DstConf.py?rev=167200#L138][SVN]]
   * *Brunel*: [[https://svnweb.cern.ch/trac/lhcb/browser/Brunel/trunk/Rec/Brunel/python/Brunel/Configuration.py?rev=167200][SVN]]
   * *Moore*: [[https://svnweb.cern.ch/trac/lhcb/browser/Moore/trunk/Hlt/Moore/python/Moore/Configuration.py?rev=167200#L455][SVN]]
   * *HltLine*: [[https://svnweb.cern.ch/trac/lhcb/browser/Hlt/trunk/Hlt/HltLine/python/HltLine/HltDecodeRaw.py?rev=167200][SVN]] (!HltDecodeRaw.py)

---+++ - The most basic LHCb script

%SYNTAX{ syntax="python"}% 
#The simplest possible LHCb script
from Gaudi.Configuration import *
from Configurables import LHCbApp
LHCbApp()
from GaudiConf import IOHelper
IOHelper().inputFiles([“some_file.dst”])
%ENDSYNTAX%

Let's demonstrate what you need to do to do some juggling with:

%SYNTAX{ syntax="python"}% 
from Configurables import DecodeRawEvent, RawEventFormatConf, RawEventJuggler, RecombineRawEvent
ApplicationMgr().TopAlg+=[GaudiSequencer(“Spam”)]
%ENDSYNTAX%

---+++ - Moving around banks so that multiple TCKs can be stored.

Starting from the most basic script:
   * create a file where the trigger banks have been copied to a TCK-dependent location %SYNTAX{ syntax="python"}% 
Writer=InputCopyStream(“MyStream”)
IOHelper().outStream(“my_copy_file.dst”,writer=Writer)

RawEventJuggler().Input=“Pit”
RawEventJuggler().Output=3.0
RawEventJuggler().TCK=“0x990099”
RawEventJuggler().Sequencer=GaudiSequencer(“Spam”)
RawEventJuggler().KillExtraBanks=True
RawEventJuggler().KillExtraNodes=True
RawEventJuggler().WriterOptItemList=Writer

%ENDSYNTAX%
   * use that file in a later Gaudi job %SYNTAX{ syntax="python"}% 
#
# Either
#

#Recreate original locations on demand
RawEventJuggler().Input=3.0
RawEventJuggler().Output=“Stripping20”
RawEventJuggler().TCK=“0x990099”
RawEventJuggler().Sequencer                  =GaudiSequencer(“Spam”)


#
# Or
#

#just use the new version
RawEventFormatConf().TCK=“0x990099”
DecodeRawEvent().OverrideInputs=3.0

%ENDSYNTAX%

---+++ - Move around events into "Stripping20" format

Starting from the most basic script:
   * create a file where the banks have been split up !RawEvent living in *DAQ/RawEvent*%SYNTAX{ syntax="python"}% 
Writer=InputCopyStream(“MyStream”)
IOHelper().outStream(“my_copy_file.dst”,writer=Writer)

RawEventJuggler().Input=“Pit”
RawEventJuggler().Output=“Stripping20”
RawEventJuggler().Sequencer=GaudiSequencer(“Spam”)
RawEventJuggler().KillExtraBanks=True
RawEventJuggler().KillExtraNodes=True
RawEventJuggler().WriterOptItemList=Writer

%ENDSYNTAX%
   * use this file in another later Gaudi Job, e.g. Moore%SYNTAX{ syntax="python"}% 
#
# Either
#

#recreate the original locations
RawEventJuggler().Input=“Stripping20”
RawEventJuggler().Output=“Moore”
RawEventJuggler().Sequencer                  =GaudiSequencer(“Spam”)



#
# Or
#

#just use the new locations
DecodeRawEvent().OverrideInputs       = “Stripping20”

#
# Or
#

#recreate DAQ/RawEvent
RecombineRawEvent()

%ENDSYNTAX%

---+++ - Move around events into "Pit" format for Brunel/Moore/L0App/Swimming

Same as the above answer for moving into Stripping20 format, but swap the Input and Output. =Input = "Stripping20", Output =0.0= .

---+++ - Custom move one bank somewhere

Starting from the most basic script:
   * define a new "custom" location %SYNTAX{ syntax="python"}% 
RawEventFormatConf().forceLoad()
myLocs=RawEventFormatConf().Locations
myLocs[“Custom”]=myLocs[2.0]
myLocs[“Custom”][“TT”]=“My/Custom/Location”

%ENDSYNTAX%
   * move to this location %SYNTAX{ syntax="python"}% 

RawEventJuggler().Input=2.0
RawEventJuggler().Output=“Custom”
RawEventJuggler().Sequencer=GaudiSequencer(“Spam”)
RawEventJuggler().KillExtraBanks=True
RawEventJuggler().KillExtraNodes=True
RawEventJuggler().WriterOptItemList=Writer

%ENDSYNTAX%

---+++ - Customize a decoder

   * edit a decoder property %SYNTAX{ syntax="python"}% 
from DAQSys.Decoders import DecoderDB as ddb

ddb["DecodeVeloRawBuffer/createVeloClusters"].Properties[“OutputLevel”]=DEBUG

%ENDSYNTAX%

---+++ - Run only one decoder

   * add one deoder to a specific sequence %SYNTAX{ syntax="python"}% 
from DAQSys.Decoders import DecoderDB as ddb
from DAQSys.DecoderClass import decodersForBank

mySeq.Members+=[d.setup() for d in decodersForBank(ddb,“Velo”)]

%ENDSYNTAX%

---+++ - Move and kill banks/locations, insert a sequence as a post-config action

Sometimes you will need to move banks around even though there are other sequencers around. If you intend to kill some banks or locations, you can't use !DataOnDemand, so, what can you do? You'll need to run the Juggler as a sequence, but insert it first in the list of running algs as a sequence. This might not entirely work either if somethign is trying to access event data before your sequence is run or if someone else has also used a post-config action. So it might be best to re-arrange and write out a juggled file in a pre-processor step.

Starting from the most basic script:
%SYNTAX{ syntax="python"}% 
doIExpectYouToTalk=False
slowmovingdeathtrap=GaudiSequencer("JuggleSeq")

RawEventJuggler().Sequencer=slowmovingdeathtrap
RawEventJuggler().DataOnDemand=doIExpectYouToTalk
RawEventJuggler().KillExtraNodes=not doIExpectYouToTalk

def noMrBondIExpectYouToDie():
    ApplicationMgr().TopAlg=[slowmovingdeathtrap]+ApplicationMgr().TopAlg

if not doIExpectYouToTalk:
    appendPostConfigAction(noMrBondIExpectYouToDie)
%ENDSYNTAX%

---+++ - Permanently adding a new raw bank/decoder into the system

There are several steps to take to add a new raw bank.
   1 Add the bank type to the enumerated list of possible banks and to the list of how to translate this into a string [[http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/davinci/releases/latest/doxygen/dc/d76/class_l_h_cb_1_1_raw_bank.html][RawBank.h]]
   1 Create the class which actually is the bank itself, ideally using GOD XML
   1 Create the encoder/decoder for this bank, which should be similar to an existing decoder, ideally, with configurable input and output locations. Ideally the input locations should be a list of possibilities.
   1 Create a new versioned !RawEventFormat entry which adds your new bank into the !RawEventFormat database. Do not edit existing entries, if this bank is from a new detector, you will need to at least create two new versions, one which describes where this bank is when it comes out of the pit, the input to !L0App, Tesla, Moore, and Brunel, the second which describes where it should be put *after* Brunel, into which of the possible !RawEvents.
   1 Add the configurations for your decoder into the !DecoderDB, copy an existing entry. It should be rather straight forward given all the examples in the package.

---+++ - Example of workflow in stripping20 and juggling that would work for swimming the same stripping:

<img src="%ATTACHURLPATH%/Swim1.png" alt="Stripping workflow" width='600' />

<img src="%ATTACHURLPATH%/Swim2.png" alt="Moore swimming workflow" width='600' />

<img src="%ATTACHURLPATH%/Swim3.png" alt="DaVinci swimming workflow" width='600' />

-- Main.RobLambert - 21 Jan 2014
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	DAQ1.png	r1	manage	162.9 K	2014-01-22 - 14:40	RobLambert	OD for DAQSys
png	Juggler1.png	r1	manage	151.9 K	2014-01-22 - 14:35	RobLambert	Object diagram for the juggler
png	Swim1.png	r1	manage	91.6 K	2014-02-18 - 12:07	RobLambert	Stripping20 raw event locations
png	Swim2.png	r1	manage	137.0 K	2014-02-18 - 12:07	RobLambert	Swimming Moore for S20
png	Swim3.png	r1	manage	141.9 K	2014-02-18 - 12:07	RobLambert	Swimming DaVinci
This topic: LHCb > LHCbTrigger > LHCbTriggerForUsers > RawEventJuggler
Topic revision: r13 - 2014-03-31 - RobLambert
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback