LHCb File Summary Record

  • FileSummaryRecord:
    • AKA FSR is a section of data within an LHCb data file reserved for metadata about the file content.
  • Frustrating Summary Record:
    • due to the ongoing problems of memory management and lack of maturity of this concept, I have often used this acronym instead.

....... Unsustainable

The Event store, Gaudi and Root files

LHCb data formats:

  • LHCb data are stored in MDF files (see RawEvent) or in ROOT files.
  • ROOT format starts with a single "Tree" of data, off which we hang "Branches" of C++ classes and containers, and finally "Leaves" of the member data held by those C++ classes.
  • LHCb ROOT files store data in a tree saved as "Event", which is why all locations start with "/Event/..."

Root Users: If you are a user familiar with ROOT, you will have noticed some strange features of Root files. For example:

  • the number of events stored in a root file is only known if you finish reading over the entire file
  • there is no file-header within which you are free to add metadata about the file, counters, and other information
  • To store data within a root file can be very costly, or very cheap, depending on how the compression handles it, and how many identical copies of data you are storing.

Metadata? In a sense Root files have no concept of user metadata. Although they store information about which ROOT version was used and can store entire arbirtrary C++ classes they lack the simple abilities of several other file types of up-front metadata in a dedicated file header or footer (see, for example, XML or .doc, or .pdf, .jpeg ...).

Gaudi users: If you are a user familiar with the Gaudi framework, you will also have noticed some odd things:

  • Gaudi uses the baseclass DataObject to store events to root trees, and most of these classes are created automatically from xml descriptions by the Gaudi Object Description, or GOD.
  • Each opened file is allocated a GUID (grid-unique-identification) independently of whether it ever goes to the grid
  • GUIDs are not-completely-unique-but-probably-pretty-unique hashes of the file content which are used instead of the filename to identify files and objects within Gaudi
  • Gaudi does store some provenance information about files, in terms of links to older file GUIDs
  • The provenance of an opened file is known to Gaudi via a GUID associated to folders in the TES. If a folder location was ever written to a file in the past, it can be resurrected through a local xml catalog which links file GUIDs to physical file names (PFNs).

Metadata? So, in a sense Gaudi does keep some metadata about files. However, the metatadata automatically stored with Gaudi is stored for every event. Events have their own metatdata, which may be compressed away by Root compression, but in principle the event does not care about file information.

LHCb users: If you are a user familiar with the LHCb usage of the Gaudi framework, you will have noticed even more odd things:

  • LHCb does store file metadata, in the file, for example in the RecHeader, GenHeader, such as:
    • the version of the software used,
    • the database tags applied in previous processing steps
    • This metadata is stored for every event

Metadata? In an ideal future-proof metadata system, files would arrive with the full knowledge of how they were produced, such that if you have access to the file, you know everything. In LHCb you require several systems to interrogate that information, the book-keeping, the Dirac production system, the released software, the TWiki pages...

Advantages to storing all metadata per event:

  • Events and their provenance can be uniquely identified
  • Events are self-contained, if an event makes it through your filtering system you are guaranteed to have its metadata.
  • You can forget about the file containing a given event

Disadvantages to storing all metadata per event:

  • If an event does not make it through your filter, its metadata is also lost.
  • If metadata are common to thousands of events this is wasteful in disk space and memory usage.
  • Adding new metadata becomes prohibitive in terms of file size.

The FileSummaryRecord

The concept: The concept of a file-summary-record (FSR) is a relatively new addition to the Gaudi Framework, it is the combination of several things.

  • A new "event" store: since Root files have no concept of metadata, and Gaudi did not have a concept of metadata either, we implemented FSRs as a secondary event store, a new tree kept in the same root file, starting with "FileRecords"
  • New DataObjects: since Gaudi can only write DataObjects to Root files using the existing framework, we needed to add every metadata concept we can think of into the event class model, creating dedicated DataObjects
  • New services/algorithms: this datastore needs algorithms which fill it, and services which provide accessors to the data.
  • Automatic Propagation: FSRs are expected to be transparently passed from file to file hand-over-hand by the dedicated services. File metadata only increases, and is preserved in a tree structure using the GUID of the input files as directories.

Resources:

What do I need to do to enable FSRs correctly?

  • FSRs are Gaudi constructs that can only live on Root-formatted output files, like SIM/DST etc.
  • To write FSRs needs special background services to be in place (an extra event store) and an extra writer to be added to each file (to write the FSRs).
  • IOHelper (and IOExample) are helper classes which wrap up all of the various things required to enable FSRs, which includes the special event store and special writers.
  • Most configurables with an "OutputFile"-like configurable option/slot use IOHelper correctly to add FSRs when required. These configurables might have a "WriteFSR" option to them, which by default is usually "True".
  • LHCbApp(), the most basic LHCb application, itself uses IOHelper to set up the underlying services. Since most LHCb applications also configure LHCbApp, you usually don't need to do anything here.
  • If you are not using LHCbApp, or some other automated output of an existing configurable you will need to call the right things from IOHelper yourself.
  • outputAlgs is a method of IOHelper which returns a list of algorithms, one of which will be an OutputStream-like object, another with be a RecordStream-like object. In case you are adding your own writer in the middle of a sequence, you need this method.
  • outStream is another method which does not return anything, but does automatically add write algorithms to the ApplicationMgr().OutStream, which is a sequence of algorithms run after all others.
  • setupServices() is another method which configures all the correct services, it is used automatically by LHCbApp.
  • For more details use the Doxygen or SVN of IOHelper, or use the in-built python help.

What's there in your FSR? How to check if everything is OK?

SetupProject DaVinci
$APPCONFIGROOT/scripts/CheckFSRs.py <somefilename>

Automatic propagation: A SHRUBBERY

  • TES: FSRs in the current TES, for the current file yet to be written are stored under "/FileRecords"
  • Opened Files: On opening a new file, all its FSRs are duplicated into the tree under "/FileRecords/<GUID>/..."
  • Topology: Since this is iterated over many production applications, the resulting tree is very deep and broad, but also sparse:
    • deep: there are many levels in the tree
    • broad: there are many branches at each level in the tree
    • sparse: each branch will have only a few entries, often only one double or int at the lowest hanging level, and there is nto duplication to multiple "events" each file appears only once.
  • Oh: We end up with a bush rather than a tree.

Services and integration into Gaudi

To understand which services are required to be active and their options in order to propagate FSRs correctly, one can inspect what is done by IOHelper().setupServices(). SVN DOxygen

Interaction with applications/configurables

  • IOHelper and IOExtension
    • have the keyword writeFSR embedded into the outputAlgs and outStream methods.
    • Activating this flag provokes the additional writing of the FSRs onto your output file

  • LHCb Applications:
    • DaVinci, Moore, L0App, FileMerger, DSTConf, and many other LHCb configurables have a WriteFSR slot.
    • In the case the configurable is used directly to create output, this is propagated to the relevant IOHelper method.

  • What happens in Brunel?:
    • In real data. LumiFSRs are initially created from counters created by the HLT.

  • What happens in DaVinci?:
    • FSRs are propagated from DST to DST
    • FSRs are merged in production when thousands of files are used as input
    • Luminosity FSRs can be combined and compared with the conditions database calibrations to arrive at a measure of the luminosity with uncertainties.

Applications/use cases

- LumiCounters

  • In calculating the luminosity the events you did not select have equal importance to the events you did select.
  • This means that some non-event-wise metadata is 100% necessary, and FSRs are the solution implemented in LHCb.
  • To ensure that information on the total number of events seen by a job is not lost, and to eventually calculate the luminosity, the following metadata are required:
  1. Counters: various counters on event rates from lumi-limited well-calibrated lines
  2. EventCountFSR: a counter of how many events were seen from each file in order to validate that all events were read correctly
  3. Time stamps: the first event time of a file, and the last event time of a file must be known to use the conditions database calibrations
  4. Run numbers: the run numbers present in a certain file must be known.

Each of these FSRs is ensured to be mergable such that the memory and file-size overhead is kept down.

  • Limits on mergability:
    1. Brunel can not have processed events form different runs into the same DST.
    2. Merging across runs later in processing has had problems in the past, but nominally it should work with the latest production software.

Future Applications/use cases:

  • (note that these twiki do not exist yet, because this is still only suggested work at the time of writing)
  • IOFSR: recording metadata about the file provenance direct into the FSR. This could eventually solve the memory explosion problem.
  • GeneratorLevelFSRs: propagate all generator counters for perfect generator-level-cut efficiency evaluation. NB: this will need a lot of merging.

Comparison in pictures

A Root TreeAn FSR Bush

Major issue: Memory and file-size explosion

  • Memory-explosion.
    • As we discussed above, the FSR tree we create is very deep and broad, but also sparse.
    • ROOT files are efficient at storing shallow, narrow, dense trees.
      • shallow: not very many levels
      • narrow: not many branches per level
      • dense: many hundreds or thousands of entries per event, many thousands of almost identical event structures.
    • The further you push ROOT away from this regime, the more file size grows, and the more memory is consumed on reading/writing the files.

  • Packing, compressing, allocating:
  • ROOT files auto-compress themselves, and auto-adjust their caches, assuming that they are shallow, narrow, dense trees, and this is a very good assumption.
  • LHCb files often hold very much the same information from one event to the next, especially in high-occupancy situations.
  • ROOT optimizes the memory layout of reading and writing files after 10 events have been written.read.
  • FSRs, however, are only written/read once, and so the maximum size is always used as a cache and compression is completely pointless, it only costs disk space and CPU time.

  • Extreme example:
  • in a simplified tree model, each branch is a string of the branch name and a pointer to the keyed container object.
  • the contained object must have non-zero number of members to interface correctly with what ROOT expects from objects within its trees, some of that information can be stored as metadata of the branch, and some is stored in the leaves, once per data member,
  • The EventCountFSR is a very simple FSR which stores two long long ints.
  • If 8 of them are stored to a file as a tree with only 2 bifurcations at each branch, instead of storing 8*2 packed ints we are now storing
    • Size on file: (1+2+4+8) * (string + pointer + container + root-specific-rubbish ) + 8*2 packed ints
    • Size in memory: (1+2+4+8) * (string + pointer + container + root-specific-rubbish + bucket + cache) + 8*2 packed ints
    • Mathematically: [Sum( depth=1 to deepest) branches^depth]*overhead + branches^deepest *2 packed ints
    • Note that the root-specific-rubbish + bucket + cache is likely to be of order 10kB and so is vastly bigger than the size of the two packed ints.
  • The deeper the tree the exponentially more memory is required to store these objects. The broader the tree, the polynomially more memory is required.

It is silly to think about it, but adding one low-level FSR right now is more expensive on memory usage and file size than adding the same information to every event, where it will be compressed away. So FSRs should only be used for the MINIMAL possible use cases which are not addressable in any other way.

Merging!:

  • To overcome the memory usage problems, we must translate the deep, broad, sparse trees into narrow, shallow, dense, trees.
  • This is accomplished by merging FSRs together into a single top-level FSR.
  • In production this must be done whenever many files are concatenated into one file.
  • For user productions of large sparse file mergings, for example a sub-selection of 1/1000 events, and in particular with microDSTs, this should also be done.
  • Since the growth of the memory usage is exponential in terms of depth, even reducing the depth by one level can have a massive improving effect.

Related talks:

Related Pages:


-- RobLambert - 24 Mar 2014

Topic attachments
I Attachment HistorySorted ascending Action Size Date Who Comment
PNGpng Slide1.PNG r1 manage 221.8 K 2014-03-24 - 16:53 RobLambert A root tree
PNGpng Slide2.PNG r1 manage 315.6 K 2014-03-24 - 16:54 RobLambert A root bush

This topic: LHCb > WebHome > LHCbComputing > FileSummaryRecord
Topic revision: r9 - 2014-04-14 - RobLambert
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback