MDF Files

How to convert DST to MDF?

Python way (new)

I used the following python file within a Panoramix v17r1 environment. (You need the latest aka "head" version of DAQ/MDF as of this writing.)

import Gaudi.Configuration as CFG
import Configurables as Configs

# define some numeric values
MDF_NONE = 1
MDF_RECORDS = 2
MDF_BANKS = 3

# create an mdf writer
wr = Configs.LHCb__MDFWriter('Writer_2')   # arg is some name
wr.Connection = '/tmp/test.mdf'                 # output file
wr.InputDataType = MDF_NONE      
wr.DataType = MDF_RECORDS
wr.Compress = 0
wr.GenerateMD5 = True

# create a dumper (in case u want to check which raw banks are wrote)
dmp = Configs.LHCb__RawEventTestDump('Dump')
dmp.CheckData = 0
dmp.DumpData = 1
dmp.FullDump = 0
dmp.OutputLevel = 3


from LHCbConfig import *

lhcbApp.DataType = "2009"

appConf = ApplicationMgr( OutputLevel = INFO, AppName = 'readtest')
EventSelector().PrintFreq  = 100

appConf.OutStream.append(wr)
#appConf.TopAlg.append(dmp)  # uncomment this if u want a dump per event for control

import GaudiPython
gbl=GaudiPython.gbl
from gaudigadgets import *
appMgr = GaudiPython.AppMgr()

# event selector (define input)
sel = appMgr.evtsel()
sel.open(['PFN:/castor/cern.ch/user/d/dijkstra/Stripped-L0-MC09/test-100k.raw'])

# event service
evt = appMgr.evtsvc()

n = 1000  # number of events to run on
nread=0
nL0YES=0
while n>0 :
  n-=1
  appMgr.run(1)
  # check L0
  L0DUReport = evt['Trig/L0/L0DUReport']
  if L0DUReport == None :
    print 'probably end of input, or...'
    break  # probably end of input
  nread+=1
  if nread<2:
    evt.dumpAll()
    print L0DUReport
    for i in range(0,9):
      print i,L0DUReport.channelName(i)


print 'read ',nread,' events'
print 'L0DU accepted ',nL0YES,' events'

C++ opts way (classic)

The following option file run with Moore (or I think any Gaudi-, Davinci-, etc. App) will create an MDF file with the events from the DST. (Most of this code provided by Markus Frank)

NOTE: There seems to be an upper limit for the file size of the mdf file so you might be forced to chop a large DST into smaller chunks containg less events. But I am not sure on that. Could have also been a problem with my file system.

You might want to have a look on the "How to prepare a custom sample" section to prepare a .dst aka .raw file, that you then change into a .mdf. I recently (2009-04-17) checked the following .opts file works with Gaudi.exe and settings for DaVinci v19r12. TODO: Translate this into a python configurable.

#pragma print off
#include "$STDOPTS/LHCbApplication.opts"
#include "$STDOPTS/DstDicts.opts"
#include "$STDOPTS/DC06Conditions.opts"
#pragma print on

ApplicationMgr.EvtMax =  10000;
EventSelector.PrintFreq = 100;
ApplicationMgr.OutputLevel = 10;

ApplicationMgr.HistogramPersistency = "ROOT";
HistogramPersistencySvc.OutputFile = "Moore_minbias.root";

ApplicationMgr.ExtSvc +=  [ "ToolSvc" , "AuditorSvc" ] ;
ApplicationMgr.AuditAlgorithms = 1;
AuditorSvc.Auditors +=  [ "TimingAuditor/TIMER" ] ;
/////// BEWARE: The silly Twiki messes up the next line, go to edit to see it correct //////////////
MessageSvc.Format = "% F%40W<img src="/twiki/pub/TWiki/TWikiDocGraphics/starred.gif" alt="PICK" title="PICK" width="16" height="16" border="0" />7W%R%T %0W%M";
ToolSvc.SequencerTimerTool.OutputLevel = 4;

EventPersistencySvc.CnvServices += { "LHCb::RawDataCnvSvc" };
ApplicationMgr.OutStream += { "LHCb::MDFWriter/Writer_2" };
Writer_2.Connection       = "file:///local_home/snies/DC06_L0_v1_lumi2.mdf";     // the mdf we want to create
Writer_2.Compress         = 0;
Writer_2.ChecksumType     = 1;
Writer_2.GenerateMD5      = true;

EventSelector.Input   = {
  "DATAFILE='PFN:/local_home/snies/DC06_L0_v1_lumi2_MuonHadron.dst' TYP='POOL_ROOTTREE' OPT='READ'"
};

-- StephanNies - 18 Jul 2008 -- Updated on 18 April 2008

How to run Moore on MDF files? (only needed for C++ .opts)

LEGACY CODE! -- NEW VERSIONS OF MOORE COME WITH A Moore.py THAT AUTOMTICALLY RECOGNIZES MDF FILES!

This option file worked for me with Moore_v4r2. You may want to get my mdf files for testing: /castor/cern.ch/user/s/snies/mdf

#pragma print off
#include "$STDOPTS/LHCbApplication.opts"
#include "$STDOPTS/DstDicts.opts"
//#include "$HLTSYSROOT/options/L0.opts"
#include "$HLTSYSROOT/options/L0FromRaw.opts"
#include "$HLTSYSROOT/options/Hlt1.opts"
#include "$HLTSYSROOT/options/Hlt2.opts"
#include "$STDOPTS/DC06Conditions.opts"
#pragma print on

ApplicationMgr.EvtMax =  10000;
EventSelector.PrintFreq = 1;
ApplicationMgr.OutputLevel = 10;

ApplicationMgr.HistogramPersistency = "NONE";

ApplicationMgr.ExtSvc +=  [ "ToolSvc" , "AuditorSvc" ] ;
ApplicationMgr.AuditAlgorithms = 1;

/////// BEWARE: The silly Twiki messes up the next line, go to "edit page" to see it correct //////////////
MessageSvc.Format = "% F%40W<img src="/twiki/pub/TWiki/TWikiDocGraphics/starred.gif" alt="PICK" title="PICK" width="16" height="16" border="0" />7W%R%T %0W%M";
//////////////////////////////////////////////////////////////////////////////////////////////////
ToolSvc.SequencerTimerTool.OutputLevel = 4;

EventPersistencySvc.CnvServices      = {"LHCb::RawDataCnvSvc"};
EventSelector.PrintFreq              = 100;
EventSelector.Input   = {
  "DATAFILE='PFN:file:///local_home/snies/DC06_L0_v1_lumi2.mdf' SVC='LHCb::MDFSelector'" 
};

-- StephanNies - 18 Jul 2008

How to prepare a custom sample for conversion into MDF

The following code is an example of a Python script that chooses L0DU accepted events. The idea is to prepare a custom selection of events, that you than can convert into an MDF file in order to use it as a test sample for Moore. It also provides some histograms in order to check the selection. (Code provided by Hans Dijkstra )

# get the basic configuration from here
from LHCbConfig import *
lhcbApp.DataType = "DC06"

importOptions('$L0DUOPTS/L0DUFromRaw.opts')

# define a helper class:
class irange(object) :
   def __init__(self, b, e ):
     self.begin, self.end  = b, e
   def __iter__(self):
     it = self.begin
     while it != self.end :
       yield it.__deref__()
       it.__postinc__(1)

rawwriter  = OutputStream('RawWriter', Preload = False,
  ItemList = ["/Event#1","/Event/DAQ#1","/Event/DAQ/RawEvent#1","/Event/DAQ/ODIN#1"],
  Output   = "DATAFILE='mdf-test3.raw' TYP='POOL_ROOTTREE' OPT='REC' ")
appConf = ApplicationMgr( OutputLevel = INFO, AppName = 'readtest')
appConf.OutStream = [rawwriter]

appConf.HistogramPersistency = "HBOOK"
HistogramPersistencySvc().OutputFile  = "test2.hbook"

EventSelector().PrintFreq  = 1000

import GaudiPython
gbl=GaudiPython.gbl
from gaudigadgets import *

appMgr = GaudiPython.AppMgr()
sel    = appMgr.evtsel()
#sel.open(['PFN:/afs/cern.ch/lhcb/group/trigger/vol3/dijkstra/Selections/Bd2MuMuKst-lum5.dst'])
sel.open(['PFN:/afs/cern.ch/user/d/dijkstra/python/l0-run/mdf-test.raw'])
evt = appMgr.evtsvc()

hist  = appMgr.histsvc()
#Declare retrieve function for hist ids.
fh=hist.retrieve
h=hist.book('1','L0Du decision',2,-0.5,1.5)
h=hist.book('11','hadron Et',256,0.,5.12)
h=hist.book('12','e Et',256,0.,5.12)
h=hist.book('13','photon Et',256,0.,5.12)
h=hist.book('14','muon1 pt',256,0.,5.12)
h=hist.book('21','hadron Et',256,0.,5.12)
h=hist.book('22','e Et',256,0.,5.12)
h=hist.book('23','photon Et',256,0.,5.12)
h=hist.book('24','muon1 pt',256,0.,5.12)

appMgr.algorithm('RawWriter').Enable = False  # stop automatic execution of RawWriter

n = 1000
nread=0
nL0YES=0
while n>0 :
 n-=1
 appMgr.run(1)
 # check L0
 L0DUReport = evt['Trig/L0/L0DUReport']
 if L0DUReport == None : break  # probably end of input
 nread+=1
 if nread<2: evt.dumpAll()
 if L0DUReport.decision()<>0:
      nL0YES+=1
      datas      = L0DUConfig.data()
      for x in irange(datas.begin(),datas.end()) :
        idata = x.second
        if idata.name()=='Hadron(Et)': fh('11').fill(idata.value()/1000.)
        if idata.name()=='Electron(Et)': fh('12').fill(idata.value()/1000.)
        if idata.name()=='Photon(Et)': fh('13').fill(idata.value()/1000.)
        if idata.name()=='Muon1(Pt)': fh('14').fill(idata.value()/1000.)
      rc = appMgr.algorithm('RawWriter').execute() # output event
 else:
      fh('1').fill(0)
      L0DUConfig = L0DUReport.configuration()
      datas      = L0DUConfig.data()
      for x in irange(datas.begin(),datas.end()) :
        idata = x.second
        if idata.name()=='Hadron(Et)': fh('21').fill(idata.value()/1000.)
        if idata.name()=='Electron(Et)': fh('22').fill(idata.value()/1000.)
        if idata.name()=='Photon(Et)': fh('23').fill(idata.value()/1000.)
        if idata.name()=='Muon1(Pt)': fh('24').fill(idata.value()/1000.)



print 'read ',nread,' events'
print 'L0DU accepted ',nL0YES,' events'

Other places to look at

L0MuonHowToReadMDFOnMarmuon4 (How to read and analyze the raw data aka MDF on disk)

-- StephanNies - 03 Aug 2009