How to create HLT TCKs

Creating split HLT1 and HLT2 TCKs for 2016

Producing a dataset with at least one accepted event per HLT line

*This step is not normally needed, as the existing dataset should be sufficient. Please consult with one of the HLT experts if in doubt.* There is a script in Hlt/HltPiquetScripts which does this. scripts/Moore_Hlt1Hlt2_MakeDataForTCK_Parallel.py This script runs Moore with a configurable number of parallel processes. It keeps track of which lines have fired, and only writes out an accepted event if it is the first time for any of the lines. It is slightly more efficient though, since it doesn't actually require that the line fires -- only that the next to last algorithm has been executed. First step is to make sure that we have suitable input data. The process is most efficient if we start from a mix of raw data from the different streams. E.g,.

nsls /castor/cern.ch/grid/lhcb/data/2015/RAW/TURCAL/LHCb/COLLISION15/164699/ | head -n 5

getpack Hlt/HltPiquetScripts

lb-run Moore v25r1 python Hlt/HltPiquetScripts/scripts/Moore_Hlt1Hlt2_MakeDataForTCKs_Parallel.py 2>&1 | tee log

Creating and testing split HLT TCKs.

Before you begin, you should have an appropriate hex key for the new TCKs. You can e.g. use the latest (use SetupMoore --list-versions) released Moore (v25r2 in this example) with TCKsh to find out:

lb-run Moore v25r2 TCKsh
>>> listConfigurations()
.... ....
MOORE_v25r2
    Calibration_VeloMicroBias
      0x11301715 : b95ffa736bfc93d10bae16e76a3a6c96 : Prescale = 1
    Physics_pp_May2016
      0x11291600 : 8eed26d4c1e458200a7bb0ff69deeaf3 : Hlt1, Physics_pp_May2016, 0x1600
      0x21291600 : c3269de2a6f7e0bd687a82af0879778d : Hlt2, Physics_pp_May2016, 0x1600

Now you are ready to create and test the TCKs. First, create the Moore environment. If you need to check out versions of packages that are different from the Moore release, then you will need to do an lb-dev. The following is an example with Moore v25r2.

lb-dev Moore v25r2
cd MooreDev_v25r2
# getpack XYZ if needed
make purge
make -j 8 install

First we need to prepare the input data. The starting point is a small sample of real data events which is selected such that all HLT lines get to their next to last algorithm. It is currently at

 /eos/lhcb/wg/HLT/2016CommissioningDatasets/OneAcceptedPerLine_v25r1_Physics_pp_May2016_190416 

#RedoL0.py
from Gaudi.Configuration import ApplicationMgr
from Configurables import L0App
app = L0App()
app.ReplaceL0Banks = True
app.outputFile = "RedoL0.mdf"
app.EvtMax = -1
app.TCK = '0x1600'
app.DataType = '2015'
app.CondDBtag = 'cond-20150828'
app.DDDBtag = 'dddb-20150724'

from GaudiConf import IOHelper
IOHelper('MDF').inputFiles(['mdf:root://eoslhcb.cern.ch//eos/lhcb/wg/HLT/2016CommissioningDatasets/OneAcceptedPerLine_v25r1_Physics_pp_May2016_190416/Moore%02d.mdf' %i for i in range(0,15)]) 

from Configurables import EventSelector
EventSelector().PrintFreq = 10

from Configurables import HltRoutingBitsFilter
rb_filter = HltRoutingBitsFilter( "RoutingBitsFilter" )
rb_filter.RequireMask = [0x0, 0x4, 0x0]

# HLT Bank killer                                                                                                                                                                                                                               
from Configurables import bankKiller
hlt_banks = [ 'HltDecReports','HltRoutingBits','HltSelReports','HltVertexReports','HltLumiSummary','HltTrackReports' ]
killer = bankKiller( 'RemoveInputHltRawBanks',  BankTypes = hlt_banks )

ApplicationMgr().TopAlg = [rb_filter, killer]

./run gaudirun.py RedoL0.py 2>&1 | tee RedoL0.log

The scripts to create and test the Hlt TCKs live in the Hlt/Moore package under tests/options/TCK/ You can see them e.g. with

./run ls '$MOOREROOT/tests/options/TCK'

cp `./run echo '$MOOREROOT'`/tests/options/TCK/* .  ## for some reason I can't just do ./run cp '$MOOREROOT/tests/options/TCK/*' .

IOHelper('MDF').inputFiles(['PFN:RedoL0.mdf']) 

./run gaudirun.py CreateTCK1.py 2>&1 | tee CreateTCK1.log

ID=`cat CreateTCK1.log | grep HltGenConfig | grep "top id for this config" | cut -d ":" -f 2 | xargs`

./run python CreateTCKEntry.py ${ID} 0x112F1600 2>&1 | tee CreateTCKEntry1.log

./run gaudirun.py TestTCK1.py 2>&1 | tee TestTCK1.log

./run gaudirun.py CreateTCK2.py 2>&1 | tee CreateTCK2.log
ID=`cat CreateTCK2.log | grep HltGenConfig | grep "top id for this config" | cut -d ":" -f 2 | xargs`
TCK2=0x212F1600
CreateTCKEntry.py ${ID} ${TCK2} 2>&1 | tee CreateTCKEntry2.log
./run gaudirun.py TestTCK2.py 2>&1 | tee TestTCK2.log

# copy_TCK.py
from TCKUtils.utils import *
source_cas = ConfigAccessSvc("ConfigAccessSvcSource", File = 'TCKData/config.cdb') ## copy the new entries from here
target_cas = ConfigAccessSvc("ConfigAccessSvcTarget", File = 'TCK/HltTCK/config.cdb', Mode = 'ReadWrite') ## into here

# either of these would work
for source in ("0x*12F1600"):
#for source in (0x112F1600,0x212F1600):
    glob = 'TCK/0x%08x' % source if type(source) == int else "TCK/" + source
    copy(source = source_cas, target = target_cas, glob = glob)

lb-run --nightly lhcb-head TUE Moore HEAD python copy_TCK.py

$> SetupProject --build-env Moore v24r0
$> cd $User_release_area/Moore_v24r0
$> mkdir TCKData
$> cp /path/to/CreateTCK.py.txt CreateTCK.py

$> SetupProject Moore v24r0
$> cd $User_release_area/Moore_v24r0
$> gaudirun.py CreateTCK.py

$> TCKsh
>>> listConfigurations()
>>> quit()

$> TCKsh
>>> cas_rw = ConfigAccessSvc( Mode = 'ReadWrite', File='/path/to/config.cdb' )
>>> printConfigurations( getConfigurations( cas = cas_rw ) )
starting appMgr @ pid = 720
MOORE_v12r7
    Physics_May2011
          <NONE> : 24e17751af7447fed73ec35b108218b6 : Test TCK
>>> id = '24e17751af7447fed73ec35b108218b6'
>>> createTCKEntries( { 0x00900032 : id }, cas = cas_rw )
creating mapping TCK: 0x00900032 -> ID: 24e17751af7447fed73ec35b108218b6
>>> quit()

from Configurables import ConfigCDBAccessSvc
ConfigCDBAccessSvc().File = 'config.cdb'
Moore().TCKData='/path/to/TCKData'

$> SetupProject Moore v12r7
$> cd $User_release_area/Moore_v12r7/scripts
$> cp runMoore_CreateDST.py.txt runMoore_CreateDST.py
$> gaudirun.py runMoore_CreateDST.py

from Configurables import ConfigAccessSvc
ConfigAccessSvc().File = '/path/to/your/config.tar'