Stripping coordinator's cheatsheet


What is an rDST

When data is collected from the LHCb detector, the raw data will be transferred to the LHCb associated Tier 1 sites for the reconstruction to produce rDST files. The rDST files are used for stripping jobs where events are selected for physics analysis. Selected events are written to DST files and distributed in identical copies to all the Tier 1 sites. These files are then accessible for physics analysis by individual collaborators.

(Taken from Streaming Task Force page)

What is a .xml file

Physical file names (PFN): these are targeted to a site and a given file access protocol. Supported protocols: castor: (CERN), rfio: (CERN, PIC, CNAF), dcache: (GridKa, RAL), gfal (Grid File Access Library) plugin of ROOT

Logical File Names (LFN): these are not targeted to a given site and allow the jobs to run on the Grid. The application has to resolve it to a PFN by means of a File Catalog (FC). The FC can either be the LHCb full catalog (LFC) or a local restricted catalog in XML language (XML catalog).

The genCatalog XML script allows translating a list of files given by their LFN. An XML file catalog allows Gaudi to translate the LFN in the jobOptions file to the appropriate PFN. One can also require that the parent files of the files used are included in the catalog (see the -d --depth option). This is necessary for navigating to ancestor files or using Event Tag Collections.

Note that your rDST might be in the swtest "freezer" but if requiring raw banks the job will also need access to the raw file which is stored on tape (here depth=2 in the genCatalog XML script).

This is an example of an entry in a .xml file

</File>
<File ID="6cbb0118-ce8e-11e7-a034-001e67ddcb96">
<physical>
<pfn filetype="ROOT_All" name="mdf:root://clhcbstager.ads.rl.ac.uk//castor/ads.rl.ac.uk/prod/lhcb/buffer/lhcb/data/2017/RAW/FULL/LHCb/COLLISION17/202214/202214_0000000854.raw?svcClass=lhcbDst" se="RAL-BUFFER"/>
<pfn filetype="ROOT_All" name="mdf:root://eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/swtest/lhcb/data/2017/RAW/FULL/LHCb/COLLISION17/202214/202214_0000000854.raw" se="CERN-SWTEST"/>
</physical>
<logical>
<lfn name="/lhcb/data/2017/RAW/FULL/LHCb/COLLISION17/202214/202214_0000000854.raw"/>
</logical>

(Adapted from Using Gen Catalog)

SVN trickery (obsolete)

This might be useful as mistakes with svn can happen.In this link there are the guidelines of svn.

To tag a version i.e: StrippingSelections to a particular tag execute the following command:

svn cp -r 198298 https://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/ https://svn.cern.ch/reps/lhcb/Stripping/tags/Phys/StrippingSelections/v4r9/ -m "Tagging v4r9"

GIT basics

All basic information and links can be found in https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb.

To checkout single files from a different branch than the branch you're currently working on, say stripping24r1p1-patches, do

 git fetch origin stripping24r1p1-patches
 git checkout origin/stripping24r1p1-patches -- Phys/StrippingSettings/python/StrippingSettings/Stripping24r1p1/LineConfigDictionaries_Calib.py Phys/StrippingSettings/dbase/stripping24r1p1

Fix a messed up history in git

If a merge or commit has been pushed to the repository incorrectly, you can undo it (in a full checkout of the package) with

git revert <commit hash>

You can get the commit hash from `git log`. If it's a merge commit then you need to specify the parent with

git revert -m <parent ID> <commit hash>

In the output of `git log` for a merge commit you'll see the two parent commits that were merged on the "Merge:" line, so normally you'll want to pick 1 or 2.

However, `git revert` changes the files back to the way they were before the commit, but it doesn't revert their history - the change back is logged as a new edit in the history. If you want also to revert the history then you need to find the hash of the commit before the unwanted change and do

git reset <commit hash>

which resets the tree to that commit, then check what's being reverted with

git status

Then revert it with

git reset --hard

Check that the history is as you expect with `git log`, then force push to the remote repo with

git push -f <remote> <branch>

The history on the remote branch will now also be rewritten. Note that gitlab doesn't let you force push to protected branches, so if the target branch is protected you'll need to remove its protection temporarily to do the force push and then re-protect it.

This works if you want to remove all commits after a certain point. If you only want to remove selected commits then instead of `git reset ` and then `git reset --hard`, you want to do

git rebase -i <commit hash>

again the commit hash here is that of the commit before the earliest commit you want to revert. This will open an editor with a list of all commits after that point. Delete the lines of all the commits you want to remove and then save and quit the editor. Then check that the history is OK with `git log`, and force push to the remote branch.

How to get the xml file from RDSTs

lhcb-proxy-init

lb-run LHCbDirac /prod dirac-bookkeeping-genXMLCatalog --Options=YourData.py --Catalog=yourdatacatalog.xml --Depth=2

Useful dirac commands

lb-run LHCbDirac /prod dirac-bookkeeping-file-metadata lfn gives you the number of events, run number, DQ flag amongst others

lb-run LHCbDirac /prod dirac-bookkeeping-file-path --LFN lfn gives you the actual location (useful for freezer cleaning)

lb-run LHCbDirac /prod dirac-dms-lfn-metadata lfn lists the metadata of the file (owner, size, ...)

lb-run LHCbDirac /prod dirac-dms-lfn-replicas lfn shows the existing replicas of the file

lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/CollisionYEAR/CONDITIONS/RealData/90000000/RAW --Trigger gives you the stable beam time

lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/CollisionYEAR/CONDITIONS/RealData///90000000/  --Trigger gives you the size of the stripped data

Please note that you have to write RealData instead of Real Data.

Inspect a DST file with Bender

For example if you'd like to check the Stripping decision of the following MDST file /lhcb/validation/Collision15/BHADRON.MDST/00067793/0000/00067793_00000191_1.bhadron.mdst

lb-run LHCbDirac/latest dirac-bookkeeping-genXMLCatalog -l /lhcb/validation/Collision15/BHADRON.MDST/00067793/0000/00067793_00000191_1.bhadron.mdst
lb-run Bender/latest bender -x pool_xml_catalog.xml -l /lhcb/validation/Collision15/BHADRON.MDST/00067793/0000/00067793_00000191_1.bhadron.mdst

In Bender, seekStripDecision('StrippingZVTOP_High*') will give you the decision for the lines you're interested in. To look at the candidates, use get('Bhadron/Phys/CONTAINER/Particles').

Require RawEvent

If certain raw banks are missing for a single event, which should not be the case, replace DaVinci().appendToMainSequence( [ sc.sequence() ] ) with

sc_seq = sc.sequence()
sc_seq.RequireObjects = ['DAQ/RawEvent']
DaVinci().appendToMainSequence( [ sc_seq ] )

Prerequisites

Stripping coordinators need the following rights:

  • Have lhcb_tech role in DIRAC to be able to modify production steps
  • Be members of lhcb-release-managers e-group to be able to tag projects in tag collector
  • Be members of z5:doclibrarians AFS group to be able to publish stripping lines documentation on the web pages
  • Be subscribed to lhcb-stripping-managers for obvious reasons.

Actions before stripping selections deadline

The deadline for submitting new or modifying the existing stripping selections is decided by the PPG, taking into account the time of conferences where the new results have to be presented, processing time and the time to prepare the stripping and DaVinci release after the deadline. The latter can take up to one month and depends on the number of iterations required to reach the limits on bandwidth and/or timing.

Prepare test samples

Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed. It can make sense to take (parts of) several runs as the test sample if the running conditions or trigger configuration was changing significantly. Once the run for the test sample has been identified, inform bookkeeping manager and ask to copy it to "freezer". The python options have to be prepared and put to Phys/StrippingSelections/tests/data/.

Prepare patch for DaVinci

For incremental and IFT (re-)strippings, DaVinci patches need to be prepared. For incremental re-strippings, the patch should be created from the DaVinci version corresponding to the stripping version and for IFT, make sure you patch from the DaVinci version using the correct Reco version.

If you need to change the environment, do

. LbLogin.sh -c x86_64-slc6-gcc49-opt

or run with

lb-run -c x86_64-slc6-gcc49-opt

To build the patch, you can do


export CMTPROJECTPATH=${CMTPROJECTPATH/$User_release_area/$PWD}
export User_release_area=$PWD
export LBCONFIGURATIONROOT="/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_vXrY/LbConfiguration"

for project in Phys Stripping Analysis DaVinci ; do
    git clone "ssh://git@gitlab.cern.ch:7999/lhcb/${project}.git"
    cd "$project"
    lb-project-init
    git fetch
    git checkout <your_branch>
    make configure >& stdout-make-configure
    make install >& stdout-make-install
    cd ..
done 

FYI, you might need to update the versions of Phys and Analysis in DaVinci or Stripping if you only clone these two packages. The version of Stripping has to match in Stripping and DaVinci.

Actions after stripping selections deadline

After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings.

Run retention tests (Grid)

Setup DaVinci from the nightlies for the branch (e.g. 2017-patches) and the build (e.g. Tue for Tuesday) do

cd <wherever>
lb-dev --nightly <nightly slot> DaVinci/<branch>
This will create the DaVinci branch bound to the nightly slot

cd DaVinciDev_<branch>
git lb-use Stripping
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSelections
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSettings
git lb-checkout Stripping/Stripping<XY> Phys/StrippingArchive
git lb-checkout Stripping/Stripping<XY> Phys/CommonParticlesArchive
make configure && make
Note that Stripping is the branch here, not the stripping version!

In<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py you might want to change LocalFile to DiracFile and check the platform.

Start ganga and for example for S29 on test data Reco16_Run18259

execfile('<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py')
j  = make_stripping_job('S29-loose', '<wherever>/DaVinciDev_HEAD/', '<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/data/Reco16_Run182594.py', nfiles = -1, reportfreq = 50000, evtmax = 1000000, dirac = True, Version = 'Stripping29', SplitStreamsByWG = True)
j.submit() 

to run the retention tests. ./Phys/StrippingSelections/tests/coordinators/StrippingReport.py -d DIRECTORY=searches for all strippingreport.py’s and adds them together. The retention table for the twiki can be obtained via =./Phys/StrippingSelections/tests/coordinators/StrippingReport.py -d DIRECTORY -r --hlt2rate RATE > retentions.txt, which uses a single output file at the moment.

Run retention tests (locally)

Run

gaudirun.py -T $STRIPPINGSELECTIONSROOT/tests/coordinators/retentions/TestFromSettings_all.py path_to_input_data 

to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script. You need to change the WG and Stripping, and possibly year and tags. If you want to run a smaller test of e.g. 100k, set EvtMax accordingly. It is advisable to create separate folders for each WG to avoid that the output DSTs get overwritten (you will need them later).

It is advisable to run the tests in a tmux session by setting the following alias alias ptmux="pagsh.krb -c 'kinit && tmux'" first. This way your kerberos token will not expire afte a couple of hours.

FYI SL corresponds to Semileptonic, Calib to ALL and BnoC to Charmless.

Run memory use tests

To test the global memory consumption, a single, reasonably long (100k-200k events) stripping job is needed (see step above). Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory use as a function of event number.

  python(2) analyseDaVinciLog.py logfile.txt -p name_of_your_plot.png
  display name_of_your_plot.png

The options -c and -l are currently not working.

To run safely in the production, jobs don't have to take more than 4 Gb of memory. Make sure there are no big jumps in memory use after ~50-100k events. They can indicate memory leaks. In case of doubt, use valgrind to find memory leaks. Memory auditor can also help to find an algorithm with memory leak. To enable it, add the following to the DaVinci options:

  ApplicationMgr().ExtSvc += [ 'AuditorSvc' ]
  AuditorSvc().Auditors += [ 'MemoryAuditor' ]

Run bandwidth tests

The total bandwidth for the stripping should be agreed with the bookkeeping manager and the PPG. For S28, the full Stripping had around 150 Mb/s. The maximum bandwidth of the incremental restripping should be "small" compared to it (not more than 20% of the full stripping).

Bandwidth and timing tests are run on the test sample. Scripts are provided (files ganga.py, submit_jobs.sh). Since large number of events is required for reliable calculation, the test jobs are submitted to the GRID (use submit_jobs.py after setting up Ganga environment).

Script retentions.py or retentions_new.py can be used after the test jobs finish to calculate the retentions and timing of individual lines, streams and WGs and generate tables in the twiki format. One needs to edit its header to change the job IDs of the test jobs (or local folder where the logfiles are stored), DaVinci version and, possibly, the trigger rate. The streamEventSize has to be changed each time you made a change to the Stripping. To get the numbers for the streamEventSize, do

ls -ltr path_to_output_dsts

and add these numbers to the correct streams of the corresponding WG. FTAG is called MDST in retentions_new.py.

The trigger rate is obtained as follows:

  • Run the follwing script to get the summary.xml (in this example for 2015).
   from Configurables import DaVinci
   from Gaudi.Configuration import importOptions

   DaVinci().DataType = '2015'
   DaVinci().Lumi = True
   DaVinci().EvtMax = 100000

   from Configurables import CondDB
   CondDB().LatestGlobalTagByDataType = "2015"

   from Configurables import LHCbApp
   LHCbApp().XMLSummary = "summary.xml"

   importOptions("$STRIPPINGSELECTIONSROOT/tests/data/Reco15a_Run164668.py")
  • In the summary.xml file, get the total number of events.
  • For the run you've ran over e.g. Run164668 in the example above, get the trigger rate from:
    dirac-bookkeeping-get-stats --BK /LHCb/Collision15//RealData//RAW --Run 164668 --Trigger
    (here: 11939.9 events/second).
  • The trigger rate can also be calculated from the number of events divided by the duration of the run. Here: 43'079'080 events/3600s, which is almost the same as the trigger rate above.
  • Don't bother about events/s and Hz.
  • Sometimes people ask for the number of stable beam seconds corresponding to the (sub)data set you ran over: Just divide the number of events you ran over by the trigger rate (here: 8.38s).

After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists.It is very important that all WGs check that the lines they need are included, and report back if anything is missing.

Known lines with higher retentions than usually allowed are (justified):

Run line independence tests

These should be run once the stripping selections are frozen. Assuming that the lines for 2011 and 2012 data are the same as each other, only one year needs to be checked. The scripts to run these tests are attached to this TWiki page. (The TWiki software has appended ".txt" to all of these filenames for some reason, this should be removed obviously)

The DaVinci scripts DV-Stripping20r0p3-Indep.py and DV-Stripping20r0p3-Indep-all.py run the stripping and save the stripping decisions in a ROOT NTuple (these scripts are set up to run S20r0p3, but changing to other strippings should be straightforward). The Indep-all.py runs all of the lines in a particular stripping, while Indep.py runs only a single line. Of course each script must be run over the same input events.

The ganga script ganga_all.py runs DV-Stripping20r0p3-Indep-all.py in ganga. This only needs to be run once, producing a single ganga job. The ganga script ganga_line.py runs DV-Stripping20r0p3-Indep.py in ganga, for a certain stripping line. This needs to be run as many times as there are lines in the given stripping. This produces many ganga jobs. To run these many jobs, there is a small script called submit_jobs.py. This reads in the various line names from a plain text file, which can be produced by running a short test of DV-Stripping20r0p3-Indep-all.py locally. Running submit_jobs.py to submit a lot of ganga jobs can take many hours.

Once the NTuples have been created, a script called check_tuples.py is used to run over the various single-line NTuples, and compare the stripping decisions with the NTuple from the whole-stripping job. The script can deal with incomplete or failed subjobs by skipping over the missing data. The only part of check_tuples.py that should need changing is the job numbers for the ganga jobs, and the location of the gangadir directory holding the output of the jobs.

If there is a mismatch between the stripping decision of a given line for a certain event, you will see a mesage like this in the stdout of check_tuples.py:

Checking line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision (job 724)subjob 0 checked, 48501 events, 37 selected Stripping decision is different: line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision, event 29990 (job 724, subjob 1): all=0, single=1

This indicates that this line gave a positive decision on this event when the lines was run on its own, but a negative decision when ran as part of the whole stripping. Such mismatches can be caused by a known feature of the way that the stripping deals with events where the maximum number of allowed combinations is reached in an algorithm such as CombineParticles or SubPIDMMFilter (details of this can be found in the Stripping Coordinator Report here:https://indico.cern.ch/event/201535/session/3/contribution/29/material/slides/1.pdf). This feature has a tendency to affect lines in the Beauty2Charm module, but occasionally affects other lines as well. To check whether this feature is the cause of a mismatch, grep the log files of the single-line stripping job for "Maximum number of allowed particles reached". You are looking for a line like:

D+2HHHSubPIDSelBeauty2Charm WARNING SubPIDMMFilter:: Maximum number of allowed particles reached

If one (or more) of these warnings is found at an event number matching that given in the output of check_tuples.py, then this feature is to blame and (as this is a known issue) no action needs to be taken. If no such warning is found, then there is some other, new issue with this line that requires further investigation.

Debugging with gdb

If there's an issue with the stripping that's not easy to work out, eg, if a line is hanging for some reason, gdb can be useful. To run the stripping with gdb use, eg

lb-run -c x86_64-slc6-gcc49-dbg DaVinci/<version> sh -c 'gdb --args python $(which gaudirun.py) <options>'

See also here. You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with

kill -2 <pid>

from another terminal it immediately continued.

Once gdb has started, turn on logging with

set logging on

then start the job with run. You can then interrupt whenever you like with ctrl-c, or if you're investigating a crash you can just wait for it, then you can use backtrace to get the method call backtrace. Use continue to continue the job after an interrupt.

Once you have the backtrace you can print info on an object given its address with, eg

print *(class CombineParticles*)0x614b8940

or

print *('DaVinci::N4BodyDecays' *) 0x5dc58cf0

(note the single quotes for objects in a namespace) then you can search the output, of which there will likely be a lot. If you're trying to work out which line a problematic object originates from, search the output for anything containing 'Line' or 'Stripping'. Then you can have a look at the module that the line belongs to.

If you're trying to catch some output, you can set a break point for any output to console with

b write if 1==$rdi

or

b fwrite if $rcx==&_IO_2_1_stdout_

Release Stripping project

Help on release procedure is at PrepareProjectReleaseGit page. Please check the steps following the instructions on ProdReqPreLaunchTest. For full DST productions, no ancestors need to be included.

If the above instructions fail for RDST files, remove the RAW from the generated ProdConf*.py and run with the other options locally.

Check nightlies and correct if needed

Nightlies page is here. Nightlies have to be checked not only before the release of Stripping, but also from time to time during the development of the stripping selections by the analysts to spot problems early. In case of problems, inform the responsible person. In some cases, the nightly tests failures are expected because of changes introduced, and updating the test reference is needed to fix them.

Tests run by the nightlies are defined by the files under

    $PACKAGEROOT/tests/qmtest/package.qms/*.qmt

Scripts to be run live under $PACKAGEROOT/tests/ and references, if needed, are under $PACKAGEROOT/tests/refs.

To run the test of a package, after building do:

   make test

Under cmt, you could run a single test, assuming the tests are in a directory davinci.qms and the test is called mytest.qmt, with:

   cmt TestPackage davinci.mytest

but I'm not sure how to do this under cmake.

To update the test reference (after running the test):

   cd tests/refs
   mv ***.ref.new ***.ref

If the list of trigger lines has to be updated, you can do lb-run Moore/latest Phys/StrippingSelections/tests/coordinators/get_hlt_lines.py --tck 0x11431621 --outputfile 0x11431621.py. At some point during Run 2, the following notation was introduced: 0x1 are HLT1 TCKs, 0x2 are HLT2, and 0x5 and 0x6 are for MC.

Prepare StrippingArchive, CommonParticlesArchive and StrippingSettings database

StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections, which are used only for development. CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified and lives in Phys now.

Get DaVinci from the nightlies eg. for the branch (e.g. 2017-patches) and the build (e.g. Tue for Tuesday) do (See also the setup instructions in "Run retention tests (Grid)")

cd <wherever>
lb-dev --nightly-cvmfs --nightly <yourbranch> <day> DaVinci/HEAD
cd DaVinciDev_HEAD
git lb-use Stripping
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSelections
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSettings
git lb-checkout Stripping/Stripping<XY> Phys/StrippingArchive
git lb-checkout Stripping/Stripping<XY> Phys/CommonParticlesArchive
make configure && make

./run \$STRIPPINGSELECTIONSROOT/tests/coordinators/create_archives.py Stripping<XY> "<Description>"

assuming that the changes you want are in a branch called Stripping. The latter command creates or updates StrippingArchive, CommonParticlesArchive and if necessary the StrippingSettings dbase. Use git status to see what has been made.

The dbase file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory:

git clone ssh://git@gitlab.cern.ch:7999/lhcb/Stripping.git
cd Stripping
git checkout <yourbranch>
cd Stripping/Phys/StrippingSettings/dbase/
cp from/the/path/where/it/is/strippingXX .
git add strippingXX
git commit strippingXX -m "update dbase for sXX"
git push

Check lost tags in tag collector

Tag collector page is here. Go to "Lost tags" and add them if needed.

Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)

This section describes how to update the reference files used for the nightly tests.

Clone DaVinci like this

git clone ssh://git@gitlab.cern.ch:7999/lhcb/Project.git
cd Project
lb-project-init
# You may have to use a different branch of DV depending on the nightly test you are looking at eg. lhcb-2016-patches
git checkout -b xxx
make

OR

lb-dev --nightly lhcb-2017-patches DaVinci/2017-patches
cd ./DaVinciDev_2017-patches
git lb-use DaVinci
git lb-checkout DaVinci/2017-patches DaVinciTests
make
For more info on lb- commands see here.

This will grab the same configuration as the nightly slot called lhcb-2017-patches . A slot is a consistent set of inter-related software projects that should be built and tested together. The configuration of a slot includes, in addition to the list of configured projects, the list of platforms the slot should be built for and some extra metadata for the fine tuning of the build. Projects in 2017-patches for example check out DaVinci branch 2017-patches plus all non-WIP merge requests targeting the 2017-patches branch.

For more info on the nightlies see here

(See here and here for even more on lhcb software development)

If required also get the Phys package the same way

git lb-use Phys
git lb-checkout DaVinci/2017-patches Phys

First, update the stripping list in Phys/StrippingCache/CMakeLists.txt where XXrY is the stripping version ans xx the data year

set(extraopts_XXrY
${AppConfig_DIR}/options/DaVinci/DataType-20xx.py
)
and put the stripping version in the strip list below

and add options under Phys/StrippingCache/options

DV-StrippingXXrY-Stripping.py

To check which functors has not been cached, do

export LOKI_DISABLE_PYTHON=1

before testing. This is already included in $DAVINCITESTSROOT/tests/options/DisableLoKiPythonFunctors.py. Just make sure this option file is included in the qmtest, which you need to create under DaVinciTests/tests/qmtest/stripping.qms/.

You might need to create an empty file for the reference, otherwise it won't be created.

To test, do

make test ARGS='-N'
make test ARGS='-R DaVinciTests.stripping.test_strippingXXrY_collisionxx_recoyy'

If the test results differ from the reference this will create the file DaVinciTests/tests/refs/.ref.new, which you rename without the .new and have to commit. The test results are in build/Testing/Temporary/LastTest_*.log. At the end of the file it will display the differences between this test and the reference. This should match what is seen in the "Output Diff" of the failed nightly test

Tricks: The tests under DaVinci/DaVinciTests/tests/qmtest/stripping.qms/ need to included in StrippingCache/CMakeList.txt. The nightlies configuration can be found on gitlab, e.g. here. In the nightlies, always the latest version of TMVAWeights is used, which is why v999 appears in the nightlies tests.

To commit:

git add <reference file>
git commit -m 'new reference file for ....'
git lb-push DaVinci <name-of-new-branch>
You can then make a merge request for your new branch into whichever branch of DaVinci you are working on. Check if the nightly passed the next day smile

Tag packages in Stripping project

Go through all the packages in Stripping and look at the release.notes. If there’re any updates since the last release then you add in the tag header with the incremented version. Increment the version in CMakeLists. For StrippingSys: The release notes should be the collection of the release notes for all packages that have changed since the last release. Add the tag header and increment the version in CMakeLists.txt. In the top directory of Stripping, set the version of Stripping in CMakeLists.txt and the version of Phys on which it depends in CMakeLists.txt. Push the changes, go to the Stripping gitlab page, click on "tags" and then "new tag". Name the tag, give a short message for the tag, select the branch from which the tag should be made, and copy the release notes of StrippingSys into the release notes of the tag. In case you need to re-tag, go to the "tags" page and click "delete tag" on which ever tag you want to replace, then remake the tag as above.

To check if a tag exists, you can grep the CMakeLists.txt, e.g. grep v4r19 $LHCb_release_area/STRIPPING/STRIPPING_v*/Phys/StrippingSelections/CMakeLists.txt or use cmt/requirements if you're looking at older versions.

Tag packages in DaVinci project

The only package that needs to be tagged by the Stripping coordinators is the StrippingCache which lives in DaVinci. Follow the same steps as for the previous step.

Tag extra packages in PARAM

Packages like TMVAWeights required by some RelatedInfoTools live in PARAM and might also need to be tagged.

getpack TMVAWeights h
<modify release notes and update versions wherever required>
svn ci -m "releasing new version vXrY"
tag_package TMVAWeights vXrY

Then open a new JIRA task in LHCBDEP asking for deployment.

Add Stripping project to lhcb-prerelease slot in the nightlies

Go to nightlies page, open "Configuration Editor", choose the slot needed (most likely lhcb-prerelease), double-click on Stripping tag and change the STRIPPING version there. Make sure "disabled" flag is "false".

If you do not want to wait for the nightlies, follow the instructions here ProjectRelease#For_Release_Managers

Prepare AppConfig options

The stripping options are in AppConfig/options/DaVinci/DV-StrippingXX-Stripping.py. In most cases the DaVinci options for stripping can be copied from the previous round, with minor changes:

  • Change stripping version used to retrieve configuration from StrippingSettings and StrippingArchive.
  • Stripping TCK has to be changed. Current convention is the following: TCK = 0xVVVVSSSS, where VVVV is the DaVinci version, and SSSS is the Stripping number (e.g. for Stripping20r1p1 running on DaVinci v32r2p5 TCK = 0x32252011).
  • If necessary create DataType -<yyyy>.py, where <yyyy> corresponds to the year.
If the stripping configuration is expected to run in MC, the corresponding MC stripping options (DV-StrippingXX-Stripping-MC*.py) have to be updated. Please note that AppConfig still lives in svn.

In the case that only a patch of AppConfig needs to be deployed, prepare the tag as as usual: set the version number in cmt/requirements, and add in the release header in doc/release.notes, and commit. Then tag the package as usual with tag_package AppConfig vXrY, at which point it should appear here. No build is required as AppConfig only contains options files, so you can request deployment in JIRA, an example request is here. That's all!

Trigger a release build of DaVinci

For minor changes, eg. changing the Stripping version picked up by DaVinci, we are allowed to create a tag for DaVinci, which follows the same procedure as the Stripping. However, we cannot delete a tag from DaVinci if we mess up! After creating the Stripping and DaVinci tag,

> lb-sdb-import DaVinci vXrY

triggers a release build, which you should check here. If the builds are ok, you can go ahead and request deployment of DaVinci citing the number of the release build (see below). Please note that it can take around 30min before the release build appears on the website.

Ask for release in Jira

Ask Stripping, DaVinci and AppConfig to be released using JIRA. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).

Prepare production steps

Go to https://lhcb-portal-dirac.cern.ch/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert (currently Marco Cattaneo and Liang Sun). Test the steps before marking them as "Ready".

To test the steps, you should follow the instructions given here. You might need to test different versions of LHCbDirac until you find a working one. The test will create a Local_* folder, in which you find prodConf_*.py. You should remove the raw file from the list of inputs, potentially edit the number of events, and depending on where you test prodConf_*.py, add the full path to the xml catalogue.

Run validation

Announce the steps on the operations elog, and send an email to the data managers and production managers asking for a validation giving a suitable run range. The typical size of a validation comprises a couple thousand files.

Run production

Debugging tricks for problems in production

To see all algorithms running, add

from Configurables import NameAuditor, AuditorSvc
AuditorSvc ().Auditors.append(NameAuditor ())

to your options and redirect the output to tmp as the logfile might become several GB large.

The timings of each algorithms for each event can be written by adding

from Configurables import TimingAuditor, AuditorSvc, ChronoStatSvc, SequencerTimerTool, ChronoAuditor
from Gaudi.Configuration import INFO

TimingAuditor ().addTool(SequencerTimerTool,name="TIMER")
TimingAuditor ().TIMER.NameSize = 60
TimingAuditor ().OutputLevel = INFO
AuditorSvc ().Auditors.append(ChronoAuditor ("Chrono"))
ChronoStatSvc ().PerEventFile = 'eventtimings.txt' 

to your options (copy the Stripping options from AppConfog and replace the existing TimingAuditor with the lines above). In 'eventtimings.txt', all algorithms are listed with the respective timings. Please keep in mind that not all algorithms are being run for each event.

Prepare web documentation

Reference documentation about all lines (their cuts, sequence of algorithms, streams, output locations) and standard particles is available at the Stripping project webpages. These pages are generated automatically using the HltGenConfig algorithm (the same one that is used to generate HltTCK). Here are the instructions on generating these pages:

  • Prepare build environment for DaVinci version used in stripping production:
        > lb-dev DaVinci vVVrRpP 
        > cd DaVinciDev_vVVrRpP 

  • For DaVinci versions prior to v36r0, it was necessary to getpack HltDAQ package, copy HltGenConfig algorithms from Hlt/HltCommon to HltDAQ and build it:
            > getpack Hlt/HltDAQ
            > SetupProject DaVinci vVVrRpP
            > cd Hlt/HltDAQ/cmt
            > cp $LHCBRELEASES/HLT/HLT_v20r1p1/Hlt/HltCommon/src/HltGenConfig.* ../src/
            > cmt make
       
    These steps are NOT needed for new DaVinci (v36r0 and more recent) because HltGenConfig became available in DaVinci.

  • Create an empty directory in a place with plenty of space
  • Build and set the environment variables:
        > make
        > ./run bash
        > mkdir strippingXX
        > cd strippingXX
        > cp $STRIPPINGSELECTIONSROOT/tests/coordinators/documentation/* .

  • Edit run.sh, make_html.sh and tck.py to use the correct AppConfig options (found in the production steps), DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation). Look up the TCK in
    $APPCONFIGOPTS/DaVinci/DV-StrippingXX-Stripping.py
    or just concatenate the DV and Stripping versions, e.g. for DV v38r1p4 and Stripping 24r0p1, this results in 0x38142401.

  • Run the ./run.sh script which will run 10000 events (to make sure most on-demand services are initialised) and produce the stripping config in the file config.tar
        > ./run.sh

  • Run ./make_html.sh script which will create web documantation in the "html" folder using config.tar (note that utils.py module is required by this script)
        > ./make_html.sh

  • cp -r html/strippingXX $LHCBDOC/stripping/config/ and edit $LHCBDOC/stripping/maindesc.html.
Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt DV-Stripping20r0p3-Indep-all.py.txt r1 manage 5.2 K 2014-08-21 - 18:51 LaurenceCarson Scripts to run Independence Tests
Texttxt DV-Stripping20r0p3-Indep.py.txt r1 manage 5.4 K 2014-08-21 - 18:51 LaurenceCarson Scripts to run Independence Tests
Texttxt check_tuples.py.txt r1 manage 2.1 K 2014-08-21 - 18:51 LaurenceCarson Scripts to run Independence Tests
Texttxt ganga_all.py.txt r1 manage 0.8 K 2014-08-21 - 18:51 LaurenceCarson Scripts to run Independence Tests
Texttxt ganga_line.py.txt r1 manage 0.9 K 2014-08-21 - 18:51 LaurenceCarson Scripts to run Independence Tests
Texttxt submit_jobs.py.txt r1 manage 0.2 K 2014-08-21 - 18:51 LaurenceCarson Scripts to run Independence Tests
Edit | Attach | Watch | Print version | History: r88 < r87 < r86 < r85 < r84 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r88 - 2018-11-01 - NicolaAnneSkidmore
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback