Stripping coordinator's cheatsheet

In<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py you might want to change LocalFile to DiracFile and check the platform.

Start ganga and for example for S29 on test data Reco16_Run18259

execfile('<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py')
j  = make_stripping_job('S29-loose', '<wherever>/DaVinciDev_HEAD/', '<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/data/Reco16_Run182594.py', nfiles = -1, reportfreq = 50000, evtmax = 1000000, dirac = True, Version = 'Stripping29', SplitStreamsByWG = True)
j.submit() 

to run the retention tests. ./Phys/StrippingSelections/tests/coordinators/StrippingReport.py -d DIRECTORY=searches for all strippingreport.py’s and adds them together. The retention table for the twiki can be obtained via =./Phys/StrippingSelections/tests/coordinators/StrippingReport.py -d DIRECTORY -r --hlt2rate RATE > retentions.txt, which uses a single output file at the moment.

Run retention tests (locally)

Run

gaudirun.py -T $STRIPPINGSELECTIONSROOT/tests/coordinators/retentions/TestFromSettings_all.py path_to_input_data 

to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script. You need to change the WG and Stripping, and possibly year and tags. If you want to run a smaller test of e.g. 100k, set EvtMax accordingly. It is advisable to create separate folders for each WG to avoid that the output DSTs get overwritten (you will need them later).

It is advisable to run the tests in a tmux session by setting the following alias alias ptmux="pagsh.krb -c 'kinit && tmux'" first. This way your kerberos token will not expire afte a couple of hours.

FYI SL corresponds to Semileptonic, Calib to ALL and BnoC to Charmless.

Run memory use tests

To test the global memory consumption, a single, reasonably long (100k-200k events) stripping job is needed (see step above). Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory use as a function of event number.

  python(2) analyseDaVinciLog.py logfile.txt -p name_of_your_plot.png
  display name_of_your_plot.png

The options -c and -l are currently not working.

To run safely in the production, jobs don't have to take more than 4 Gb of memory. Make sure there are no big jumps in memory use after ~50-100k events. They can indicate memory leaks. In case of doubt, use valgrind to find memory leaks. Memory auditor can also help to find an algorithm with memory leak. To enable it, add the following to the DaVinci options:

  ApplicationMgr().ExtSvc += [ 'AuditorSvc' ]
  AuditorSvc().Auditors += [ 'MemoryAuditor' ]

Run bandwidth tests

The total bandwidth for the stripping should be agreed with the bookkeeping manager and the PPG. For S28, the full Stripping had around 150 Mb/s. The maximum bandwidth of the incremental restripping should be "small" compared to it (not more than 20% of the full stripping).

Bandwidth and timing tests are run on the test sample. Scripts are provided (files ganga.py, submit_jobs.sh). Since large number of events is required for reliable calculation, the test jobs are submitted to the GRID (use submit_jobs.py after setting up Ganga environment).

Script retentions.py or retentions_new.py can be used after the test jobs finish to calculate the retentions and timing of individual lines, streams and WGs and generate tables in the twiki format. One needs to edit its header to change the job IDs of the test jobs (or local folder where the logfiles are stored), DaVinci version and, possibly, the trigger rate. The streamEventSize has to be changed each time you made a change to the Stripping. To get the numbers for the streamEventSize, do

ls -ltr path_to_output_dsts

and add these numbers to the correct streams of the corresponding WG. FTAG is called MDST in retentions_new.py.

The trigger rate is obtained as follows:

• Run the follwing script to get the summary.xml (in this example for 2015).

   from Configurables import DaVinci
   from Gaudi.Configuration import importOptions

   DaVinci().DataType = '2015'
   DaVinci().Lumi = True
   DaVinci().EvtMax = 100000

   from Configurables import CondDB
   CondDB().LatestGlobalTagByDataType = "2015"

   from Configurables import LHCbApp
   LHCbApp().XMLSummary = "summary.xml"

   importOptions("$STRIPPINGSELECTIONSROOT/tests/data/Reco15a_Run164668.py")

• In the summary.xml file, get the total number of events.
• For the run you've ran over e.g. Run164668 in the example above, get the trigger rate from:

  dirac-bookkeeping-get-stats --BK /LHCb/Collision15//RealData//RAW --Run 164668 --Trigger

  (here: 11939.9 events/second).
• The trigger rate can also be calculated from the number of events divided by the duration of the run. Here: 43'079'080 events/3600s, which is almost the same as the trigger rate above.
• Don't bother about events/s and Hz.
• Sometimes people ask for the number of stable beam seconds corresponding to the (sub)data set you ran over: Just divide the number of events you ran over by the trigger rate (here: 8.38s).

After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists.It is very important that all WGs check that the lines they need are included, and report back if anything is missing.

Known lines with higher retentions than usually allowed are (justified):

• StrippingTriggerTestLine (RD)
• StrippingFullDSTDiMuon (BandQ, shared with B2CC)
• StrippingWMu, StrippingWe, StrippingHlt2PassThroughLine and StrippingHltQEEJets (QEE)

Run line independence tests

These should be run once the stripping selections are frozen. Assuming that the lines for 2011 and 2012 data are the same as each other, only one year needs to be checked. The scripts to run these tests are attached to this TWiki page. (The TWiki software has appended ".txt" to all of these filenames for some reason, this should be removed obviously)

The DaVinci scripts DV-Stripping20r0p3-Indep.py and DV-Stripping20r0p3-Indep-all.py run the stripping and save the stripping decisions in a ROOT NTuple (these scripts are set up to run S20r0p3, but changing to other strippings should be straightforward). The Indep-all.py runs all of the lines in a particular stripping, while Indep.py runs only a single line. Of course each script must be run over the same input events.

The ganga script ganga_all.py runs DV-Stripping20r0p3-Indep-all.py in ganga. This only needs to be run once, producing a single ganga job. The ganga script ganga_line.py runs DV-Stripping20r0p3-Indep.py in ganga, for a certain stripping line. This needs to be run as many times as there are lines in the given stripping. This produces many ganga jobs. To run these many jobs, there is a small script called submit_jobs.py. This reads in the various line names from a plain text file, which can be produced by running a short test of DV-Stripping20r0p3-Indep-all.py locally. Running submit_jobs.py to submit a lot of ganga jobs can take many hours.

Once the NTuples have been created, a script called check_tuples.py is used to run over the various single-line NTuples, and compare the stripping decisions with the NTuple from the whole-stripping job. The script can deal with incomplete or failed subjobs by skipping over the missing data. The only part of check_tuples.py that should need changing is the job numbers for the ganga jobs, and the location of the gangadir directory holding the output of the jobs.

If there is a mismatch between the stripping decision of a given line for a certain event, you will see a mesage like this in the stdout of check_tuples.py:

Checking line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision (job 724)subjob 0 checked, 48501 events, 37 selected Stripping decision is different: line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision, event 29990 (job 724, subjob 1): all=0, single=1

This indicates that this line gave a positive decision on this event when the lines was run on its own, but a negative decision when ran as part of the whole stripping. Such mismatches can be caused by a known feature of the way that the stripping deals with events where the maximum number of allowed combinations is reached in an algorithm such as CombineParticles or SubPIDMMFilter (details of this can be found in the Stripping Coordinator Report here:https://indico.cern.ch/event/201535/session/3/contribution/29/material/slides/1.pdf). This feature has a tendency to affect lines in the Beauty2Charm module, but occasionally affects other lines as well. To check whether this feature is the cause of a mismatch, grep the log files of the single-line stripping job for "Maximum number of allowed particles reached". You are looking for a line like:

D+2HHHSubPIDSelBeauty2Charm WARNING SubPIDMMFilter:: Maximum number of allowed particles reached

If one (or more) of these warnings is found at an event number matching that given in the output of check_tuples.py, then this feature is to blame and (as this is a known issue) no action needs to be taken. If no such warning is found, then there is some other, new issue with this line that requires further investigation.

Debugging with gdb

If there's an issue with the stripping that's not easy to work out, eg, if a line is hanging for some reason, gdb can be useful. To run the stripping with gdb use, eg

lb-run -c x86_64-slc6-gcc49-dbg DaVinci/<version> sh -c 'gdb --args python $(which gaudirun.py) <options>'

See also here. You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with

kill -2 <pid>

from another terminal it immediately continued.

Once gdb has started, turn on logging with

set logging on

then start the job with run. You can then interrupt whenever you like with ctrl-c, or if you're investigating a crash you can just wait for it, then you can use backtrace to get the method call backtrace. Use continue to continue the job after an interrupt.

Once you have the backtrace you can print info on an object given its address with, eg

print *(class CombineParticles*)0x614b8940

or

print *('DaVinci::N4BodyDecays' *) 0x5dc58cf0

(note the single quotes for objects in a namespace) then you can search the output, of which there will likely be a lot. If you're trying to work out which line a problematic object originates from, search the output for anything containing 'Line' or 'Stripping'. Then you can have a look at the module that the line belongs to.

If you're trying to catch some output, you can set a break point for any output to console with

b write if 1==$rdi

or

b fwrite if $rcx==&_IO_2_1_stdout_

Release Stripping project

Help on release procedure is at PrepareProjectReleaseGit page. Please check the steps following the instructions on ProdReqPreLaunchTest. For full DST productions, no ancestors need to be included.

If the above instructions fail for RDST files, remove the RAW from the generated ProdConf*.py and run with the other options locally.

Check nightlies and correct if needed

Nightlies page is here. Nightlies have to be checked not only before the release of Stripping, but also from time to time during the development of the stripping selections by the analysts to spot problems early. In case of problems, inform the responsible person. In some cases, the nightly tests failures are expected because of changes introduced, and updating the test reference is needed to fix them.

Tests run by the nightlies are defined by the files under

    $PACKAGEROOT/tests/qmtest/package.qms/*.qmt

Scripts to be run live under $PACKAGEROOT/tests/ and references, if needed, are under $PACKAGEROOT/tests/refs.

To run the test of a package, after building do:

   make test

Under cmt, you could run a single test, assuming the tests are in a directory davinci.qms and the test is called mytest.qmt, with:

   cmt TestPackage davinci.mytest

but I'm not sure how to do this under cmake.

To update the test reference (after running the test):

   cd tests/refs
   mv ***.ref.new ***.ref

If the list of trigger lines has to be updated, you can do lb-run Moore/latest Phys/StrippingSelections/tests/coordinators/get_hlt_lines.py --tck 0x11431621 --outputfile 0x11431621.py. At some point during Run 2, the following notation was introduced: 0x1 are HLT1 TCKs, 0x2 are HLT2, and 0x5 and 0x6 are for MC.

Prepare StrippingArchive, CommonParticlesArchive and StrippingSettings database

StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections, which are used only for development. CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified and lives in Phys now.

Get DaVinci from the nightlies eg. for the branch (e.g. 2017-patches) and the build (e.g. Tue for Tuesday) do (See also the setup instructions in "Run retention tests (Grid)")

cd <wherever>
lb-dev --nightly-cvmfs --nightly <yourbranch> <day> DaVinci/HEAD
cd DaVinciDev_HEAD
git lb-use Stripping
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSelections
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSettings
git lb-checkout Stripping/Stripping<XY> Phys/StrippingArchive
git lb-checkout Stripping/Stripping<XY> Phys/CommonParticlesArchive
make configure && make

./run \$STRIPPINGSELECTIONSROOT/tests/coordinators/create_archives.py Stripping<XY> "<Description>"

assuming that the changes you want are in a branch called Stripping. The latter command creates or updates StrippingArchive, CommonParticlesArchive and if necessary the StrippingSettings dbase. Use git status to see what has been made.

The dbase file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory:

git clone ssh://git@gitlab.cern.ch:7999/lhcb/Stripping.git
cd Stripping
git checkout <yourbranch>
cd Stripping/Phys/StrippingSettings/dbase/
cp from/the/path/where/it/is/strippingXX .
git add strippingXX
git commit strippingXX -m "update dbase for sXX"
git push

Check lost tags in tag collector

Tag collector page is here. Go to "Lost tags" and add them if needed.

Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)

This section describes how to update the reference files used for the nightly tests.

Clone DaVinci like this

git clone ssh://git@gitlab.cern.ch:7999/lhcb/Project.git
cd Project
lb-project-init
# You may have to use a different branch of DV depending on the nightly test you are looking at eg. lhcb-2016-patches
git checkout -b xxx
make

OR

lb-dev --nightly lhcb-2017-patches DaVinci/2017-patches
cd ./DaVinciDev_2017-patches
git lb-use DaVinci
git lb-checkout DaVinci/2017-patches DaVinciTests
make

This will grab the same configuration as the nightly slot called lhcb-2017-patches . A slot is a consistent set of inter-related software projects that should be built and tested together. The configuration of a slot includes, in addition to the list of configured projects, the list of platforms the slot should be built for and some extra metadata for the fine tuning of the build. Projects in 2017-patches for example check out DaVinci branch 2017-patches plus all non-WIP merge requests targeting the 2017-patches branch.

For more info on the nightlies see here

(See here and here for even more on lhcb software development)

If required also get the Phys package the same way

git lb-use Phys
git lb-checkout DaVinci/2017-patches Phys

First, update the stripping list in Phys/StrippingCache/CMakeLists.txt where XXrY is the stripping version ans xx the data year

set(extraopts_XXrY
${AppConfig_DIR}/options/DaVinci/DataType-20xx.py
)

and add options under Phys/StrippingCache/options

DV-StrippingXXrY-Stripping.py

To check which functors has not been cached, do

export LOKI_DISABLE_PYTHON=1

before testing. This is already included in $DAVINCITESTSROOT/tests/options/DisableLoKiPythonFunctors.py. Just make sure this option file is included in the qmtest, which you need to create under DaVinciTests/tests/qmtest/stripping.qms/.

You might need to create an empty file for the reference, otherwise it won't be created.

To test, do

make test ARGS='-N'
make test ARGS='-R DaVinciTests.stripping.test_strippingXXrY_collisionxx_recoyy'

If the test results differ from the reference this will create the file DaVinciTests/tests/refs/.ref.new, which you rename without the .new and have to commit. The test results are in build/Testing/Temporary/LastTest_*.log. At the end of the file it will display the differences between this test and the reference. This should match what is seen in the "Output Diff" of the failed nightly test

Tricks: The tests under DaVinci/DaVinciTests/tests/qmtest/stripping.qms/ need to included in StrippingCache/CMakeList.txt. The nightlies configuration can be found on gitlab, e.g. here. In the nightlies, always the latest version of TMVAWeights is used, which is why v999 appears in the nightlies tests.

To commit:

git add <reference file>
git commit -m 'new reference file for ....'
git lb-push DaVinci <name-of-new-branch>

Tag packages in Stripping project

Go through all the packages in Stripping and look at the release.notes. If there’re any updates since the last release then you add in the tag header with the incremented version. Increment the version in CMakeLists. For StrippingSys: The release notes should be the collection of the release notes for all packages that have changed since the last release. Add the tag header and increment the version in CMakeLists.txt. In the top directory of Stripping, set the version of Stripping in CMakeLists.txt and the version of Phys on which it depends in CMakeLists.txt. Push the changes, go to the Stripping gitlab page, click on "tags" and then "new tag". Name the tag, give a short message for the tag, select the branch from which the tag should be made, and copy the release notes of StrippingSys into the release notes of the tag. In case you need to re-tag, go to the "tags" page and click "delete tag" on which ever tag you want to replace, then remake the tag as above.

To check if a tag exists, you can grep the CMakeLists.txt, e.g. grep v4r19 $LHCb_release_area/STRIPPING/STRIPPING_v*/Phys/StrippingSelections/CMakeLists.txt or use cmt/requirements if you're looking at older versions.

Tag packages in DaVinci project

The only package that needs to be tagged by the Stripping coordinators is the StrippingCache which lives in DaVinci. Follow the same steps as for the previous step.

Tag extra packages in PARAM

Packages like TMVAWeights required by some RelatedInfoTools live in PARAM and might also need to be tagged.

getpack TMVAWeights h
<modify release notes and update versions wherever required>
svn ci -m "releasing new version vXrY"
tag_package TMVAWeights vXrY

Then open a new JIRA task in LHCBDEP asking for deployment.

Add Stripping project to lhcb-prerelease slot in the nightlies

Go to nightlies page, open "Configuration Editor", choose the slot needed (most likely lhcb-prerelease), double-click on Stripping tag and change the STRIPPING version there. Make sure "disabled" flag is "false".

If you do not want to wait for the nightlies, follow the instructions here ProjectRelease#For_Release_Managers

Prepare AppConfig options

The stripping options are in AppConfig/options/DaVinci/DV-StrippingXX-Stripping.py. In most cases the DaVinci options for stripping can be copied from the previous round, with minor changes:

• Change stripping version used to retrieve configuration from StrippingSettings and StrippingArchive.
• Stripping TCK has to be changed. Current convention is the following: TCK = 0xVVVVSSSS, where VVVV is the DaVinci version, and SSSS is the Stripping number (e.g. for Stripping20r1p1 running on DaVinci v32r2p5 TCK = 0x32252011).
• If necessary create DataType -<yyyy>.py, where <yyyy> corresponds to the year.

In the case that only a patch of AppConfig needs to be deployed, prepare the tag as as usual: set the version number in cmt/requirements, and add in the release header in doc/release.notes, and commit. Then tag the package as usual with tag_package AppConfig vXrY, at which point it should appear here. No build is required as AppConfig only contains options files, so you can request deployment in JIRA, an example request is here. That's all!

Trigger a release build of DaVinci

For minor changes, eg. changing the Stripping version picked up by DaVinci, we are allowed to create a tag for DaVinci, which follows the same procedure as the Stripping. However, we cannot delete a tag from DaVinci if we mess up! After creating the Stripping and DaVinci tag,

> lb-sdb-import DaVinci vXrY

triggers a release build, which you should check here. If the builds are ok, you can go ahead and request deployment of DaVinci citing the number of the release build (see below). Please note that it can take around 30min before the release build appears on the website.

Ask for release in Jira

Ask Stripping, DaVinci and AppConfig to be released using JIRA. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).

Prepare production steps

Go to https://lhcb-portal-dirac.cern.ch/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert (currently Marco Cattaneo and Liang Sun). Test the steps before marking them as "Ready".

To test the steps, you should follow the instructions given here. You might need to test different versions of LHCbDirac until you find a working one. The test will create a Local_* folder, in which you find prodConf_*.py. You should remove the raw file from the list of inputs, potentially edit the number of events, and depending on where you test prodConf_*.py, add the full path to the xml catalogue.

Run validation

Announce the steps on the operations elog, and send an email to the data managers and production managers asking for a validation giving a suitable run range. The typical size of a validation comprises a couple thousand files.

Run production

Debugging tricks for problems in production

To see all algorithms running, add

from Configurables import NameAuditor, AuditorSvc
AuditorSvc ().Auditors.append(NameAuditor ())

to your options and redirect the output to tmp as the logfile might become several GB large.

The timings of each algorithms for each event can be written by adding

from Configurables import TimingAuditor, AuditorSvc, ChronoStatSvc, SequencerTimerTool, ChronoAuditor
from Gaudi.Configuration import INFO

TimingAuditor ().addTool(SequencerTimerTool,name="TIMER")
TimingAuditor ().TIMER.NameSize = 60
TimingAuditor ().OutputLevel = INFO
AuditorSvc ().Auditors.append(ChronoAuditor ("Chrono"))
ChronoStatSvc ().PerEventFile = 'eventtimings.txt' 

to your options (copy the Stripping options from AppConfog and replace the existing TimingAuditor with the lines above). In 'eventtimings.txt', all algorithms are listed with the respective timings. Please keep in mind that not all algorithms are being run for each event.

Prepare web documentation

Reference documentation about all lines (their cuts, sequence of algorithms, streams, output locations) and standard particles is available at the Stripping project webpages. These pages are generated automatically using the HltGenConfig algorithm (the same one that is used to generate HltTCK). Here are the instructions on generating these pages:

• Prepare build environment for DaVinci version used in stripping production:

        > lb-dev DaVinci vVVrRpP 
        > cd DaVinciDev_vVVrRpP 

• For DaVinci versions prior to v36r0, it was necessary to getpack HltDAQ package, copy HltGenConfig algorithms from Hlt/HltCommon to HltDAQ and build it:

          > getpack Hlt/HltDAQ
          > SetupProject DaVinci vVVrRpP
          > cd Hlt/HltDAQ/cmt
          > cp $LHCBRELEASES/HLT/HLT_v20r1p1/Hlt/HltCommon/src/HltGenConfig.* ../src/
          > cmt make
     

  These steps are NOT needed for new DaVinci (v36r0 and more recent) because HltGenConfig became available in DaVinci.

• Create an empty directory in a place with plenty of space
• Build and set the environment variables:

        > make
        > ./run bash
        > mkdir strippingXX
        > cd strippingXX
        > cp $STRIPPINGSELECTIONSROOT/tests/coordinators/documentation/* .

• Edit run.sh, make_html.sh and tck.py to use the correct AppConfig options (found in the production steps), DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation). Look up the TCK in

  $APPCONFIGOPTS/DaVinci/DV-StrippingXX-Stripping.py

  or just concatenate the DV and Stripping versions, e.g. for DV v38r1p4 and Stripping 24r0p1, this results in 0x38142401.

• Run the ./run.sh script which will run 10000 events (to make sure most on-demand services are initialised) and produce the stripping config in the file config.tar

        > ./run.sh

• Run ./make_html.sh script which will create web documantation in the "html" folder using config.tar (note that utils.py module is required by this script)

        > ./make_html.sh

• cp -r html/strippingXX $LHCBDOC/stripping/config/ and edit $LHCBDOC/stripping/maindesc.html.