Stripping coordinator's cheatsheet
What is an rDST
When data is collected from the LHCb detector, the raw data will be transferred to the LHCb associated Tier 1 sites for the reconstruction to produce rDST files. The rDST files are used for stripping jobs where events are selected for physics analysis. Selected events are written to DST files and distributed in identical copies to all the Tier 1 sites. These files are then accessible for physics analysis by individual collaborators.
(Taken from
Streaming Task Force page)
What is a .xml file
Physical file names (PFN): these are targeted to a site and a given file access protocol. Supported protocols:
castor: (CERN), rfio: (CERN, PIC, CNAF), dcache: (
GridKa, RAL), gfal (Grid File Access Library) plugin of ROOT
Logical File Names (LFN): these are not targeted to a given site and allow the jobs to run on the Grid. The application has to resolve it to a PFN by means of a File Catalog (FC). The FC can either be the LHCb full catalog (LFC) or a local restricted catalog in XML language (XML catalog).
The genCatalog XML script allows translating a list of files given by their LFN.
An XML file catalog allows Gaudi to translate the LFN in the jobOptions file to the appropriate PFN. One can also require that the parent files of the files used are included in the catalog (see the -d --depth option). This is necessary for navigating to ancestor files or using Event Tag Collections.
Note that your rDST might be in the swtest "freezer" but if requiring raw banks the job will also need access to the raw file which is stored on tape (here depth=2 in the genCatalog XML script).
This is an example of an entry in a .xml file
</File>
<File ID="6cbb0118-ce8e-11e7-a034-001e67ddcb96">
<physical>
<pfn filetype="ROOT_All" name="mdf:root://clhcbstager.ads.rl.ac.uk//castor/ads.rl.ac.uk/prod/lhcb/buffer/lhcb/data/2017/RAW/FULL/LHCb/COLLISION17/202214/202214_0000000854.raw?svcClass=lhcbDst" se="RAL-BUFFER"/>
<pfn filetype="ROOT_All" name="mdf:root://eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/swtest/lhcb/data/2017/RAW/FULL/LHCb/COLLISION17/202214/202214_0000000854.raw" se="CERN-SWTEST"/>
</physical>
<logical>
<lfn name="/lhcb/data/2017/RAW/FULL/LHCb/COLLISION17/202214/202214_0000000854.raw"/>
</logical>
(Adapted from
Using Gen Catalog)
SVN trickery (obsolete)
This might be useful as mistakes with svn can happen.In this link there are the
guidelines of svn.
To tag a version i.e:
StrippingSelections to a particular tag execute the following command:
svn cp -r 198298 https://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/ https://svn.cern.ch/reps/lhcb/Stripping/tags/Phys/StrippingSelections/v4r9/ -m "Tagging v4r9"
GIT basics
All basic information and links can be found in
https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb.
To checkout single files from a different branch than the branch you're currently working on, say stripping24r1p1-patches, do
git fetch origin stripping24r1p1-patches
git checkout origin/stripping24r1p1-patches -- Phys/StrippingSettings/python/StrippingSettings/Stripping24r1p1/LineConfigDictionaries_Calib.py Phys/StrippingSettings/dbase/stripping24r1p1
Fix a messed up history in git
If a merge or commit has been pushed to the repository incorrectly, you can undo it (in a full checkout of the package) with
git revert <commit hash>
You can get the commit hash from `git log`. If it's a merge commit then you need to specify the parent with
git revert -m <parent ID> <commit hash>
In the output of `git log` for a merge commit you'll see the two parent commits that were merged on the "Merge:" line, so normally you'll want to pick 1 or 2.
However, `git revert` changes the files back to the way they were before the commit, but it doesn't revert their history - the change back is logged as a new edit in the history. If you want also to revert the history then you need to find the hash of the commit before the unwanted change and do
git reset <commit hash>
which resets the tree to that commit, then check what's being reverted with
git status
Then revert it with
git reset --hard
Check that the history is as you expect with `git log`, then force push to the remote repo with
git push -f <remote> <branch>
The history on the remote branch will now also be rewritten. Note that gitlab doesn't let you force push to protected branches, so if the target branch is protected you'll need to remove its protection temporarily to do the force push and then re-protect it.
This works if you want to remove all commits after a certain point. If you only want to remove selected commits then instead of `git reset
` and then `git reset --hard`, you want to do
git rebase -i <commit hash>
again the commit hash here is that of the commit before the earliest commit you want to revert. This will open an editor with a list of all commits after that point. Delete the lines of all the commits you want to remove and then save and quit the editor. Then check that the history is OK with `git log`, and force push to the remote branch.
How to get the xml file from RDSTs
lhcb-proxy-init
lb-run LHCbDirac /prod dirac-bookkeeping-genXMLCatalog --Options=YourData.py --Catalog=yourdatacatalog.xml --Depth=2
Useful dirac commands
lb-run LHCbDirac /prod dirac-bookkeeping-file-metadata lfn
gives you the number of events, run number, DQ flag amongst others
lb-run LHCbDirac /prod dirac-bookkeeping-file-path --LFN lfn
gives you the actual location (useful for freezer cleaning)
lb-run LHCbDirac /prod dirac-dms-lfn-metadata lfn
lists the metadata of the file (owner, size, ...)
lb-run LHCbDirac /prod dirac-dms-lfn-replicas lfn
shows the existing replicas of the file
lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/CollisionYEAR/CONDITIONS/RealData/90000000/RAW --Trigger
gives you the stable beam time
lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/CollisionYEAR/CONDITIONS/RealData///90000000/ --Trigger
gives you the size of the stripped data
Please note that you have to write RealData instead of Real Data.
Inspect a DST file with Bender
For example if you'd like to check the Stripping decision of the following MDST file /lhcb/validation/Collision15/BHADRON.MDST/00067793/0000/00067793_00000191_1.bhadron.mdst
lb-run LHCbDirac/latest dirac-bookkeeping-genXMLCatalog -l /lhcb/validation/Collision15/BHADRON.MDST/00067793/0000/00067793_00000191_1.bhadron.mdst
lb-run Bender/latest bender -x pool_xml_catalog.xml -l /lhcb/validation/Collision15/BHADRON.MDST/00067793/0000/00067793_00000191_1.bhadron.mdst
In Bender, seekStripDecision('StrippingZVTOP_High*')
will give you the decision for the lines you're interested in. To look at the candidates, use get('Bhadron/Phys/CONTAINER/Particles')
.
If certain raw banks are missing for a single event, which should not be the case, replace DaVinci().appendToMainSequence( [ sc.sequence() ] )
with
sc_seq = sc.sequence()
sc_seq.RequireObjects = ['DAQ/RawEvent']
DaVinci().appendToMainSequence( [ sc_seq ] )
Prerequisites
Stripping coordinators need the following rights:
- Have
lhcb_tech
role in DIRAC to be able to modify production steps
- Be members of
lhcb-release-managers
e-group to be able to tag projects in tag collector
- Be members of
z5:doclibrarians
AFS group to be able to publish stripping lines documentation on the web pages
- Be subscribed to
lhcb-stripping-managers
for obvious reasons.
Actions before stripping selections deadline
The deadline for submitting new or modifying the existing stripping selections is decided by the PPG, taking into account the time of conferences where the new results have to be presented, processing time and the time to prepare the stripping and DaVinci release after the deadline. The latter can take up to one month and depends on the number of iterations required to reach the limits on bandwidth and/or timing.
Prepare test samples
Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed. It can make sense to take (parts of) several runs as the test sample if the running conditions or trigger configuration was changing significantly. Once the run for the test sample has been identified, inform bookkeeping manager and ask to copy it to "freezer". The python options have to be prepared and put to Phys/StrippingSelections/tests/data/
.
Prepare patch for DaVinci
For incremental and IFT (re-)strippings, DaVinci patches need to be prepared. For incremental re-strippings, the patch should be created from the DaVinci version corresponding to the stripping version and for IFT, make sure you patch from the DaVinci version using the correct Reco version.
If you need to change the environment, do
. LbLogin.sh -c x86_64-slc6-gcc49-opt
or run with
lb-run -c x86_64-slc6-gcc49-opt
To build the patch, you can do
export CMTPROJECTPATH=${CMTPROJECTPATH/$User_release_area/$PWD}
export User_release_area=$PWD
export LBCONFIGURATIONROOT="/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_vXrY/LbConfiguration"
for project in Phys Stripping Analysis DaVinci ; do
git clone "ssh://git@gitlab.cern.ch:7999/lhcb/${project}.git"
cd "$project"
lb-project-init
git fetch
git checkout <your_branch>
make configure >& stdout-make-configure
make install >& stdout-make-install
cd ..
done
FYI, you might need to update the versions of Phys and Analysis in DaVinci or Stripping if you only clone these two packages. The version of Stripping has to match in Stripping and DaVinci.
Actions after stripping selections deadline
After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings
.
Run retention tests (Grid)
Setup DaVinci from the nightlies for the branch (e.g. 2017-patches) and the build (e.g. Tue for Tuesday) do
cd <wherever>
lb-dev --nightly <nightly slot> DaVinci/<branch>
This will create the DaVinci branch bound to the nightly slot
cd DaVinciDev_<branch>
git lb-use Stripping
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSelections
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSettings
git lb-checkout Stripping/Stripping<XY> Phys/StrippingArchive
git lb-checkout Stripping/Stripping<XY> Phys/CommonParticlesArchive
make configure && make
Note that Stripping is the branch here, not the stripping version!
In<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py you might want to change LocalFile to DiracFile and check the platform.
Start ganga and for example for S29 on test data Reco16_Run18259
execfile('<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py')
j = make_stripping_job('S29-loose', '<wherever>/DaVinciDev_HEAD/', '<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/data/Reco16_Run182594.py', nfiles = -1, reportfreq = 50000, evtmax = 1000000, dirac = True, Version = 'Stripping29', SplitStreamsByWG = True)
j.submit()
to run the retention tests. ./Phys/StrippingSelections/tests/coordinators/StrippingReport.py -d DIRECTORY=searches for all strippingreport.py’s and adds them together. The retention table for the twiki can be obtained via =./Phys/StrippingSelections/tests/coordinators/StrippingReport.py -d DIRECTORY -r --hlt2rate RATE > retentions.txt
, which uses a single output file at the moment.
Run retention tests (locally)
Run
gaudirun.py -T $STRIPPINGSELECTIONSROOT/tests/coordinators/retentions/TestFromSettings_all.py path_to_input_data
to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script. You need to change the WG and Stripping, and possibly year and tags. If you want to run a smaller test of e.g. 100k, set EvtMax accordingly. It is advisable to create separate folders for each WG to avoid that the output DSTs get overwritten (you will need them later).
It is advisable to run the tests in a tmux session by setting the following alias alias ptmux="pagsh.krb -c 'kinit && tmux'"
first. This way your kerberos token will not expire afte a couple of hours.
FYI SL corresponds to Semileptonic, Calib to ALL and BnoC to Charmless.
Run memory use tests
To test the global memory consumption, a single, reasonably long (100k-200k events) stripping job is needed (see step above). Store the log of this job in the file, then use analyseDaVinciLog.py
script to plot the memory use as a function of event number.
python(2) analyseDaVinciLog.py logfile.txt -p name_of_your_plot.png
display name_of_your_plot.png
The options -c and -l are currently not working.
To run safely in the production, jobs don't have to take more than 4 Gb of memory. Make sure there are no big jumps in memory use after ~50-100k events. They can indicate memory leaks. In case of doubt, use valgrind
to find memory leaks. Memory auditor can also help to find an algorithm with memory leak. To enable it, add the following to the DaVinci options:
ApplicationMgr().ExtSvc += [ 'AuditorSvc' ]
AuditorSvc().Auditors += [ 'MemoryAuditor' ]
Run bandwidth tests
The total bandwidth for the stripping should be agreed with the bookkeeping manager and the PPG. For S28, the full Stripping had around 150 Mb/s. The maximum bandwidth of the incremental restripping should be "small" compared to it (not more than 20% of the full stripping).
Bandwidth and timing tests are run on the test sample. Scripts are provided (files ganga.py
, submit_jobs.sh
). Since large number of events is required for reliable calculation, the test jobs are submitted to the GRID (use submit_jobs.py
after setting up Ganga
environment).
Script retentions.py
or retentions_new.py
can be used after the test jobs finish to calculate the retentions and timing of individual lines, streams and WGs and generate tables in the twiki format. One needs to edit its header to change the job IDs of the test jobs (or local folder where the logfiles are stored), DaVinci version and, possibly, the trigger rate. The streamEventSize has to be changed each time you made a change to the Stripping. To get the numbers for the streamEventSize, do
ls -ltr path_to_output_dsts
and add these numbers to the correct streams of the corresponding WG. FTAG is called MDST in retentions_new.py
.
The trigger rate is obtained as follows:
- Run the follwing script to get the
summary.xml
(in this example for 2015).
from Configurables import DaVinci
from Gaudi.Configuration import importOptions
DaVinci().DataType = '2015'
DaVinci().Lumi = True
DaVinci().EvtMax = 100000
from Configurables import CondDB
CondDB().LatestGlobalTagByDataType = "2015"
from Configurables import LHCbApp
LHCbApp().XMLSummary = "summary.xml"
importOptions("$STRIPPINGSELECTIONSROOT/tests/data/Reco15a_Run164668.py")
After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists.It is very important that all WGs check that the lines they need are included, and report back if anything is missing.
Known lines with higher retentions than usually allowed are (justified):
Run line independence tests
These should be run once the stripping selections are frozen. Assuming that the lines for 2011 and 2012 data are the same as each other, only one year needs to be checked. The scripts to run these tests are attached to this TWiki page. (The TWiki software has appended ".txt" to all of these filenames for some reason, this should be removed obviously)
The DaVinci scripts DV-Stripping20r0p3-Indep.py and DV-Stripping20r0p3-Indep-all.py run the stripping and save the stripping decisions in a ROOT NTuple (these scripts are set up to run S20r0p3, but changing to other strippings should be straightforward). The Indep-all.py runs all of the lines in a particular stripping, while Indep.py runs only a single line. Of course each script must be run over the same input events.
The ganga script ganga_all.py runs DV-Stripping20r0p3-Indep-all.py in ganga. This only needs to be run once, producing a single ganga job. The ganga script ganga_line.py runs DV-Stripping20r0p3-Indep.py in ganga, for a certain stripping line. This needs to be run as many times as there are lines in the given stripping. This produces many ganga jobs. To run these many jobs, there is a small script called submit_jobs.py. This reads in the various line names from a plain text file, which can be produced by running a short test of DV-Stripping20r0p3-Indep-all.py locally. Running submit_jobs.py to submit a lot of ganga jobs can take many hours.
Once the NTuples have been created, a script called check_tuples.py is used to run over the various single-line NTuples, and compare the stripping decisions with the NTuple from the whole-stripping job. The script can deal with incomplete or failed subjobs by skipping over the missing data. The only part of check_tuples.py that should need changing is the job numbers for the ganga jobs, and the location of the gangadir directory holding the output of the jobs.
If there is a mismatch between the stripping decision of a given line for a certain event, you will see a mesage like this in the stdout of check_tuples.py:
Checking line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision (job 724)subjob 0 checked, 48501 events, 37 selected Stripping decision is different: line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision, event 29990 (job 724, subjob 1): all=0, single=1
This indicates that this line gave a positive decision on this event when the lines was run on its own, but a negative decision when ran as part of the whole stripping. Such mismatches can be caused by a known feature of the way that the stripping deals with events where the maximum number of allowed combinations is reached in an algorithm such as CombineParticles or SubPIDMMFilter (details of this can be found in the Stripping Coordinator Report here:https://indico.cern.ch/event/201535/session/3/contribution/29/material/slides/1.pdf). This feature has a tendency to affect lines in the Beauty2Charm module, but occasionally affects other lines as well. To check whether this feature is the cause of a mismatch, grep the log files of the single-line stripping job for "Maximum number of allowed particles reached". You are looking for a line like:
D+2HHHSubPIDSelBeauty2Charm WARNING SubPIDMMFilter:: Maximum number of allowed particles reached
If one (or more) of these warnings is found at an event number matching that given in the output of check_tuples.py, then this feature is to blame and (as this is a known issue) no action needs to be taken. If no such warning is found, then there is some other, new issue with this line that requires further investigation.
Debugging with gdb
If there's an issue with the stripping that's not easy to work out, eg, if a line is hanging for some reason, gdb can be useful. To run the stripping with gdb use, eg
lb-run -c x86_64-slc6-gcc49-dbg DaVinci/<version> sh -c 'gdb --args python $(which gaudirun.py) <options>'
See also here. You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with
kill -2 <pid>
from another terminal it immediately continued.
Once gdb has started, turn on logging with
set logging on
then start the job with run
. You can then interrupt whenever you like with ctrl-c, or if you're investigating a crash you can just wait for it, then you can use backtrace
to get the method call backtrace. Use continue
to continue the job after an interrupt.
Once you have the backtrace you can print info on an object given its address with, eg
print *(class CombineParticles*)0x614b8940
or
print *('DaVinci::N4BodyDecays' *) 0x5dc58cf0
(note the single quotes for objects in a namespace) then you can search the output, of which there will likely be a lot. If you're trying to work out which line a problematic object originates from, search the output for anything containing 'Line' or 'Stripping'. Then you can have a look at the module that the line belongs to.
If you're trying to catch some output, you can set a break point for any output to console with
b write if 1==$rdi
or
b fwrite if $rcx==&_IO_2_1_stdout_
Release Stripping project
Help on release procedure is at PrepareProjectReleaseGit page. Please check the steps following the instructions on ProdReqPreLaunchTest. For full DST productions, no ancestors need to be included.
If the above instructions fail for RDST files, remove the RAW from the generated ProdConf*.py and run with the other options locally.
Check nightlies and correct if needed
Nightlies page is here
. Nightlies have to be checked not only before the release of Stripping, but also from time to time during the development of the stripping selections by the analysts to spot problems early. In case of problems, inform the responsible person. In some cases, the nightly tests failures are expected because of changes introduced, and updating the test reference is needed to fix them.
Tests run by the nightlies are defined by the files under
$PACKAGEROOT/tests/qmtest/package.qms/*.qmt
Scripts to be run live under $PACKAGEROOT/tests/
and references, if needed, are under $PACKAGEROOT/tests/refs
.
To run the test of a package, after building do:
make test
Under cmt, you could run a single test, assuming the tests are in a directory davinci.qms
and the test is called mytest.qmt
, with:
cmt TestPackage davinci.mytest
but I'm not sure how to do this under cmake.
To update the test reference (after running the test):
cd tests/refs
mv ***.ref.new ***.ref
If the list of trigger lines has to be updated, you can do lb-run Moore/latest Phys/StrippingSelections/tests/coordinators/get_hlt_lines.py --tck 0x11431621 --outputfile 0x11431621.py
. At some point during Run 2, the following notation was introduced: 0x1 are HLT1 TCKs, 0x2 are HLT2, and 0x5 and 0x6 are for MC.
StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections, which are used only for development. CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified and lives in Phys now.
Get DaVinci from the nightlies eg. for the branch (e.g. 2017-patches) and the build (e.g. Tue for Tuesday) do (See also the setup instructions in "Run retention tests (Grid)")
cd <wherever>
lb-dev --nightly-cvmfs --nightly <yourbranch> <day> DaVinci/HEAD
cd DaVinciDev_HEAD
git lb-use Stripping
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSelections
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSettings
git lb-checkout Stripping/Stripping<XY> Phys/StrippingArchive
git lb-checkout Stripping/Stripping<XY> Phys/CommonParticlesArchive
make configure && make
./run \$STRIPPINGSELECTIONSROOT/tests/coordinators/create_archives.py Stripping<XY> "<Description>"
assuming that the changes you want are in a branch called Stripping. The latter command creates or updates StrippingArchive, CommonParticlesArchive and if necessary the StrippingSettings dbase. Use git status to see what has been made.
The dbase file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory:
git clone ssh://git@gitlab.cern.ch:7999/lhcb/Stripping.git
cd Stripping
git checkout <yourbranch>
cd Stripping/Phys/StrippingSettings/dbase/
cp from/the/path/where/it/is/strippingXX .
git add strippingXX
git commit strippingXX -m "update dbase for sXX"
git push
Check lost tags in tag collector
Tag collector page is here
. Go to "Lost tags" and add them if needed.
Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)
This section describes how to update the reference files used for the nightly tests.
Clone DaVinci like this
git clone ssh://git@gitlab.cern.ch:7999/lhcb/Project.git
cd Project
lb-project-init
# You may have to use a different branch of DV depending on the nightly test you are looking at eg. lhcb-2016-patches
git checkout -b xxx
make
OR
lb-dev --nightly lhcb-2017-patches DaVinci/2017-patches
cd ./DaVinciDev_2017-patches
git lb-use DaVinci
git lb-checkout DaVinci/2017-patches DaVinciTests
make
For more info on lb- commands see here.
This will grab the same configuration as the nightly slot called lhcb-2017-patches . A slot is a consistent set of inter-related software projects that should be built and tested together. The configuration of a slot includes, in addition to the list of configured projects, the list of platforms the slot should be built for and some extra metadata for the fine tuning of the build. Projects in 2017-patches for example check out DaVinci branch 2017-patches plus all non-WIP merge requests targeting the 2017-patches branch.
For more info on the nightlies see here
(See here
and here for even more on lhcb software development)
If required also get the Phys package the same way
git lb-use Phys
git lb-checkout DaVinci/2017-patches Phys
First, update the stripping list in Phys/StrippingCache/CMakeLists.txt where XXrY is the stripping version ans xx the data year
set(extraopts_XXrY
${AppConfig_DIR}/options/DaVinci/DataType-20xx.py
)
and put the stripping version in the strip list below
and add options under Phys/StrippingCache/options
DV-StrippingXXrY-Stripping.py
To check which functors has not been cached, do
export LOKI_DISABLE_PYTHON=1
before testing. This is already included in $DAVINCITESTSROOT/tests/options/DisableLoKiPythonFunctors.py
. Just make sure this option file is included in the qmtest, which you need to create under DaVinciTests/tests/qmtest/stripping.qms/
.
You might need to create an empty file for the reference, otherwise it won't be created.
To test, do
make test ARGS='-N'
make test ARGS='-R DaVinciTests.stripping.test_strippingXXrY_collisionxx_recoyy'
If the test results differ from the reference this will create the file DaVinciTests/tests/refs/.ref.new
, which you rename without the .new and have to commit. The test results are in build/Testing/Temporary/LastTest_*.log
. At the end of the file it will display the differences between this test and the reference. This should match what is seen in the "Output Diff" of the failed nightly test
Tricks: The tests under DaVinci/DaVinciTests/tests/qmtest/stripping.qms/
need to included in StrippingCache/CMakeList.txt
. The nightlies configuration can be found on gitlab, e.g. here
. In the nightlies, always the latest version of TMVAWeights is used, which is why v999 appears in the nightlies tests.
To commit:
git add <reference file>
git commit -m 'new reference file for ....'
git lb-push DaVinci <name-of-new-branch>
You can then make a merge request for your new branch into whichever branch of DaVinci you are working on. Check if the nightly passed the next day
Tag packages in Stripping project
Go through all the packages in Stripping and look at the release.notes. If there’re any updates since the last release then you add in the tag header with the incremented version. Increment the version in CMakeLists. For StrippingSys: The release notes should be the collection of the release notes for all packages that have changed since the last release. Add the tag header and increment the version in CMakeLists.txt. In the top directory of Stripping, set the version of Stripping in CMakeLists.txt and the version of Phys on which it depends in CMakeLists.txt. Push the changes, go to the Stripping gitlab page, click on "tags" and then "new tag". Name the tag, give a short message for the tag, select the branch from which the tag should be made, and copy the release notes of StrippingSys into the release notes of the tag. In case you need to re-tag, go to the "tags" page and click "delete tag" on which ever tag you want to replace, then remake the tag as above.
To check if a tag exists, you can grep the CMakeLists.txt, e.g. grep v4r19 $LHCb_release_area/STRIPPING/STRIPPING_v*/Phys/StrippingSelections/CMakeLists.txt
or use cmt/requirements
if you're looking at older versions.
Tag packages in DaVinci project
The only package that needs to be tagged by the Stripping coordinators is the StrippingCache which lives in DaVinci. Follow the same steps as for the previous step.
Tag extra packages in PARAM
Packages like TMVAWeights required by some RelatedInfoTools live in PARAM and might also need to be tagged.
getpack TMVAWeights h
<modify release notes and update versions wherever required>
svn ci -m "releasing new version vXrY"
tag_package TMVAWeights vXrY
Then open a new JIRA task in LHCBDEP asking for deployment.
Add Stripping project to lhcb-prerelease slot in the nightlies
Go to nightlies page
, open "Configuration Editor", choose the slot needed (most likely lhcb-prerelease
), double-click on Stripping tag and change the STRIPPING version there. Make sure "disabled" flag is "false".
If you do not want to wait for the nightlies, follow the instructions here ProjectRelease#For_Release_Managers
The stripping options are in AppConfig/options/DaVinci/DV-StrippingXX-Stripping.py
. In most cases the DaVinci options for stripping can be copied from the previous round, with minor changes:
- Change stripping version used to retrieve configuration from StrippingSettings and StrippingArchive.
- Stripping TCK has to be changed. Current convention is the following: TCK = 0xVVVVSSSS, where VVVV is the DaVinci version, and SSSS is the Stripping number (e.g. for Stripping20r1p1 running on DaVinci v32r2p5 TCK = 0x32252011).
- If necessary create DataType -<yyyy>.py, where <yyyy> corresponds to the year.
If the stripping configuration is expected to run in MC, the corresponding MC stripping options (DV-StrippingXX-Stripping-MC*.py
) have to be updated. Please note that AppConfig still lives in svn.
In the case that only a patch of AppConfig needs to be deployed, prepare the tag as as usual: set the version number in cmt/requirements
, and add in the release header in doc/release.notes
, and commit. Then tag the package as usual with tag_package AppConfig vXrY
, at which point it should appear here
. No build is required as AppConfig only contains options files, so you can request deployment in JIRA
, an example request is here
. That's all!
Trigger a release build of DaVinci
For minor changes, eg. changing the Stripping version picked up by DaVinci, we are allowed to create a tag for DaVinci, which follows the same procedure as the Stripping. However, we cannot delete a tag from DaVinci if we mess up! After creating the Stripping and DaVinci tag,
> lb-sdb-import DaVinci vXrY
triggers a release build, which you should check here
. If the builds are ok, you can go ahead and request deployment of DaVinci citing the number of the release build (see below). Please note that it can take around 30min before the release build appears on the website.
Ask for release in Jira
Ask Stripping, DaVinci and AppConfig to be released using JIRA
. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).
Prepare production steps
Go to https://lhcb-portal-dirac.cern.ch/DIRAC/
, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert (currently Marco Cattaneo and Liang Sun). Test the steps before marking them as "Ready".
To test the steps, you should follow the instructions given here. You might need to test different versions of LHCbDirac until you find a working one. The test will create a Local_*
folder, in which you find prodConf_*.py
. You should remove the raw file from the list of inputs, potentially edit the number of events, and depending on where you test prodConf_*.py
, add the full path to the xml catalogue.
Run validation
Announce the steps on the operations elog, and send an email to the data managers and production managers asking for a validation giving a suitable run range. The typical size of a validation comprises a couple thousand files.
Run production
Debugging tricks for problems in production
To see all algorithms running, add
from Configurables import NameAuditor, AuditorSvc
AuditorSvc ().Auditors.append(NameAuditor ())
to your options and redirect the output to tmp as the logfile might become several GB large.
The timings of each algorithms for each event can be written by adding
from Configurables import TimingAuditor, AuditorSvc, ChronoStatSvc, SequencerTimerTool, ChronoAuditor
from Gaudi.Configuration import INFO
TimingAuditor ().addTool(SequencerTimerTool,name="TIMER")
TimingAuditor ().TIMER.NameSize = 60
TimingAuditor ().OutputLevel = INFO
AuditorSvc ().Auditors.append(ChronoAuditor ("Chrono"))
ChronoStatSvc ().PerEventFile = 'eventtimings.txt'
to your options (copy the Stripping options from AppConfog and replace the existing TimingAuditor with the lines above). In 'eventtimings.txt', all algorithms are listed with the respective timings. Please keep in mind that not all algorithms are being run for each event.
Prepare web documentation
Reference documentation about all lines (their cuts, sequence of algorithms, streams, output locations) and standard particles is available at the Stripping project webpages
. These pages are generated automatically using the HltGenConfig algorithm (the same one that is used to generate HltTCK). Here are the instructions on generating these pages:
- Prepare build environment for DaVinci version used in stripping production:
> lb-dev DaVinci vVVrRpP
> cd DaVinciDev_vVVrRpP
- Create an empty directory in a place with plenty of space
- Build and set the environment variables:
> make
> ./run bash
> mkdir strippingXX
> cd strippingXX
> cp $STRIPPINGSELECTIONSROOT/tests/coordinators/documentation/* .
- Run the ./run.sh script which will run 10000 events (to make sure most on-demand services are initialised) and produce the stripping config in the file config.tar
> ./run.sh
- Run ./make_html.sh script which will create web documantation in the "html" folder using config.tar (note that utils.py module is required by this script)
> ./make_html.sh
-
cp -r html/strippingXX $LHCBDOC/stripping/config/
and edit $LHCBDOC/stripping/maindesc.html
.