Difference: StrippingCoordinatorsCheatsheet (1 vs. 88)

Revision 882018-11-01 - NicolaAnneSkidmore

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 231 to 231
  Run
Changed:
<
<
gaudirun.py -T TestFromSettings_all.py path_to_input_data
>
>
gaudirun.py -T $STRIPPINGSELECTIONSROOT/tests/coordinators/retentions/TestFromSettings_all.py path_to_input_data
  to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script. You need to change the WG and Stripping, and possibly year and tags. If you want to run a smaller test of e.g. 100k, set EvtMax accordingly. It is advisable to create separate folders for each WG to avoid that the output DSTs get overwritten (you will need them later).

Revision 872018-10-30 - NicolaAnneSkidmore

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 201 to 201
 
cd <wherever>
Changed:
<
<
lb-dev --nightly-cvmfs --nightly DaVinci/HEAD cd DaVinciDev_HEAD
>
>
lb-dev --nightly DaVinci/ This will create the DaVinci branch bound to the nightly slot

cd DaVinciDev_<branch>
 git lb-use Stripping git lb-checkout Stripping/Stripping Phys/StrippingSelections
Added:
>
>
git lb-checkout Stripping/Stripping Phys/StrippingSettings git lb-checkout Stripping/Stripping Phys/StrippingArchive git lb-checkout Stripping/Stripping Phys/CommonParticlesArchive
 make configure && make
Added:
>
>
Note that Stripping is the branch here, not the stripping version!
  In<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py you might want to change LocalFile to DiracFile and check the platform.
Line: 385 to 394
  StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections, which are used only for development. CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified and lives in Phys now.
Changed:
<
<
Get DaVinci from the nightlies eg. for the branch (e.g. 2017-patches) and the build (e.g. Tue for Tuesday) do
>
>
Get DaVinci from the nightlies eg. for the branch (e.g. 2017-patches) and the build (e.g. Tue for Tuesday) do (See also the setup instructions in "Run retention tests (Grid)")
 
cd <wherever>
Line: 400 to 409
  ./run \$STRIPPINGSELECTIONSROOT/tests/coordinators/create_archives.py Stripping ""
Changed:
<
<
assuming that the chages you want are in a branch called Stripping. The latter command cretaes or updates StrippingArchive, CommonParticlesArchive and if necessary the StrippingSettings dbase.
>
>
assuming that the changes you want are in a branch called Stripping. The latter command creates or updates StrippingArchive, CommonParticlesArchive and if necessary the StrippingSettings dbase. Use git status to see what has been made.
  The dbase file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory:

Revision 862018-10-11 - NicolaAnneSkidmore

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet


Added:
>
>

What is an rDST

When data is collected from the LHCb detector, the raw data will be transferred to the LHCb associated Tier 1 sites for the reconstruction to produce rDST files. The rDST files are used for stripping jobs where events are selected for physics analysis. Selected events are written to DST files and distributed in identical copies to all the Tier 1 sites. These files are then accessible for physics analysis by individual collaborators.

(Taken from Streaming Task Force page)

What is a .xml file

Physical file names (PFN): these are targeted to a site and a given file access protocol. Supported protocols: castor: (CERN), rfio: (CERN, PIC, CNAF), dcache: (GridKa, RAL), gfal (Grid File Access Library) plugin of ROOT

Logical File Names (LFN): these are not targeted to a given site and allow the jobs to run on the Grid. The application has to resolve it to a PFN by means of a File Catalog (FC). The FC can either be the LHCb full catalog (LFC) or a local restricted catalog in XML language (XML catalog).

The genCatalog XML script allows translating a list of files given by their LFN. An XML file catalog allows Gaudi to translate the LFN in the jobOptions file to the appropriate PFN. One can also require that the parent files of the files used are included in the catalog (see the -d --depth option). This is necessary for navigating to ancestor files or using Event Tag Collections.

Note that your rDST might be in the swtest "freezer" but if requiring raw banks the job will also need access to the raw file which is stored on tape (here depth=2 in the genCatalog XML script).

This is an example of an entry in a .xml file

</File>
<File ID="6cbb0118-ce8e-11e7-a034-001e67ddcb96">
<physical>
<pfn filetype="ROOT_All" name="mdf:root://clhcbstager.ads.rl.ac.uk//castor/ads.rl.ac.uk/prod/lhcb/buffer/lhcb/data/2017/RAW/FULL/LHCb/COLLISION17/202214/202214_0000000854.raw?svcClass=lhcbDst" se="RAL-BUFFER"/>
<pfn filetype="ROOT_All" name="mdf:root://eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/swtest/lhcb/data/2017/RAW/FULL/LHCb/COLLISION17/202214/202214_0000000854.raw" se="CERN-SWTEST"/>
</physical>
<logical>
<lfn name="/lhcb/data/2017/RAW/FULL/LHCb/COLLISION17/202214/202214_0000000854.raw"/>
</logical>

(Adapted from Using Gen Catalog)

 

SVN trickery (obsolete)

This might be useful as mistakes with svn can happen.In this link there are the guidelines of svn.

Revision 852018-10-11 - NicolaAnneSkidmore

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 409 to 409
 git lb-checkout DaVinci/2017-patches DaVinciTests make
Changed:
<
<
For more info on lb- commands see here. This will grab the same configuration as the nightly slot lhcb-2017-patches with all pending merge requests applied. See here and here for more on this.
>
>
For more info on lb- commands see here.

This will grab the same configuration as the nightly slot called lhcb-2017-patches . A slot is a consistent set of inter-related software projects that should be built and tested together. The configuration of a slot includes, in addition to the list of configured projects, the list of platforms the slot should be built for and some extra metadata for the fine tuning of the build. Projects in 2017-patches for example check out DaVinci branch 2017-patches plus all non-WIP merge requests targeting the 2017-patches branch.

For more info on the nightlies see here

(See here and here for even more on lhcb software development)

  If required also get the Phys package the same way
Line: 462 to 470
 git commit -m 'new reference file for ....' git lb-push DaVinci <name-of-new-branch>
Changed:
<
<
You can then make a merge request for your new branch into whichever branch of DaVinci you are working on. Name with the prefix WIP: and check if the nightly passed the next day smile
>
>
You can then make a merge request for your new branch into whichever branch of DaVinci you are working on. Check if the nightly passed the next day smile
 

Revision 842018-10-10 - NicolaAnneSkidmore

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 389 to 389
  This section describes how to update the reference files used for the nightly tests.
Changed:
<
<
This is not compiling if you use git lb-checkout. Clone DaVinci properly like this
>
>
Clone DaVinci like this
 
git clone ssh://git@gitlab.cern.ch:7999/lhcb/Project.git
Line: 400 to 400
 make
Added:
>
>
OR
 
Changed:
<
<
First, update the stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options. To check which functors has not been cached, do
>
>
lb-dev --nightly lhcb-2017-patches DaVinci/2017-patches
cd ./DaVinciDev_2017-patches
git lb-use DaVinci
git lb-checkout DaVinci/2017-patches DaVinciTests
make
For more info on lb- commands see here. This will grab the same configuration as the nightly slot lhcb-2017-patches with all pending merge requests applied. See here and here for more on this.

If required also get the Phys package the same way

git lb-use Phys
git lb-checkout DaVinci/2017-patches Phys

First, update the stripping list in Phys/StrippingCache/CMakeLists.txt where XXrY is the stripping version ans xx the data year

set(extraopts_XXrY
${AppConfig_DIR}/options/DaVinci/DataType-20xx.py
)
and put the stripping version in the strip list below

and add options under Phys/StrippingCache/options

DV-StrippingXXrY-Stripping.py

To check which functors has not been cached, do

 
export LOKI_DISABLE_PYTHON=1
Changed:
<
<
before testing. This is already included in $DAVINCITESTSROOT/tests/options/DisableLoKiPythonFunctors.py. Just make sure this option file is included in the qmtest, which you need to create under DaVinciTests/tests/qmtest/stripping.qms/. You might need to create an empty file for the reference, otherwise it won't be created.
>
>
before testing. This is already included in $DAVINCITESTSROOT/tests/options/DisableLoKiPythonFunctors.py. Just make sure this option file is included in the qmtest, which you need to create under DaVinciTests/tests/qmtest/stripping.qms/.

You might need to create an empty file for the reference, otherwise it won't be created.

  To test, do
make test ARGS='-N'
Changed:
<
<
make test ARGS='-R DaVinciTests.stripping.'
>
>
make test ARGS='-R DaVinciTests.stripping.test_strippingXXrY_collisionxx_recoyy'

If the test results differ from the reference this will create the file DaVinciTests/tests/refs/.ref.new, which you rename without the .new and have to commit. The test results are in build/Testing/Temporary/LastTest_*.log. At the end of the file it will display the differences between this test and the reference. This should match what is seen in the "Output Diff" of the failed nightly test

 
Deleted:
<
<
This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well. The test results are in build/Testing/Temporary/LastTest_*.log.
  Tricks: The tests under DaVinci/DaVinciTests/tests/qmtest/stripping.qms/ need to included in StrippingCache/CMakeList.txt. The nightlies configuration can be found on gitlab, e.g. here. In the nightlies, always the latest version of TMVAWeights is used, which is why v999 appears in the nightlies tests.
Added:
>
>
To commit:
 
git add <reference file>
git commit -m 'new reference file for ....'
Changed:
<
<
git push -u origin xxx
>
>
git lb-push DaVinci <name-of-new-branch>
 
Added:
>
>
You can then make a merge request for your new branch into whichever branch of DaVinci you are working on. Name with the prefix WIP: and check if the nightly passed the next day smile
 

Revision 832018-10-10 - NicolaAnneSkidmore

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 387 to 387
 

Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)

Added:
>
>
This section describes how to update the reference files used for the nightly tests.
 This is not compiling if you use git lb-checkout. Clone DaVinci properly like this
Added:
>
>
 
git clone ssh://git@gitlab.cern.ch:7999/lhcb/Project.git
cd Project
Line: 392 to 395
 git clone ssh://git@gitlab.cern.ch:7999/lhcb/Project.git cd Project lb-project-init
Changed:
<
<
# use '-b' only if that branch does not exist in the remote repository git checkout -b JIRATICKET-123 make make test # edit edit edit... git add Some/Package/src/MyStuff.cpp git commit -m 'fixing feature abc (JIRATICKET-123)' vim Some/Package/src/MyStuff.cpp
>
>
# You may have to use a different branch of DV depending on the nightly test you are looking at eg. lhcb-2016-patches git checkout -b xxx
 make
Deleted:
<
<
make test git commit -a -m 'improved fix to JIRATICKET-123' git push -u origin JIRATICKET-123
 
Line: 423 to 416
  Tricks: The tests under DaVinci/DaVinciTests/tests/qmtest/stripping.qms/ need to included in StrippingCache/CMakeList.txt. The nightlies configuration can be found on gitlab, e.g. here. In the nightlies, always the latest version of TMVAWeights is used, which is why v999 appears in the nightlies tests.
Added:
>
>
git add <reference file>
git commit -m 'new reference file for ....'
git push -u origin xxx

 

Tag packages in Stripping project

Go through all the packages in Stripping and look at the release.notes. If there’re any updates since the last release then you add in the tag header with the incremented version. Increment the version in CMakeLists. For StrippingSys: The release notes should be the collection of the release notes for all packages that have changed since the last release. Add the tag header and increment the version in CMakeLists.txt. In the top directory of Stripping, set the version of Stripping in CMakeLists.txt and the version of Phys on which it depends in CMakeLists.txt. Push the changes, go to the Stripping gitlab page, click on "tags" and then "new tag". Name the tag, give a short message for the tag, select the branch from which the tag should be made, and copy the release notes of StrippingSys into the release notes of the tag. In case you need to re-tag, go to the "tags" page and click "delete tag" on which ever tag you want to replace, then remake the tag as above.

Revision 822018-10-09 - NicolaAnneSkidmore

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 387 to 387
 

Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)

Changed:
<
<
This is not compiling if you use git lb-checkout. Clone DaVinci properly!
>
>
This is not compiling if you use git lb-checkout. Clone DaVinci properly like this
git clone ssh://git@gitlab.cern.ch:7999/lhcb/Project.git
cd Project
lb-project-init
# use '-b' only if that branch does not exist in the remote repository
git checkout -b JIRATICKET-123
make
make test
# edit edit edit...
git add Some/Package/src/MyStuff.cpp
git commit -m 'fixing feature abc (JIRATICKET-123)'
vim Some/Package/src/MyStuff.cpp
make
make test
git commit -a -m 'improved fix to JIRATICKET-123'
git push -u origin JIRATICKET-123
  First, update the stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options. To check which functors has not been cached, do

Revision 812018-10-01 - NicolaAnneSkidmore

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 495 to 495
 
  • Prepare build environment for DaVinci version used in stripping production:
Changed:
<
<
> lb-dev DaVinci vVVrRpP
>
>
> lb-dev DaVinci vVVrRpP > cd DaVinciDev_vVVrRpP
 
  • For DaVinci versions prior to v36r0, it was necessary to getpack HltDAQ package, copy HltGenConfig algorithms from Hlt/HltCommon to HltDAQ and build it:
            > getpack Hlt/HltDAQ
Line: 514 to 515
  > cd strippingXX > cp $STRIPPINGSELECTIONSROOT/tests/coordinators/documentation/* .
Changed:
<
<
  • Edit run.sh, make_html.sh and tck.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation). Look up the TCK in
    $APPCONFIGOPTS/DaVinci/DV-Stripping24r0p1-Stripping.py
    or just concatenate the DV and Stripping versions, e.g. for DV v38r1p4 and Stripping 24r0p1, this results in 0x38142401.
>
>
  • Edit run.sh, make_html.sh and tck.py to use the correct AppConfig options (found in the production steps), DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation). Look up the TCK in
    $APPCONFIGOPTS/DaVinci/DV-StrippingXX-Stripping.py
    or just concatenate the DV and Stripping versions, e.g. for DV v38r1p4 and Stripping 24r0p1, this results in 0x38142401.
 
  • Run the ./run.sh script which will run 10000 events (to make sure most on-demand services are initialised) and produce the stripping config in the file config.tar

Revision 802018-05-25 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 82 to 82
  lb-run LHCbDirac /prod dirac-bookkeeping-file-metadata lfn gives you the number of events, run number, DQ flag amongst others
Added:
>
>
lb-run LHCbDirac /prod dirac-bookkeeping-file-path --LFN lfn gives you the actual location (useful for freezer cleaning)
 lb-run LHCbDirac /prod dirac-dms-lfn-metadata lfn lists the metadata of the file (owner, size, ...)

lb-run LHCbDirac /prod dirac-dms-lfn-replicas lfn shows the existing replicas of the file

Revision 792018-03-27 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 172 to 172
 git lb-checkout Stripping/Stripping Phys/StrippingSelections make configure && make
Changed:
<
<
In<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py you might want to change LocalFile to DiracFile.
>
>
In<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py you might want to change LocalFile to DiracFile and check the platform.
  Start ganga and for example for S29 on test data Reco16_Run18259

Revision 782018-02-13 - MichaelAlexander

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 22 to 22
  git fetch origin stripping24r1p1-patches git checkout origin/stripping24r1p1-patches -- Phys/StrippingSettings/python/StrippingSettings/Stripping24r1p1/LineConfigDictionaries_Calib.py Phys/StrippingSettings/dbase/stripping24r1p1
Added:
>
>

Fix a messed up history in git

If a merge or commit has been pushed to the repository incorrectly, you can undo it (in a full checkout of the package) with

git revert <commit hash>

You can get the commit hash from `git log`. If it's a merge commit then you need to specify the parent with

git revert -m <parent ID> <commit hash>

In the output of `git log` for a merge commit you'll see the two parent commits that were merged on the "Merge:" line, so normally you'll want to pick 1 or 2.

However, `git revert` changes the files back to the way they were before the commit, but it doesn't revert their history - the change back is logged as a new edit in the history. If you want also to revert the history then you need to find the hash of the commit before the unwanted change and do

git reset <commit hash>

which resets the tree to that commit, then check what's being reverted with

git status

Then revert it with

git reset --hard

Check that the history is as you expect with `git log`, then force push to the remote repo with

git push -f <remote> <branch>

The history on the remote branch will now also be rewritten. Note that gitlab doesn't let you force push to protected branches, so if the target branch is protected you'll need to remove its protection temporarily to do the force push and then re-protect it.

This works if you want to remove all commits after a certain point. If you only want to remove selected commits then instead of `git reset ` and then `git reset --hard`, you want to do

git rebase -i <commit hash>

again the commit hash here is that of the commit before the earliest commit you want to revert. This will open an editor with a list of all commits after that point. Delete the lines of all the commits you want to remove and then save and quit the editor. Then check that the history is OK with `git log`, and force push to the remote branch.

 

How to get the xml file from RDSTs

lhcb-proxy-init

Revision 772018-01-22 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 472 to 472
 
        > ./make_html.sh
Changed:
<
<
  • Copy html/strippingXX to /afs/cern.ch/lhcb/software/releases/DOC/stripping/config/ and edit /afs/cern.ch/lhcb/software/releases/DOC/stripping/maindesc.html.
>
>
  • cp -r html/strippingXX $LHCBDOC/stripping/config/ and edit $LHCBDOC/stripping/maindesc.html.
 
META FILEATTACHMENT attachment="check_tuples.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="check_tuples.py.txt" path="check_tuples.py.txt" size="2137" user="lcarson" version="1"
META FILEATTACHMENT attachment="DV-Stripping20r0p3-Indep.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="DV-Stripping20r0p3-Indep.py.txt" path="DV-Stripping20r0p3-Indep.py.txt" size="5494" user="lcarson" version="1"

Revision 762017-12-18 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 141 to 141
  to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script. You need to change the WG and Stripping, and possibly year and tags. If you want to run a smaller test of e.g. 100k, set EvtMax accordingly. It is advisable to create separate folders for each WG to avoid that the output DSTs get overwritten (you will need them later).
Changed:
<
<
It is advisable to run the tests in a tmux session by setting the following alias `alias ptmux="pagsh.krb -c 'kinit && tmux'" first. This way your kerberos token will not expire afte a couple of hours.
>
>
It is advisable to run the tests in a tmux session by setting the following alias alias ptmux="pagsh.krb -c 'kinit && tmux'" first. This way your kerberos token will not expire afte a couple of hours.
  FYI SL corresponds to Semileptonic, Calib to ALL and BnoC to Charmless.

Revision 752017-12-07 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 337 to 337
  This is not compiling if you use git lb-checkout. Clone DaVinci properly!
Changed:
<
<
Update stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options To check which functors has not been cached, do
>
>
First, update the stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options. To check which functors has not been cached, do
 
export LOKI_DISABLE_PYTHON=1
Changed:
<
<
before testing. This is included in $DAVINCITESTSROOT/tests/options/DisableLoKiPythonFunctors.py. To test, do
>
>
before testing. This is already included in $DAVINCITESTSROOT/tests/options/DisableLoKiPythonFunctors.py. Just make sure this option file is included in the qmtest, which you need to create under DaVinciTests/tests/qmtest/stripping.qms/. You might need to create an empty file for the reference, otherwise it won't be created.

To test, do

 
make test ARGS='-N'
make test ARGS='-R DaVinciTests.stripping.<name of your test from the command before>'
Changed:
<
<
This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well. The test results are in build/Testing/Temporary/LastTest_*.log. You might need to create an empty file for the reference, otherwise it won't be created.
>
>
This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well. The test results are in build/Testing/Temporary/LastTest_*.log.
  Tricks: The tests under DaVinci/DaVinciTests/tests/qmtest/stripping.qms/ need to included in StrippingCache/CMakeList.txt. The nightlies configuration can be found on gitlab, e.g. here. In the nightlies, always the latest version of TMVAWeights is used, which is why v999 appears in the nightlies tests.

Revision 742017-12-04 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 56 to 58
 
sc_seq = sc.sequence()
sc_seq.RequireObjects = ['DAQ/RawEvent']
Changed:
<
<
DaVinci().appendToMainSequence( [ sc_seq ] )
>
>
DaVinci().appendToMainSequence( [ sc_seq ] )
 

Prerequisites

Line: 199 to 199
  After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists.It is very important that all WGs check that the lines they need are included, and report back if anything is missing.
Added:
>
>
Known lines with higher retentions than usually allowed are (justified):
 

Run line independence tests

These should be run once the stripping selections are frozen. Assuming that the lines for 2011 and 2012 data are the same as each other, only one year needs to be checked. The scripts to run these tests are attached to this TWiki page. (The TWiki software has appended ".txt" to all of these filenames for some reason, this should be removed obviously)

Revision 732017-12-04 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 13 to 13
 

GIT basics

Changed:
<
<
All the basic information and links can be found in https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb
>
>
All basic information and links can be found in https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb.

To checkout single files from a different branch than the branch you're currently working on, say stripping24r1p1-patches, do

 git fetch origin stripping24r1p1-patches
 git checkout origin/stripping24r1p1-patches -- Phys/StrippingSettings/python/StrippingSettings/Stripping24r1p1/LineConfigDictionaries_Calib.py Phys/StrippingSettings/dbase/stripping24r1p1
 

How to get the xml file from RDSTs

Line: 35 to 40
  Please note that you have to write RealData instead of Real Data.
Added:
>
>

Inspect a DST file with Bender

For example if you'd like to check the Stripping decision of the following MDST file /lhcb/validation/Collision15/BHADRON.MDST/00067793/0000/00067793_00000191_1.bhadron.mdst

lb-run LHCbDirac/latest dirac-bookkeeping-genXMLCatalog -l /lhcb/validation/Collision15/BHADRON.MDST/00067793/0000/00067793_00000191_1.bhadron.mdst
lb-run Bender/latest bender -x pool_xml_catalog.xml -l /lhcb/validation/Collision15/BHADRON.MDST/00067793/0000/00067793_00000191_1.bhadron.mdst

In Bender, seekStripDecision('StrippingZVTOP_High*') will give you the decision for the lines you're interested in. To look at the candidates, use get('Bhadron/Phys/CONTAINER/Particles').

Require RawEvent

If certain raw banks are missing for a single event, which should not be the case, replace DaVinci().appendToMainSequence( [ sc.sequence() ] ) with

sc_seq = sc.sequence()
sc_seq.RequireObjects = ['DAQ/RawEvent']
DaVinci().appendToMainSequence( [ sc_seq ] )
 

Prerequisites

Stripping coordinators need the following rights:

Line: 82 to 106
  cd .. done
Added:
>
>
FYI, you might need to update the versions of Phys and Analysis in DaVinci or Stripping if you only clone these two packages. The version of Stripping has to match in Stripping and DaVinci.
 

Actions after stripping selections deadline

After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings.

Line: 105 to 131
 j = make_stripping_job('S29-loose', '/DaVinciDev_HEAD/', '/DaVinciDev_HEAD/Phys/StrippingSelections/tests/data/Reco16_Run182594.py', nfiles = -1, reportfreq = 50000, evtmax = 1000000, dirac = True, Version = 'Stripping29', SplitStreamsByWG = True) j.submit()
Changed:
<
<
to run the retention tests.
>
>
to run the retention tests. ./Phys/StrippingSelections/tests/coordinators/StrippingReport.py -d DIRECTORY=searches for all strippingreport.py’s and adds them together. The retention table for the twiki can be obtained via =./Phys/StrippingSelections/tests/coordinators/StrippingReport.py -d DIRECTORY -r --hlt2rate RATE > retentions.txt, which uses a single output file at the moment.
 

Run retention tests (locally)

Run

Line: 265 to 292
  cd tests/refs mv *.ref.new *.ref
Added:
>
>
If the list of trigger lines has to be updated, you can do lb-run Moore/latest Phys/StrippingSelections/tests/coordinators/get_hlt_lines.py --tck 0x11431621 --outputfile 0x11431621.py. At some point during Run 2, the following notation was introduced: 0x1 are HLT1 TCKs, 0x2 are HLT2, and 0x5 and 0x6 are for MC.
 

Prepare StrippingArchive, CommonParticlesArchive and StrippingSettings database

StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections, which are used only for development. CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified and lives in Phys now.

Line: 309 to 338
 
export LOKI_DISABLE_PYTHON=1
Changed:
<
<
before testing. To test, do
>
>
before testing. This is included in $DAVINCITESTSROOT/tests/options/DisableLoKiPythonFunctors.py. To test, do
 
make test ARGS='-N'
Line: 317 to 346
  This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well. The test results are in build/Testing/Temporary/LastTest_*.log. You might need to create an empty file for the reference, otherwise it won't be created.
Added:
>
>
Tricks: The tests under DaVinci/DaVinciTests/tests/qmtest/stripping.qms/ need to included in StrippingCache/CMakeList.txt. The nightlies configuration can be found on gitlab, e.g. here. In the nightlies, always the latest version of TMVAWeights is used, which is why v999 appears in the nightlies tests.
 

Tag packages in Stripping project

Go through all the packages in Stripping and look at the release.notes. If there’re any updates since the last release then you add in the tag header with the incremented version. Increment the version in CMakeLists. For StrippingSys: The release notes should be the collection of the release notes for all packages that have changed since the last release. Add the tag header and increment the version in CMakeLists.txt. In the top directory of Stripping, set the version of Stripping in CMakeLists.txt and the version of Phys on which it depends in CMakeLists.txt. Push the changes, go to the Stripping gitlab page, click on "tags" and then "new tag". Name the tag, give a short message for the tag, select the branch from which the tag should be made, and copy the release notes of StrippingSys into the release notes of the tag. In case you need to re-tag, go to the "tags" page and click "delete tag" on which ever tag you want to replace, then remake the tag as above.

Added:
>
>
To check if a tag exists, you can grep the CMakeLists.txt, e.g. grep v4r19 $LHCb_release_area/STRIPPING/STRIPPING_v*/Phys/StrippingSelections/CMakeLists.txt or use cmt/requirements if you're looking at older versions.
 

Tag packages in DaVinci project

The only package that needs to be tagged by the Stripping coordinators is the StrippingCache which lives in DaVinci. Follow the same steps as for the previous step.

Line: 368 to 401
  Go to https://lhcb-portal-dirac.cern.ch/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert (currently Marco Cattaneo and Liang Sun). Test the steps before marking them as "Ready".
Changed:
<
<
To test the steps, you should follow the instructions given here. You might need to test different versions of LHCbDirac until you find a working one. The test will create a Local_* folder, in which you find prodConf_*.py. You should remove the raw file from the list of inputs, potentially edit the number of events, and depending on where you test prodConf_*.py, add the full path to the xml catalogue.
>
>
To test the steps, you should follow the instructions given here. You might need to test different versions of LHCbDirac until you find a working one. The test will create a Local_* folder, in which you find prodConf_*.py. You should remove the raw file from the list of inputs, potentially edit the number of events, and depending on where you test prodConf_*.py, add the full path to the xml catalogue.
 

Run validation

Added:
>
>
Announce the steps on the operations elog, and send an email to the data managers and production managers asking for a validation giving a suitable run range. The typical size of a validation comprises a couple thousand files.
 

Run production

Debugging tricks for problems in production

Revision 722017-11-30 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 23 to 23
 

Useful dirac commands

Changed:
<
<
lb-run LHCbDirac /prod dirac-bookkeeping-file-metadata gives you the number of events, run number, DQ flag amongst others
>
>
lb-run LHCbDirac /prod dirac-bookkeeping-file-metadata lfn gives you the number of events, run number, DQ flag amongst others
 
Changed:
<
<
lb-run LHCbDirac /prod dirac-dms-lfn-metadata lists the metadata of the file (owner, size, ...)
>
>
lb-run LHCbDirac /prod dirac-dms-lfn-metadata lfn lists the metadata of the file (owner, size, ...)
 
Changed:
<
<
lb-run LHCbDirac /prod dirac-dms-lfn-replicas shows the existing replicas of the file
>
>
lb-run LHCbDirac /prod dirac-dms-lfn-replicas lfn shows the existing replicas of the file
 
Changed:
<
<
lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/Collision//RealData/90000000/RAW --Trigger gives you the stable beam time
>
>
lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/CollisionYEAR/CONDITIONS/RealData/90000000/RAW --Trigger gives you the stable beam time
 
Changed:
<
<
lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/Collision//RealData///90000000/  --Trigger gives you the size of the stripped data
>
>
lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/CollisionYEAR/CONDITIONS/RealData///90000000/  --Trigger gives you the size of the stripped data

Please note that you have to write RealData instead of Real Data.

 

Prerequisites

Line: 366 to 368
  Go to https://lhcb-portal-dirac.cern.ch/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert (currently Marco Cattaneo and Liang Sun). Test the steps before marking them as "Ready".
Changed:
<
<
To test the steps, you should follow the instructions given here. You might need to test different versions of LHCbDirac until you find a working one. The test will create a Local_* folder, in which you find =prodConf_*.py. You should remove the raw file from the list of inputs, potentially edit the number of events, and depending on where you test prodConf_*.py, add the full path to the xml catalogue.
>
>
To test the steps, you should follow the instructions given here. You might need to test different versions of LHCbDirac until you find a working one. The test will create a Local_* folder, in which you find prodConf_*.py. You should remove the raw file from the list of inputs, potentially edit the number of events, and depending on where you test prodConf_*.py, add the full path to the xml catalogue.
 

Run validation

Revision 712017-11-29 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 363 to 365
 

Prepare production steps

Go to https://lhcb-portal-dirac.cern.ch/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert (currently Marco Cattaneo and Liang Sun). Test the steps before marking them as "Ready".

Added:
>
>
To test the steps, you should follow the instructions given here. You might need to test different versions of LHCbDirac until you find a working one. The test will create a Local_* folder, in which you find =prodConf_*.py. You should remove the raw file from the list of inputs, potentially edit the number of events, and depending on where you test prodConf_*.py, add the full path to the xml catalogue.
 

Run validation

Run production

Revision 702017-11-20 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 313 to 310
 make test ARGS='-N' make test ARGS='-R DaVinciTests.stripping.'
Changed:
<
<
This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well. The test results are in build/Testing/Temporary/LastTest_*.log.
>
>
This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well. The test results are in build/Testing/Temporary/LastTest_*.log. You might need to create an empty file for the reference, otherwise it won't be created.
 

Tag packages in Stripping project

Revision 692017-11-20 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 310 to 313
 make test ARGS='-N' make test ARGS='-R DaVinciTests.stripping.'
Changed:
<
<
This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well.
>
>
This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well. The test results are in build/Testing/Temporary/LastTest_*.log.
 

Tag packages in Stripping project

Revision 682017-11-17 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 300 to 299
 

Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)

Changed:
<
<
Update stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options To check which functors has not been cached, do
>
>
This is not compiling if you use git lb-checkout. Clone DaVinci properly!
 
Added:
>
>
Update stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options To check which functors has not been cached, do
 
export LOKI_DISABLE_PYTHON=1
Line: 306 to 306
 export LOKI_DISABLE_PYTHON=1

before testing. To test, do

Deleted:
<
<
 
make test ARGS='-N'
Changed:
<
<
make test ARGS='-R DaVinciTests.stripping.'
>
>
make test ARGS='-R DaVinciTests.stripping.'
  This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well.

Revision 672017-11-17 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 30 to 29
  lb-run LHCbDirac /prod dirac-dms-lfn-replicas shows the existing replicas of the file
Added:
>
>
lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/Collision//RealData/90000000/RAW --Trigger gives you the stable beam time

lb-run LHCbDirac /prod dirac-bookkeeping-get-stats -B /LHCb/Collision//RealData///90000000/  --Trigger gives you the size of the stripped data

 

Prerequisites

Stripping coordinators need the following rights:

Line: 106 to 111
 
gaudirun.py -T TestFromSettings_all.py path_to_input_data 
Changed:
<
<
to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script. You need to change the WG and Stripping, and possibly year and tags. If you want to run a smaller test of e.g. 100k, set EvtMax accordingly. It is advisable to create separate folders for each WG to avoid that the output DSTs get overwritten (you will need them later)
>
>
to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script. You need to change the WG and Stripping, and possibly year and tags. If you want to run a smaller test of e.g. 100k, set EvtMax accordingly. It is advisable to create separate folders for each WG to avoid that the output DSTs get overwritten (you will need them later).

It is advisable to run the tests in a tmux session by setting the following alias `alias ptmux="pagsh.krb -c 'kinit && tmux'" first. This way your kerberos token will not expire afte a couple of hours.

FYI SL corresponds to Semileptonic, Calib to ALL and BnoC to Charmless.

 

Run memory use tests

Line: 295 to 305
 
export LOKI_DISABLE_PYTHON=1
Changed:
<
<
before testing
>
>
before testing. To test, do

make test ARGS='-N'
make test ARGS='-R DaVinciTests.stripping.<name of your test from the command before>'

This will create the file DaVinciTests/tests/refs/.ref, which you might want to rename and have to commit as well.

 

Tag packages in Stripping project

Revision 662017-11-16 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 276 to 274
 assuming that the chages you want are in a branch called Stripping. The latter command cretaes or updates StrippingArchive, CommonParticlesArchive and if necessary the StrippingSettings dbase.

The dbase file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory:

Added:
>
>
 
git clone ssh://git@gitlab.cern.ch:7999/lhcb/Stripping.git
Added:
>
>
cd Stripping
 git checkout cd Stripping/Phys/StrippingSettings/dbase/ cp from/the/path/where/it/is/strippingXX .

Revision 652017-09-19 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 18 to 17
 

How to get the xml file from RDSTs

Deleted:
<
<
SetupLHCbDirac
 lhcb-proxy-init
Changed:
<
<
dirac-bookkeeping-genXMLCatalog --Options=YourData.py --Catalog=yourdatacatalog.xml --Depth=2
>
>
lb-run LHCbDirac/prod dirac-bookkeeping-genXMLCatalog --Options=YourData.py --Catalog=yourdatacatalog.xml --Depth=2

Useful dirac commands

lb-run LHCbDirac/prod dirac-bookkeeping-file-metadata gives you the number of events, run number, DQ flag amongst others

lb-run LHCbDirac/prod dirac-dms-lfn-metadata lists the metadata of the file (owner, size, ...)

lb-run LHCbDirac/prod dirac-dms-lfn-replicas shows the existing replicas of the file

 

Prerequisites

Revision 642017-08-10 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 39 to 40
  Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed. It can make sense to take (parts of) several runs as the test sample if the running conditions or trigger configuration was changing significantly. Once the run for the test sample has been identified, inform bookkeeping manager and ask to copy it to "freezer". The python options have to be prepared and put to Phys/StrippingSelections/tests/data/.
Added:
>
>

Prepare patch for DaVinci

For incremental and IFT (re-)strippings, DaVinci patches need to be prepared. For incremental re-strippings, the patch should be created from the DaVinci version corresponding to the stripping version and for IFT, make sure you patch from the DaVinci version using the correct Reco version.

If you need to change the environment, do

. LbLogin.sh -c x86_64-slc6-gcc49-opt

or run with

lb-run -c x86_64-slc6-gcc49-opt

To build the patch, you can do


export CMTPROJECTPATH=${CMTPROJECTPATH/$User_release_area/$PWD}
export User_release_area=$PWD
export LBCONFIGURATIONROOT="/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_vXrY/LbConfiguration"

for project in Phys Stripping Analysis DaVinci ; do
    git clone "ssh://git@gitlab.cern.ch:7999/lhcb/${project}.git"
    cd "$project"
    lb-project-init
    git fetch
    git checkout <your_branch>
    make configure >& stdout-make-configure
    make install >& stdout-make-install
    cd ..
done 
 

Actions after stripping selections deadline

After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings.

Line: 157 to 190
 from another terminal it immediately continued.

Once gdb has started, turn on logging with

Added:
>
>
 
Changed:
<
<
set logging on
>
>
set logging on
  then start the job with run. You can then interrupt whenever you like with ctrl-c, or if you're investigating a crash you can just wait for it, then you can use backtrace to get the method call backtrace. Use continue to continue the job after an interrupt.
Line: 168 to 202
 print (class CombineParticles)0x614b8940

or

Deleted:
<
<
 
Changed:
<
<
print *('DaVinci::N4BodyDecays' *) 0x5dc58cf0
>
>
print *('DaVinci::N4BodyDecays' *) 0x5dc58cf0
  (note the single quotes for objects in a namespace) then you can search the output, of which there will likely be a lot. If you're trying to work out which line a problematic object originates from, search the output for anything containing 'Line' or 'Stripping'. Then you can have a look at the module that the line belongs to.
Line: 176 to 208
 (note the single quotes for objects in a namespace) then you can search the output, of which there will likely be a lot. If you're trying to work out which line a problematic object originates from, search the output for anything containing 'Line' or 'Stripping'. Then you can have a look at the module that the line belongs to.

If you're trying to catch some output, you can set a break point for any output to console with

Deleted:
<
<
 
Changed:
<
<
b write if 1==$rdi
>
>
b write if 1==$rdi
  or
Deleted:
<
<
 
Changed:
<
<
b fwrite if $rcx==&_IO_2_1_stdout_
>
>
b fwrite if $rcx==&_IO_2_1_stdout_
 

Release Stripping project

Changed:
<
<
Help on release procedure is at PrepareProjectReleaseGit page. Please check the steps following the instructions on ProdReqPreLaunchTest.
>
>
Help on release procedure is at PrepareProjectReleaseGit page. Please check the steps following the instructions on ProdReqPreLaunchTest. For full DST productions, no ancestors need to be included.

If the above instructions fail for RDST files, remove the RAW from the generated ProdConf*.py and run with the other options locally.

 

Check nightlies and correct if needed

Line: 271 to 298
  The only package that needs to be tagged by the Stripping coordinators is the StrippingCache which lives in DaVinci. Follow the same steps as for the previous step.
Deleted:
<
<

Tag StrippingSys

Use tag collector, modify cmt/requirements to use the correct versions of the Stripping packages tagged at the previous step, add sensible release notes (summary of changes from all packages).

 

Tag extra packages in PARAM

Packages like TMVAWeights required by some RelatedInfoTools live in PARAM and might also need to be tagged.

Line: 286 to 310
  Then open a new JIRA task in LHCBDEP asking for deployment.
Deleted:
<
<

Update dependencies in Stripping project

  cd <where you have plenty of space>
  getpack -P -H STRIPPING HEAD
  cd STRIPPING/STRIPPING_HEAD/
  emacs cmt/project.cmt # Update dependencies here
  emacs CMakeLists.txt # Update dependencies
  svn commit -m vXrY
  tag_package -P Stripping vXrY

Check you picked up the correct packages by issuing

svnCheckTags StrippingSys vXrY --diffs

Tag stripping project using gitlab

Modify the CMakeLists.txt file under the project. The file project.cmt under cmt and the CMakeLists.txt file under StrippingSys. release.notes under doc and requirements file under StrippingSys /cmt Go to https://gitlab.cern.ch/lhcb/Stripping and sign in. Under Repository/Tags click on "New Tag" Add the Tag Name, and Release Notes that you want to appear (usually a summary of the changes in the package)

 

Add Stripping project to lhcb-prerelease slot in the nightlies

Go to nightlies page, open "Configuration Editor", choose the slot needed (most likely lhcb-prerelease), double-click on Stripping tag and change the STRIPPING version there. Make sure "disabled" flag is "false".

Line: 320 to 322
 
  • Change stripping version used to retrieve configuration from StrippingSettings and StrippingArchive.
  • Stripping TCK has to be changed. Current convention is the following: TCK = 0xVVVVSSSS, where VVVV is the DaVinci version, and SSSS is the Stripping number (e.g. for Stripping20r1p1 running on DaVinci v32r2p5 TCK = 0x32252011).
  • If necessary create DataType-<yyyy>.py, where <yyyy> corresponds to the year.
Changed:
<
<
If the stripping configuration is expected to run in MC, the corresponding MC stripping options (DV-StrippingXX-Stripping-MC*.py) have to be updated.
>
>
If the stripping configuration is expected to run in MC, the corresponding MC stripping options (DV-StrippingXX-Stripping-MC*.py) have to be updated. Please note that AppConfig still lives in svn.
 
Changed:
<
<

Ask for release in Jira

>
>
In the case that only a patch of AppConfig needs to be deployed, prepare the tag as as usual: set the version number in cmt/requirements, and add in the release header in doc/release.notes, and commit. Then tag the package as usual with tag_package AppConfig vXrY, at which point it should appear here. No build is required as AppConfig only contains options files, so you can request deployment in JIRA, an example request is here. That's all!
 
Changed:
<
<
Ask Stripping, DaVinci and AppConfig to be released using JIRA. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).
>
>

Trigger a release build of DaVinci

 
Changed:
<
<
In the case that only a patch of AppConfig needs to be deployed, prepare the tag as as usual: set the version number in cmt/requirements, and add in the release header in doc/release.notes, and commit. Then tag the package as usual with tag_package AppConfig vXrY, at which point it should appear here. No build is required as AppConfig only contains options files, so you can request deployment in JIRA, an example request is here. That's all!
>
>
For minor changes, eg. changing the Stripping version picked up by DaVinci, we are allowed to create a tag for DaVinci, which follows the same procedure as the Stripping. However, we cannot delete a tag from DaVinci if we mess up! After creating the Stripping and DaVinci tag,
 
Changed:
<
<

Prepare production steps

>
>
> lb-sdb-import DaVinci vXrY
 
Changed:
<
<
Go to https://lhcbweb.pic.es/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert (currently Marco Cattaneo and Liang Sun).
>
>
triggers a release build, which you should check here. If the builds are ok, you can go ahead and request deployment of DaVinci citing the number of the release build (see below). Please note that it can take around 30min before the release build appears on the website.
 
Changed:
<
<

Hide project and declare it released

>
>

Ask for release in Jira

Ask Stripping, DaVinci and AppConfig to be released using JIRA. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).

 
Changed:
<
<
Go to the LHCb tag collector. Check again for possible lost tags. Once they are solved, and in the home of the tag collector look for the Stripping project. If needed, click the Edit button and changes the specifications. Click the Hide button and say yes to release the project.
>
>

Prepare production steps

 
Added:
>
>
Go to https://lhcb-portal-dirac.cern.ch/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert (currently Marco Cattaneo and Liang Sun). Test the steps before marking them as "Ready".
 

Run validation

Run production

Added:
>
>

Debugging tricks for problems in production

To see all algorithms running, add

from Configurables import NameAuditor, AuditorSvc
AuditorSvc ().Auditors.append(NameAuditor ())

to your options and redirect the output to tmp as the logfile might become several GB large.

The timings of each algorithms for each event can be written by adding

from Configurables import TimingAuditor, AuditorSvc, ChronoStatSvc, SequencerTimerTool, ChronoAuditor
from Gaudi.Configuration import INFO

TimingAuditor ().addTool(SequencerTimerTool,name="TIMER")
TimingAuditor ().TIMER.NameSize = 60
TimingAuditor ().OutputLevel = INFO
AuditorSvc ().Auditors.append(ChronoAuditor ("Chrono"))
ChronoStatSvc ().PerEventFile = 'eventtimings.txt' 

to your options (copy the Stripping options from AppConfog and replace the existing TimingAuditor with the lines above). In 'eventtimings.txt', all algorithms are listed with the respective timings. Please keep in mind that not all algorithms are being run for each event.

 

Prepare web documentation

Reference documentation about all lines (their cuts, sequence of algorithms, streams, output locations) and standard particles is available at the Stripping project webpages. These pages are generated automatically using the HltGenConfig algorithm (the same one that is used to generate HltTCK). Here are the instructions on generating these pages:

Revision 632017-07-25 - MichaelAlexander

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 156 to 156
  from another terminal it immediately continued.
Changed:
<
<
Once gdb has started, start the job with run. You can then interrupt whenever you like with ctrl-c, or if you're investigating a crash you can just wait for it, then you can use backtrace to get the method call backtrace. Use continue to continue the job after an interrupt.
>
>
Once gdb has started, turn on logging with
set logging on

then start the job with run. You can then interrupt whenever you like with ctrl-c, or if you're investigating a crash you can just wait for it, then you can use backtrace to get the method call backtrace. Use continue to continue the job after an interrupt.

  Once you have the backtrace you can print info on an object given its address with, eg
print *(class CombineParticles*)0x614b8940
Changed:
<
<
then you can search the output, of which there will likely be a lot. If you're trying to work out which line a problematic object originates from, search the output for anything containing 'Line' or 'Stripping'. Then you can have a look at the module that the line belongs to.
>
>
or

print *('DaVinci::N4BodyDecays' *) 0x5dc58cf0

(note the single quotes for objects in a namespace) then you can search the output, of which there will likely be a lot. If you're trying to work out which line a problematic object originates from, search the output for anything containing 'Line' or 'Stripping'. Then you can have a look at the module that the line belongs to.

If you're trying to catch some output, you can set a break point for any output to console with

b write if 1==$rdi

or

b fwrite if $rcx==&_IO_2_1_stdout_
 

Release Stripping project

Revision 622017-07-05 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 242 to 242
 

Tag packages in Stripping project

Changed:
<
<
Go through all the packages in Stripping and look at the release.notes. If there’re any updates since the last release then you add in the tag header with the incremented version. Increment the version in CMakeLists and cmt/requirements. For StrippingSys: The release notes should be the collection of the release notes for all packages that have changed since the last release. Add the tag header and increment the version in CMakeLists.txt and cmt/requirements. In the top directory of Stripping, set the version of Stripping in CMakeLists.txt and the version of Phys on which it depends in both CMakeLists.txt and cmt/project.cmt. Push the changes, go to the Stripping gitlab page, click on "tags" and then "new tag". Name the tag, give a short message for the tag, select the branch from which the tag should be made, and copy the release notes of StrippingSys into the release notes of the tag. In case you need to re-tag, go to the "tags" page and click "delete tag" on which ever tag you want to replace, then remake the tag as above.

Then use the tag collector. Go to "Tag package" and tag each package of the Stripping project which was modified since the previous release (no need to tag packages without changes).
>
>
Go through all the packages in Stripping and look at the release.notes. If there’re any updates since the last release then you add in the tag header with the incremented version. Increment the version in CMakeLists. For StrippingSys: The release notes should be the collection of the release notes for all packages that have changed since the last release. Add the tag header and increment the version in CMakeLists.txt. In the top directory of Stripping, set the version of Stripping in CMakeLists.txt and the version of Phys on which it depends in CMakeLists.txt. Push the changes, go to the Stripping gitlab page, click on "tags" and then "new tag". Name the tag, give a short message for the tag, select the branch from which the tag should be made, and copy the release notes of StrippingSys into the release notes of the tag. In case you need to re-tag, go to the "tags" page and click "delete tag" on which ever tag you want to replace, then remake the tag as above.

 

Tag packages in DaVinci project

Revision 612017-07-04 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 220 to 220
 The dbase file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory:
git clone ssh://git@gitlab.cern.ch:7999/lhcb/Stripping.git
Added:
>
>
git checkout
 cd Stripping/Phys/StrippingSettings/dbase/ cp from/the/path/where/it/is/strippingXX . git add strippingXX

Revision 602017-06-13 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 298 to 295
 The stripping options are in AppConfig/options/DaVinci/DV-StrippingXX-Stripping.py. In most cases the DaVinci options for stripping can be copied from the previous round, with minor changes:
  • Change stripping version used to retrieve configuration from StrippingSettings and StrippingArchive.
  • Stripping TCK has to be changed. Current convention is the following: TCK = 0xVVVVSSSS, where VVVV is the DaVinci version, and SSSS is the Stripping number (e.g. for Stripping20r1p1 running on DaVinci v32r2p5 TCK = 0x32252011).
Changed:
<
<
  • Make sure DaVinci ().DataType is correct ('2011', '2012', etc.).
>
>
  • If necessary create DataType-<yyyy>.py, where <yyyy> corresponds to the year.
 If the stripping configuration is expected to run in MC, the corresponding MC stripping options (DV-StrippingXX-Stripping-MC*.py) have to be updated.

Ask for release in Jira

Revision 592017-06-12 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 351 to 354
 
        > ./make_html.sh
Changed:
<
<
  • Copy html/strippingXX to /afs/cern.ch/lhcb/software/releases/DOC/stripping/config/ and edit /afs/cern.ch/lhcb/software/releases/DOC/stripping/maindesc.html.</verbatim>
>
>
  • Copy html/strippingXX to /afs/cern.ch/lhcb/software/releases/DOC/stripping/config/ and edit /afs/cern.ch/lhcb/software/releases/DOC/stripping/maindesc.html.
 
META FILEATTACHMENT attachment="check_tuples.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="check_tuples.py.txt" path="check_tuples.py.txt" size="2137" user="lcarson" version="1"
META FILEATTACHMENT attachment="DV-Stripping20r0p3-Indep.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="DV-Stripping20r0p3-Indep.py.txt" path="DV-Stripping20r0p3-Indep.py.txt" size="5494" user="lcarson" version="1"

Revision 582017-06-12 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 251 to 251
  Use tag collector, modify cmt/requirements to use the correct versions of the Stripping packages tagged at the previous step, add sensible release notes (summary of changes from all packages).
Added:
>
>

Tag extra packages in PARAM

Packages like TMVAWeights required by some RelatedInfoTools live in PARAM and might also need to be tagged.

getpack TMVAWeights h
<modify release notes and update versions wherever required>
svn ci -m "releasing new version vXrY"
tag_package TMVAWeights vXrY

Then open a new JIRA task in LHCBDEP asking for deployment.

 

Update dependencies in Stripping project

  cd <where you have plenty of space>

Revision 572017-06-09 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 197 to 197
  cd tests/refs mv *.ref.new *.ref
Changed:
<
<

Prepare StrippingSettings database

>
>

Prepare StrippingArchive, CommonParticlesArchive and StrippingSettings database

StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections, which are used only for development. CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified and lives in Phys now.

Get DaVinci from the nightlies eg. for the branch (e.g. 2017-patches) and the build (e.g. Tue for Tuesday) do

 
Changed:
<
<
cd Phys/StrippingSettings cd cmt cmt make cd ../python/StrippingSettings python makeDB.py StrippingXX mv stripping.tmp ../../dbase/strippingXX
>
>
cd lb-dev --nightly-cvmfs --nightly DaVinci/HEAD cd DaVinciDev_HEAD git lb-use Stripping git lb-checkout Stripping/Stripping Phys/StrippingSelections git lb-checkout Stripping/Stripping Phys/StrippingSettings git lb-checkout Stripping/Stripping Phys/StrippingArchive git lb-checkout Stripping/Stripping Phys/CommonParticlesArchive make configure && make

./run \$STRIPPINGSELECTIONSROOT/tests/coordinators/create_archives.py Stripping ""

 
Changed:
<
<
This file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory:
>
>
assuming that the chages you want are in a branch called Stripping. The latter command cretaes or updates StrippingArchive, CommonParticlesArchive and if necessary the StrippingSettings dbase.
 
Added:
>
>
The dbase file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory:
 
git clone ssh://git@gitlab.cern.ch:7999/lhcb/Stripping.git
cd Stripping/Phys/StrippingSettings/dbase/
Line: 216 to 226
 git commit strippingXX -m "update dbase for sXX" git push
Deleted:
<
<

Prepare StrippingArchive

StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections which are used only for development. Once the stripping selections are fixed, copy the contents of the StrippingSelections into StrippingArchive in the appropriate subdirectory (StrippingXX). It's better to do it using svn functionality to keep track of where the selections came from:

  svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/python/StrippingSelections/ svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingArchive/python/StrippingArchive/StrippingXX/ -m "Create StrippingXX"

This command will not show the currently added stripping folder. You need to update the local repository to see it.

svn update

Then add StrippingXX to __init__.py under StrippingArchive /python/StrippingArchive, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.

IMPORTANT: do not forget to modify the __init__.py under StrippingArchive /python/StrippingArchive/StrippingXX as explained in readme.txt

Prepare CommonParticlesArchive

CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified. To prepare the package a folder with the name StrippingXX needs to be created in CommonParticlesArchive and the contents of the *.py files in CommonParticles needs to be copied. if the full folder is copied, remember to cancel the hidden file .svn, otherwise will prevent the commit to happen.

To do it directly execute the command:

  svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/CommonParticles/python/CommonParticles/
svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/CommonParticlesArchive/python/CommonParticlesArchive/StrippingXX/ -m "Added archive for StrippingXX"

This will not update your local repository, so afterwards do

svn update

to see the changes.

 

Check lost tags in tag collector

Tag collector page is here. Go to "Lost tags" and add them if needed.

Revision 562017-06-07 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 44 to 43
  After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings.
Added:
>
>

Run retention tests (Grid)

Setup DaVinci from the nightlies for the branch (e.g. 2017-patches) and the build (e.g. Tue for Tuesday) do

cd <wherever>
lb-dev --nightly-cvmfs --nightly <yourbranch> <day> DaVinci/HEAD
cd DaVinciDev_HEAD
git lb-use Stripping
git lb-checkout Stripping/Stripping<XY> Phys/StrippingSelections
make configure && make

In<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py you might want to change LocalFile to DiracFile.

Start ganga and for example for S29 on test data Reco16_Run18259

execfile('<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/ganga/StrippingUtils.py')
j  = make_stripping_job('S29-loose', '<wherever>/DaVinciDev_HEAD/', '<wherever>/DaVinciDev_HEAD/Phys/StrippingSelections/tests/data/Reco16_Run182594.py', nfiles = -1, reportfreq = 50000, evtmax = 1000000, dirac = True, Version = 'Stripping29', SplitStreamsByWG = True)
j.submit() 

to run the retention tests.

 

Run retention tests (locally)

Run

Line: 315 to 341
  > cmt make These steps are NOT needed for new DaVinci (v36r0 and more recent) because HltGenConfig became available in DaVinci.
Changed:
<
<
  • Create an empty directory in a place with plenty of space, copy some scripts used for stripping configuration dump:
  • Do
    ./run bash
    , so the environment variables are set.
>
>
  • Create an empty directory in a place with plenty of space
  • Build and set the environment variables:
 
Added:
>
>
> make > ./run bash
  > mkdir strippingXX > cd strippingXX > cp $STRIPPINGSELECTIONSROOT/tests/coordinators/documentation/* .
Changed:
<
<
  • Edit run.sh, make_html.sh and tck.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation).
>
>
  • Edit run.sh, make_html.sh and tck.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation). Look up the TCK in
    $APPCONFIGOPTS/DaVinci/DV-Stripping24r0p1-Stripping.py
    or just concatenate the DV and Stripping versions, e.g. for DV v38r1p4 and Stripping 24r0p1, this results in 0x38142401.
 
  • Run the ./run.sh script which will run 10000 events (to make sure most on-demand services are initialised) and produce the stripping config in the file config.tar

Revision 552017-05-23 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 322 to 316
  These steps are NOT needed for new DaVinci (v36r0 and more recent) because HltGenConfig became available in DaVinci.

  • Create an empty directory in a place with plenty of space, copy some scripts used for stripping configuration dump:
Added:
>
>
  • Do
    ./run bash
    , so the environment variables are set.
 
        > mkdir strippingXX
        > cd strippingXX

Revision 542017-05-12 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 241 to 241
 

Tag packages in Stripping project

Changed:
<
<
The easiest way to tag packages is to use tag collector. Go to "Tag package" and tag each package of the Stripping project which was modified since the previous release (no need to tag packages without changes).
>
>
Go through all the packages in Stripping and look at the release.notes. If there’re any updates since the last release then you add in the tag header with the incremented version. Increment the version in CMakeLists and cmt/requirements. For StrippingSys: The release notes should be the collection of the release notes for all packages that have changed since the last release. Add the tag header and increment the version in CMakeLists.txt and cmt/requirements. In the top directory of Stripping, set the version of Stripping in CMakeLists.txt and the version of Phys on which it depends in both CMakeLists.txt and cmt/project.cmt. Push the changes, go to the Stripping gitlab page, click on "tags" and then "new tag". Name the tag, give a short message for the tag, select the branch from which the tag should be made, and copy the release notes of StrippingSys into the release notes of the tag. In case you need to re-tag, go to the "tags" page and click "delete tag" on which ever tag you want to replace, then remake the tag as above.

Then use the tag collector. Go to "Tag package" and tag each package of the Stripping project which was modified since the previous release (no need to tag packages without changes).
 

Tag packages in DaVinci project

Revision 532017-05-12 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 97 to 97
  LHCbApp().XMLSummary = "summary.xml"

importOptions("$STRIPPINGSELECTIONSROOT/tests/data/Reco15a_Run164668.py")

Changed:
<
<
  • In the summary.xml file, get the total number of events and the random triggers from the field _ IntegrateBeamCrossing /Random_.
  • For the run you've ran over e.g. Run164668 in the example above, look up the run in the RunDB, eg. http://lbrundb.cern.ch/rundb/run/164668/
  • From the RunDB, get the LumiBBRate from hlt2TriggerSettings.
  • Klick on the FILL (here 4449) and get the corresponding total ineffiency, which would be 0.1448 in this example.
  • The trigger rate in Hz is now calculated as events/(random triggers/1000/(LumiBBRate - total ineffiency (not in %))). For 2015, the trigger rate is around 8600 Hz.
>
>
  • In the summary.xml file, get the total number of events.
  • For the run you've ran over e.g. Run164668 in the example above, get the trigger rate from:
    dirac-bookkeeping-get-stats --BK /LHCb/Collision15//RealData//RAW --Run 164668 --Trigger
    (here: 11939.9 events/second).
  • The trigger rate can also be calculated from the number of events divided by the duration of the run. Here: 43'079'080 events/3600s, which is almost the same as the trigger rate above.
  • Don't bother about events/s and Hz.
  • Sometimes people ask for the number of stable beam seconds corresponding to the (sub)data set you ran over: Just divide the number of events you ran over by the trigger rate (here: 8.38s).
  After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists.It is very important that all WGs check that the lines they need are included, and report back if anything is missing.

Revision 522017-05-05 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 80 to 79
  and add these numbers to the correct streams of the corresponding WG. FTAG is called MDST in retentions_new.py.
Changed:
<
<
The trigger rate is obtained as follows

Run the follwing script to get the summary.xml.

>
>
The trigger rate is obtained as follows:
 
Added:
>
>
  • Run the follwing script to get the summary.xml (in this example for 2015).
 
from Configurables import DaVinci
from Gaudi.Configuration import importOptions
Line: 98 to 96
 from Configurables import LHCbApp LHCbApp().XMLSummary = "summary.xml"
Changed:
<
<
importOptions("$STRIPPINGSELECTIONSROOT/tests/data/Reco15a_Run164668.py")
>
>
importOptions("$STRIPPINGSELECTIONSROOT/tests/data/Reco15a_Run164668.py")
  • In the summary.xml file, get the total number of events and the random triggers from the field _ IntegrateBeamCrossing /Random_.
  • For the run you've ran over e.g. Run164668 in the example above, look up the run in the RunDB, eg. http://lbrundb.cern.ch/rundb/run/164668/
  • From the RunDB, get the LumiBBRate from hlt2TriggerSettings.
  • Klick on the FILL (here 4449) and get the corresponding total ineffiency, which would be 0.1448 in this example.
  • The trigger rate in Hz is now calculated as events/(random triggers/1000/(LumiBBRate - total ineffiency (not in %))). For 2015, the trigger rate is around 8600 Hz.
  After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists. It is very important that all WGs check that the lines they need are included, and report back if anything is missing.
Line: 147 to 146
 

Release Stripping project

Changed:
<
<
Help on release procedure is at PrepareProjectReleaseGit page.
>
>
Help on release procedure is at PrepareProjectReleaseGit page. Please check the steps following the instructions on ProdReqPreLaunchTest.
 

Check nightlies and correct if needed

Revision 512017-05-04 - MichaelAlexander

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 99 to 99
 LHCbApp().XMLSummary = "summary.xml"

importOptions("$STRIPPINGSELECTIONSROOT/tests/data/Reco15a_Run164668.py")

Changed:
<
<
>
>
  After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists. It is very important that all WGs check that the lines they need are included, and report back if anything is missing.

Revision 502017-05-04 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 79 to 80
  and add these numbers to the correct streams of the corresponding WG. FTAG is called MDST in retentions_new.py.
Added:
>
>
The trigger rate is obtained as follows

Run the follwing script to get the summary.xml.

from Configurables import DaVinci
from Gaudi.Configuration import importOptions

DaVinci().DataType = '2015'
DaVinci().Lumi = True
DaVinci().EvtMax = 100000

from Configurables import CondDB
CondDB().LatestGlobalTagByDataType = "2015"

from Configurables import LHCbApp
LHCbApp().XMLSummary = "summary.xml"

importOptions("$STRIPPINGSELECTIONSROOT/tests/data/Reco15a_Run164668.py")
<verbatim\>
</verbatim>
<nop>
 After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists. It is very important that all WGs check that the lines they need are included, and report back if anything is missing.

Run line independence tests

Revision 492017-04-27 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 68 to 68
 

Run bandwidth tests

Changed:
<
<
The total bandwidth for the stripping should be agreed with the bookkeeping manager and the PPG. Currently the limit is 60 Mb/s for the full stripping. The maximum bandwidth of the incremental restripping should be "small" compared to it (not more than 20% of the full stripping, e.g. 12 MB/s).
>
>
The total bandwidth for the stripping should be agreed with the bookkeeping manager and the PPG. For S28, the full Stripping had around 150 Mb/s. The maximum bandwidth of the incremental restripping should be "small" compared to it (not more than 20% of the full stripping).
  Bandwidth and timing tests are run on the test sample. Scripts are provided (files ganga.py, submit_jobs.sh). Since large number of events is required for reliable calculation, the test jobs are submitted to the GRID (use submit_jobs.py after setting up Ganga environment).
Line: 77 to 77
 
ls -ltr path_to_output_dsts
Changed:
<
<
and add these numbers to the correct streams of the corresponding WG.
>
>
and add these numbers to the correct streams of the corresponding WG. FTAG is called MDST in retentions_new.py.
  After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists. It is very important that all WGs check that the lines they need are included, and report back if anything is missing.

Revision 482017-04-27 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 50 to 50
 
gaudirun.py -T TestFromSettings_all.py path_to_input_data 
Changed:
<
<
to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script.
>
>
to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script. You need to change the WG and Stripping, and possibly year and tags. If you want to run a smaller test of e.g. 100k, set EvtMax accordingly. It is advisable to create separate folders for each WG to avoid that the output DSTs get overwritten (you will need them later)
 

Run memory use tests

Line: 74 to 72
  Bandwidth and timing tests are run on the test sample. Scripts are provided (files ganga.py, submit_jobs.sh). Since large number of events is required for reliable calculation, the test jobs are submitted to the GRID (use submit_jobs.py after setting up Ganga environment).
Changed:
<
<
Script retentions.py can be used after the test jobs finish to calculate the retentions and timing of individual lines, streams and WGs and generate tables in the twiki format. One needs to edit its header to change the job IDs of the test jobs, DaVinci version and, possibly, the trigger rate.
>
>
Script retentions.py or retentions_new.py can be used after the test jobs finish to calculate the retentions and timing of individual lines, streams and WGs and generate tables in the twiki format. One needs to edit its header to change the job IDs of the test jobs (or local folder where the logfiles are stored), DaVinci version and, possibly, the trigger rate. The streamEventSize has to be changed each time you made a change to the Stripping. To get the numbers for the streamEventSize, do

ls -ltr path_to_output_dsts

and add these numbers to the correct streams of the corresponding WG.

  After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists. It is very important that all WGs check that the lines they need are included, and report back if anything is missing.
Line: 310 to 313
 
        > ./make_html.sh
Changed:
<
<
  • Copy html/strippingXX to /afs/cern.ch/lhcb/software/releases/DOC/stripping/config/ and edit /afs/cern.ch/lhcb/software/releases/DOC/stripping/maindesc.html.
>
>
  • Copy html/strippingXX to /afs/cern.ch/lhcb/software/releases/DOC/stripping/config/ and edit /afs/cern.ch/lhcb/software/releases/DOC/stripping/maindesc.html.</verbatim>
 
META FILEATTACHMENT attachment="check_tuples.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="check_tuples.py.txt" path="check_tuples.py.txt" size="2137" user="lcarson" version="1"
META FILEATTACHMENT attachment="DV-Stripping20r0p3-Indep.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="DV-Stripping20r0p3-Indep.py.txt" path="DV-Stripping20r0p3-Indep.py.txt" size="5494" user="lcarson" version="1"

Revision 472017-04-24 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 43 to 44
  After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings.
Changed:
<
<
--+++ Run retention tests (locally)
>
>

Run retention tests (locally)

  Run
Deleted:
<
<
 
gaudirun.py -T TestFromSettings_all.py path_to_input_data 
Line: 54 to 54
 

Run memory use tests

Changed:
<
<
To test the memory consumption, a single, reasonably long (100k-200k events) stripping job is needed. Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory use as a function of event number:
>
>
To test the global memory consumption, a single, reasonably long (100k-200k events) stripping job is needed (see step above). Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory use as a function of event number.
 
Changed:
<
<
./analyseDaVinciLog.py logfile.txt display plot.png
>
>
python(2) analyseDaVinciLog.py logfile.txt -p name_of_your_plot.png display name_of_your_plot.png

The options -c and -l are currently not working.

  To run safely in the production, jobs don't have to take more than 4 Gb of memory. Make sure there are no big jumps in memory use after ~50-100k events. They can indicate memory leaks. In case of doubt, use valgrind to find memory leaks. Memory auditor can also help to find an algorithm with memory leak. To enable it, add the following to the DaVinci options:
Added:
>
>
 
  ApplicationMgr().ExtSvc += [ 'AuditorSvc' ]
  AuditorSvc().Auditors += [ 'MemoryAuditor' ]

Revision 462017-04-20 - StefanieReichert

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 36 to 35
  The deadline for submitting new or modifying the existing stripping selections is decided by the PPG, taking into account the time of conferences where the new results have to be presented, processing time and the time to prepare the stripping and DaVinci release after the deadline. The latter can take up to one month and depends on the number of iterations required to reach the limits on bandwidth and/or timing.
Changed:
<
<

Prepare test samples

>
>

Prepare test samples

  Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed. It can make sense to take (parts of) several runs as the test sample if the running conditions or trigger configuration was changing significantly. Once the run for the test sample has been identified, inform bookkeeping manager and ask to copy it to "freezer". The python options have to be prepared and put to Phys/StrippingSelections/tests/data/.
Line: 44 to 43
  After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings.
Added:
>
>
--+++ Run retention tests (locally)

Run

gaudirun.py -T TestFromSettings_all.py path_to_input_data 

to create the dst/mdst and log files needed for the next steps. Change the WG and Stripping in the script.

 

Run memory use tests

To test the memory consumption, a single, reasonably long (100k-200k events) stripping job is needed. Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory use as a function of event number:

Line: 94 to 101
 
lb-run -c x86_64-slc6-gcc49-dbg DaVinci/<version> sh -c 'gdb --args python $(which gaudirun.py) <options>'
Changed:
<
<
See also here. You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with
>
>
See also here. You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with
 
kill -2 <pid>

from another terminal it immediately continued.

Revision 452017-04-19 - MichaelAlexander

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 94 to 94
 
lb-run -c x86_64-slc6-gcc49-dbg DaVinci/<version> sh -c 'gdb --args python $(which gaudirun.py) <options>'
Changed:
<
<
You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with
>
>
See also here. You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with
 
kill -2 <pid>

from another terminal it immediately continued.

Revision 442017-04-03 - MichaelAlexander

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 113 to 113
 

Check nightlies and correct if needed

Changed:
<
<
Nightlies page is here. Nightlies have to be checked not only before the release of Stripping, but also from time to time during the development of the stripping selections by the analysts to spot problems early. In case of problems, inform the responsible person. In some cases, the nightly tests failures are expected because of changes introduced, and updating the test reference is needed to fix them.
>
>
Nightlies page is here. Nightlies have to be checked not only before the release of Stripping, but also from time to time during the development of the stripping selections by the analysts to spot problems early. In case of problems, inform the responsible person. In some cases, the nightly tests failures are expected because of changes introduced, and updating the test reference is needed to fix them.
 
Changed:
<
<
To run the test of a package:
>
>
Tests run by the nightlies are defined by the files under
 
Changed:
<
<
cd cmt cmt TestPackage
>
>
$PACKAGEROOT/tests/qmtest/package.qms/*.qmt
 
Changed:
<
<
or to run a single test, assuming the tests are in a directory davinci.qms and the test is called mytest.qmt:
>
>
Scripts to be run live under $PACKAGEROOT/tests/ and references, if needed, are under $PACKAGEROOT/tests/refs.

To run the test of a package, after building do:

   make test

Under cmt, you could run a single test, assuming the tests are in a directory davinci.qms and the test is called mytest.qmt, with:

 
   cmt TestPackage davinci.mytest
Changed:
<
<
To update test reference (after runnung the test):
>
>
but I'm not sure how to do this under cmake.

To update the test reference (after running the test):

 
   cd tests/refs

Revision 432017-03-30 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 88 to 91
 

Debugging with gdb

If there's an issue with the stripping that's not easy to work out, eg, if a line is hanging for some reason, gdb can be useful. To run the stripping with gdb use, eg

Deleted:
<
<
 
Changed:
<
<
lb-run -c x86_64-slc6-gcc49-dbg DaVinci/ sh -c 'gdb --args python $(which gaudirun.py) '
>
>
lb-run -c x86_64-slc6-gcc49-dbg DaVinci/ sh -c 'gdb --args python $(which gaudirun.py) '
 
Changed:
<
<
You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with
kill -2 <pid>
from another terminal it immediately continued.
>
>
You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with
kill -2 <pid>

from another terminal it immediately continued.

  Once gdb has started, start the job with run. You can then interrupt whenever you like with ctrl-c, or if you're investigating a crash you can just wait for it, then you can use backtrace to get the method call backtrace. Use continue to continue the job after an interrupt.
Line: 98 to 102
 Once gdb has started, start the job with run. You can then interrupt whenever you like with ctrl-c, or if you're investigating a crash you can just wait for it, then you can use backtrace to get the method call backtrace. Use continue to continue the job after an interrupt.

Once you have the backtrace you can print info on an object given its address with, eg

Deleted:
<
<
 
Changed:
<
<
print (class CombineParticles)0x614b8940
>
>
print (class CombineParticles)0x614b8940
  then you can search the output, of which there will likely be a lot. If you're trying to work out which line a problematic object originates from, search the output for anything containing 'Line' or 'Stripping'. Then you can have a look at the module that the line belongs to.
Line: 141 to 147
 git clone ssh://git@gitlab.cern.ch:7999/lhcb/Stripping.git cd Stripping/Phys/StrippingSettings/dbase/ cp from/the/path/where/it/is/strippingXX .
Added:
>
>
git add strippingXX
 git commit strippingXX -m "update dbase for sXX"
Changed:
<
<
git push
>
>
git push
 

Prepare StrippingArchive

Line: 255 to 266
 
  • Prepare build environment for DaVinci version used in stripping production:
Changed:
<
<
> lb-dev DaVinci vVVrRpP
>
>
> lb-dev DaVinci vVVrRpP
 
  • For DaVinci versions prior to v36r0, it was necessary to getpack HltDAQ package, copy HltGenConfig algorithms from Hlt/HltCommon to HltDAQ and build it:
            > getpack Hlt/HltDAQ

Revision 422017-03-30 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 107 to 107
 

Release Stripping project

Changed:
<
<
Help on release procedure is at PrepareProjectRelease page.
>
>
Help on release procedure is at PrepareProjectReleaseGit page.
 

Check nightlies and correct if needed

Revision 412017-03-06 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 137 to 137
  mv stripping.tmp ../../dbase/strippingXX

This file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory:

Added:
>
>
 git clone ssh://git@gitlab.cern.ch:7999/lhcb/Stripping.git cd Stripping/Phys/StrippingSettings/dbase/ cp from/the/path/where/it/is/strippingXX . git commit strippingXX -m "update dbase for sXX" git push
Added:
>
>
 

Prepare StrippingArchive

Revision 402017-02-25 - MichaelAlexander

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 85 to 85
  If one (or more) of these warnings is found at an event number matching that given in the output of check_tuples.py, then this feature is to blame and (as this is a known issue) no action needs to be taken. If no such warning is found, then there is some other, new issue with this line that requires further investigation.
Added:
>
>

Debugging with gdb

If there's an issue with the stripping that's not easy to work out, eg, if a line is hanging for some reason, gdb can be useful. To run the stripping with gdb use, eg

lb-run -c x86_64-slc6-gcc49-dbg DaVinci/<version> sh -c 'gdb --args python $(which gaudirun.py) <options>'

You can use gaudirun.py --gdb, but I found this wouldn't let me interrupt, and when I interrupted with

kill -2 <pid>
from another terminal it immediately continued.

Once gdb has started, start the job with run. You can then interrupt whenever you like with ctrl-c, or if you're investigating a crash you can just wait for it, then you can use backtrace to get the method call backtrace. Use continue to continue the job after an interrupt.

Once you have the backtrace you can print info on an object given its address with, eg

print *(class CombineParticles*)0x614b8940

then you can search the output, of which there will likely be a lot. If you're trying to work out which line a problematic object originates from, search the output for anything containing 'Line' or 'Stripping'. Then you can have a look at the module that the line belongs to.

 

Release Stripping project

Help on release procedure is at PrepareProjectRelease page.

Revision 392017-02-02 - MichaelAlexander

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 217 to 217
 

Prepare production steps

Changed:
<
<
Go to https://lhcbweb.pic.es/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert.
>
>
Go to https://lhcbweb.pic.es/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert (currently Marco Cattaneo and Liang Sun).
 

Hide project and declare it released

Revision 382017-01-30 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 116 to 116
  python makeDB.py StrippingXX mv stripping.tmp ../../dbase/strippingXX
Added:
>
>
This file cannot be pushed using the lhcb git commands, as they don't support the inclusion of binary files. To include it you should do in a fresh environment and directory: git clone ssh://git@gitlab.cern.ch:7999/lhcb/Stripping.git cd Stripping/Phys/StrippingSettings/dbase/ cp from/the/path/where/it/is/strippingXX . git commit strippingXX -m "update dbase for sXX" git push
 

Prepare StrippingArchive

StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections which are used only for development. Once the stripping selections are fixed, copy the contents of the StrippingSelections into StrippingArchive in the appropriate subdirectory (StrippingXX). It's better to do it using svn functionality to keep track of where the selections came from:

Revision 372016-12-14 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 183 to 183
 

Tag stripping project using gitlab

Added:
>
>
Modify the CMakeLists.txt file under the project. The file project.cmt under cmt and the CMakeLists.txt file under StrippingSys. release.notes under doc and requirements file under StrippingSys/cmt
 Go to https://gitlab.cern.ch/lhcb/Stripping and sign in. Under Repository/Tags click on "New Tag" Add the Tag Name, and Release Notes that you want to appear (usually a summary of the changes in the package)

Revision 362016-10-07 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 225 to 225
 
  • Prepare build environment for DaVinci version used in stripping production:
Changed:
<
<
> SetupProject DaVinci vVVrRpP --build-env
>
>
> lb-dev DaVinci vVVrRpP
 
  • For DaVinci versions prior to v36r0, it was necessary to getpack HltDAQ package, copy HltGenConfig algorithms from Hlt/HltCommon to HltDAQ and build it:
            > getpack Hlt/HltDAQ

Revision 352016-09-30 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet


Changed:
<
<

SVN trickery

>
>

SVN trickery (obsolete)

  This might be useful as mistakes with svn can happen.In this link there are the guidelines of svn.
Line: 9 to 9
 This might be useful as mistakes with svn can happen.In this link there are the guidelines of svn.

To tag a version i.e: StrippingSelections to a particular tag execute the following command:

Deleted:
<
<
 
svn cp -r 198298 https://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/ https://svn.cern.ch/reps/lhcb/Stripping/tags/Phys/StrippingSelections/v4r9/ -m "Tagging v4r9"
Added:
>
>

GIT basics

All the basic information and links can be found in https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb

 

How to get the xml file from RDSTs

SetupLHCbDirac

Line: 188 to 181
 
svnCheckTags StrippingSys vXrY --diffs
Added:
>
>

Tag stripping project using gitlab

Go to https://gitlab.cern.ch/lhcb/Stripping and sign in. Under Repository/Tags click on "New Tag" Add the Tag Name, and Release Notes that you want to appear (usually a summary of the changes in the package)

 

Add Stripping project to lhcb-prerelease slot in the nightlies

Go to nightlies page, open "Configuration Editor", choose the slot needed (most likely lhcb-prerelease), double-click on Stripping tag and change the STRIPPING version there. Make sure "disabled" flag is "false".

Revision 342016-09-28 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 242 to 242
  > cd strippingXX > cp $STRIPPINGSELECTIONSROOT/tests/coordinators/documentation/* .
Changed:
<
<
  • Edit run.sh and make_html.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation).
>
>
  • Edit run.sh, make_html.sh and tck.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation).
 
  • Run the ./run.sh script which will run 10000 events (to make sure most on-demand services are initialised) and produce the stripping config in the file config.tar
        > ./run.sh
Changed:
<
<
  • Run ./make_html.py script which will create web documantation in the "html" folder using config.tar (note that utils.py module is required by this script)
>
>
  • Run ./make_html.sh script which will create web documantation in the "html" folder using config.tar (note that utils.py module is required by this script)
 
        > ./make_html.sh

Revision 332016-07-01 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 240 to 240
 
        > mkdir strippingXX
        > cd strippingXX
Changed:
<
<
> cp $STRIPPINGSELECTIONSROOT/test/coordinators/documentation/* .
>
>
> cp $STRIPPINGSELECTIONSROOT/tests/coordinators/documentation/* .
 
  • Edit run.sh and make_html.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation).

Revision 322016-06-17 - MichaelAlexander

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 204 to 204
 

Ask for release in Jira

Changed:
<
<
Ask Stripping, DaVinci and AppConfig to be released using Savannah. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).
>
>
Ask Stripping, DaVinci and AppConfig to be released using JIRA. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).

In the case that only a patch of AppConfig needs to be deployed, prepare the tag as as usual: set the version number in cmt/requirements, and add in the release header in doc/release.notes, and commit. Then tag the package as usual with tag_package AppConfig vXrY, at which point it should appear here. No build is required as AppConfig only contains options files, so you can request deployment in JIRA, an example request is here. That's all!

 

Prepare production steps

Revision 312016-06-06 - MichaelAlexander

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 26 to 26
 
  • Have lhcb_tech role in DIRAC to be able to modify production steps
  • Be members of lhcb-release-managers e-group to be able to tag projects in tag collector
  • Be members of z5:doclibrarians AFS group to be able to publish stripping lines documentation on the web pages
Added:
>
>
  • Be subscribed to lhcb-stripping-managers for obvious reasons.
 

Actions before stripping selections deadline

Revision 302016-03-08 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 189 to 201
 
  • Make sure DaVinci ().DataType is correct ('2011', '2012', etc.).
If the stripping configuration is expected to run in MC, the corresponding MC stripping options (DV-StrippingXX-Stripping-MC*.py) have to be updated.
Changed:
<
<

Ask for release in savannah

>
>

Ask for release in Jira

  Ask Stripping, DaVinci and AppConfig to be released using Savannah. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).
Line: 197 to 209
  Go to https://lhcbweb.pic.es/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert.
Added:
>
>

Hide project and declare it released

Go to the LHCb tag collector. Check again for possible lost tags. Once they are solved, and in the home of the tag collector look for the Stripping project. If needed, click the Edit button and changes the specifications. Click the Hide button and say yes to release the project.

 

Run validation

Run production

Revision 292016-03-04 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 154 to 154
  The easiest way to tag packages is to use tag collector. Go to "Tag package" and tag each package of the Stripping project which was modified since the previous release (no need to tag packages without changes).
Added:
>
>

Tag packages in DaVinci project

The only package that needs to be tagged by the Stripping coordinators is the StrippingCache which lives in DaVinci. Follow the same steps as for the previous step.

 

Tag StrippingSys

Use tag collector, modify cmt/requirements to use the correct versions of the Stripping packages tagged at the previous step, add sensible release notes (summary of changes from all packages).

Revision 282016-03-03 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Changed:
<
<
>
>

 

SVN trickery

Changed:
<
<
This might be useful as mistakes with svn can happen. In this link there are the guidelines of svn.
>
>
This might be useful as mistakes with svn can happen.In this link there are the guidelines of svn.
  To tag a version i.e: StrippingSelections to a particular tag execute the following command:
Deleted:
<
<
 
svn cp -r 198298 https://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/ https://svn.cern.ch/reps/lhcb/Stripping/tags/Phys/StrippingSelections/v4r9/ -m "Tagging v4r9"
Added:
>
>

How to get the xml file from RDSTs

SetupLHCbDirac

lhcb-proxy-init

dirac-bookkeeping-genXMLCatalog --Options=YourData.py --Catalog=yourdatacatalog.xml --Depth=2

 

Prerequisites

Stripping coordinators need the following rights:

Line: 68 to 72
  If there is a mismatch between the stripping decision of a given line for a certain event, you will see a mesage like this in the stdout of check_tuples.py:
Changed:
<
<
Checking line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision (job 724) subjob 0 checked, 48501 events, 37 selected Stripping decision is different: line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision, event 29990 (job 724, subjob 1): all=0, single=1
>
>
Checking line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision (job 724)subjob 0 checked, 48501 events, 37 selected Stripping decision is different: line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision, event 29990 (job 724, subjob 1): all=0, single=1
 
Changed:
<
<
This indicates that this line gave a positive decision on this event when the lines was run on its own, but a negative decision when ran as part of the whole stripping. Such mismatches can be caused by a known feature of the way that the stripping deals with events where the maximum number of allowed combinations is reached in an algorithm such as CombineParticles or SubPIDMMFilter (details of this can be found in the Stripping Coordinator Report here: https://indico.cern.ch/event/201535/session/3/contribution/29/material/slides/1.pdf). This feature has a tendency to affect lines in the Beauty2Charm module, but occasionally affects other lines as well. To check whether this feature is the cause of a mismatch, grep the log files of the single-line stripping job for "Maximum number of allowed particles reached". You are looking for a line like:
>
>
This indicates that this line gave a positive decision on this event when the lines was run on its own, but a negative decision when ran as part of the whole stripping. Such mismatches can be caused by a known feature of the way that the stripping deals with events where the maximum number of allowed combinations is reached in an algorithm such as CombineParticles or SubPIDMMFilter (details of this can be found in the Stripping Coordinator Report here:https://indico.cern.ch/event/201535/session/3/contribution/29/material/slides/1.pdf). This feature has a tendency to affect lines in the Beauty2Charm module, but occasionally affects other lines as well. To check whether this feature is the cause of a mismatch, grep the log files of the single-line stripping job for "Maximum number of allowed particles reached". You are looking for a line like:
  D+2HHHSubPIDSelBeauty2Charm WARNING SubPIDMMFilter:: Maximum number of allowed particles reached
Line: 146 to 144
 

Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)

Changed:
<
<
Update stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options To check which functors has not been cached, do
>
>
Update stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options To check which functors has not been cached, do
 
export LOKI_DISABLE_PYTHON=1
Line: 209 to 205
 
        > SetupProject DaVinci vVVrRpP --build-env
Changed:
<
<

>
>
  > getpack Hlt/HltDAQ > SetupProject DaVinci vVVrRpP > cd Hlt/HltDAQ/cmt

Revision 272016-01-22 - AndreaContu

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 222 to 222
 
        > mkdir strippingXX
        > cd strippingXX
Changed:
<
<
> cp ~poluekt/public/stripping20r1/strip_tck/* .
>
>
> cp $STRIPPINGSELECTIONSROOT/test/coordinators/documentation/* .
 
  • Edit run.sh and make_html.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation).

Revision 262016-01-04 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 112 to 119
 
  svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/python/StrippingSelections/ svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingArchive/python/StrippingArchive/StrippingXX/ -m "Create StrippingXX"
Added:
>
>
This command will not show the currently added stripping folder. You need to update the local repository to see it.

svn update

 Then add StrippingXX to __init__.py under StrippingArchive /python/StrippingArchive, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.

IMPORTANT: do not forget to modify the __init__.py under StrippingArchive /python/StrippingArchive/StrippingXX as explained in readme.txt

Revision 252015-11-30 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Added:
>
>

SVN trickery

This might be useful as mistakes with svn can happen. In this link there are the guidelines of svn.

To tag a version i.e: StrippingSelections to a particular tag execute the following command:

svn cp -r 198298 https://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/ https://svn.cern.ch/reps/lhcb/Stripping/tags/Phys/StrippingSelections/v4r9/ -m "Tagging v4r9"
 

Prerequisites

Stripping coordinators need the following rights:

Line: 135 to 139
  Update stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options To check which functors has not been cached, do
Added:
>
>
 
Changed:
<
<
export LOKI_DISABLE_PYTHON=1
>
>
export LOKI_DISABLE_PYTHON=1
 before testing

Tag packages in Stripping project

Revision 242015-11-25 - AndreaContu

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 134 to 134
 

Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)

Update stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options

Added:
>
>
To check which functors has not been cached, do
export LOKI_DISABLE_PYTHON=1
before testing
 

Tag packages in Stripping project

Revision 232015-11-14 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 26 to 26
 

Run memory use tests

To test the memory consumption, a single, reasonably long (100k-200k events) stripping job is needed. Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory use as a function of event number:

Added:
>
>
 
  ./analyseDaVinciLog.py logfile.txt
Changed:
<
<
display plot.png
>
>
display plot.png
 To run safely in the production, jobs don't have to take more than 4 Gb of memory. Make sure there are no big jumps in memory use after ~50-100k events. They can indicate memory leaks. In case of doubt, use valgrind to find memory leaks. Memory auditor can also help to find an algorithm with memory leak. To enable it, add the following to the DaVinci options:
Added:
>
>
 
  ApplicationMgr().ExtSvc += [ 'AuditorSvc' ]
Changed:
<
<
AuditorSvc().Auditors += [ 'MemoryAuditor' ]
>
>
AuditorSvc().Auditors += [ 'MemoryAuditor' ]
 

Run bandwidth tests

Line: 77 to 78
 Nightlies page is here. Nightlies have to be checked not only before the release of Stripping, but also from time to time during the development of the stripping selections by the analysts to spot problems early. In case of problems, inform the responsible person. In some cases, the nightly tests failures are expected because of changes introduced, and updating the test reference is needed to fix them.

To run the test of a package:

Added:
>
>
 
   cd cmt
Changed:
<
<
cmt TestPackage
>
>
cmt TestPackage
 or to run a single test, assuming the tests are in a directory davinci.qms and the test is called mytest.qmt:
Added:
>
>
 
Changed:
<
<
cmt TestPackage davinci.mytest
>
>
cmt TestPackage davinci.mytest
  To update test reference (after runnung the test):
Added:
>
>
 
   cd tests/refs
Changed:
<
<
mv *.ref.new *.ref
>
>
mv *.ref.new *.ref
 

Prepare StrippingSettings database

Line: 99 to 101
  cmt make cd ../python/StrippingSettings python makeDB.py StrippingXX
Changed:
<
<
mv stripping.tmp ../../dbase/strippingXX
>
>
mv stripping.tmp ../../dbase/strippingXX
 

Prepare StrippingArchive

Line: 105 to 106
 

Prepare StrippingArchive

StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections which are used only for development. Once the stripping selections are fixed, copy the contents of the StrippingSelections into StrippingArchive in the appropriate subdirectory (StrippingXX). It's better to do it using svn functionality to keep track of where the selections came from:

Added:
>
>
 
Changed:
<
<
svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/python/StrippingSelections/ svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingArchive/python/StrippingArchive/StrippingXX/ -m "Create StrippingXX" Then add StrippingXX to __init__.py under StrippingArchive/python/StrippingArchive, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.
>
>
svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/python/StrippingSelections/ svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingArchive/python/StrippingArchive/StrippingXX/ -m "Create StrippingXX"

Then add StrippingXX to __init__.py under StrippingArchive /python/StrippingArchive, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.

  IMPORTANT: do not forget to modify the __init__.py under StrippingArchive/python/StrippingArchive/StrippingXX as explained in readme.txt

Prepare CommonParticlesArchive

CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified. To prepare the package a folder with the name StrippingXX needs to be created in CommonParticlesArchive and the contents of the *.py files in CommonParticles needs to be copied. if the full folder is copied, remember to cancel the hidden file .svn, otherwise will prevent the commit to happen.

Added:
>
>
To do it directly execute the command:
 
  svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/CommonParticles/python/CommonParticles/
Changed:
<
<
svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/CommonParticlesArchive/python/CommonParticlesArchive/StrippingXX/ -m "Added archive for StrippingXX"
>
>
svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/CommonParticlesArchive/python/CommonParticlesArchive/StrippingXX/ -m "Added archive for StrippingXX"

This will not update your local repository, so afterwards do

svn update
 
Added:
>
>
to see the changes.
 

Check lost tags in tag collector

Added:
>
>
 Tag collector page is here. Go to "Lost tags" and add them if needed.

Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)

Line: 140 to 151
  emacs cmt/project.cmt # Update dependencies here emacs CMakeLists.txt # Update dependencies svn commit -m vXrY
Changed:
<
<
tag_package -P Stripping vXrY
>
>
tag_package -P Stripping vXrY
 Check you picked up the correct packages by issuing
Added:
>
>
 
Changed:
<
<
svnCheckTags StrippingSys vXrY --diffs
>
>
svnCheckTags StrippingSys vXrY --diffs
 

Add Stripping project to lhcb-prerelease slot in the nightlies

Added:
>
>
 Go to nightlies page, open "Configuration Editor", choose the slot needed (most likely lhcb-prerelease), double-click on Stripping tag and change the STRIPPING version there. Make sure "disabled" flag is "false".
Changed:
<
<
If you do not want to wait for the nightlies, follow the instructions here ProjectRelease#For_Release_Managers
>
>
If you do not want to wait for the nightlies, follow the instructions here ProjectRelease#For_Release_Managers
 

Prepare AppConfig options

The stripping options are in AppConfig/options/DaVinci/DV-StrippingXX-Stripping.py. In most cases the DaVinci options for stripping can be copied from the previous round, with minor changes:

Line: 177 to 189
 
  • Prepare build environment for DaVinci version used in stripping production:
Changed:
<
<
> SetupProject DaVinci vVVrRpP --build-env
>
>
> SetupProject DaVinci vVVrRpP --build-env
 

Line: 193 to 204
 
        > mkdir strippingXX
        > cd strippingXX
Changed:
<
<
> cp ~poluekt/public/stripping20r1/strip_tck/* .
>
>
> cp ~poluekt/public/stripping20r1/strip_tck/* .
 
  • Edit run.sh and make_html.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation).

  • Run the ./run.sh script which will run 10000 events (to make sure most on-demand services are initialised) and produce the stripping config in the file config.tar
Changed:
<
<
> ./run.sh
>
>
> ./run.sh
 
  • Run ./make_html.py script which will create web documantation in the "html" folder using config.tar (note that utils.py module is required by this script)
Changed:
<
<
> ./make_html.sh
>
>
> ./make_html.sh
 
  • Copy html/strippingXX to /afs/cern.ch/lhcb/software/releases/DOC/stripping/config/ and edit /afs/cern.ch/lhcb/software/releases/DOC/stripping/maindesc.html.

Revision 222015-09-22 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 108 to 108
 
  svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/python/StrippingSelections/ svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingArchive/python/StrippingArchive/StrippingXX/ -m "Create StrippingXX"
Changed:
<
<
Then add StrippingXX to __init__.py, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.
>
>
Then add StrippingXX to __init__.py under StrippingArchive/python/StrippingArchive, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.
 
Changed:
<
<
IMPORTANT: do not forget to modify the init.py as explained in readme.txt
>
>
IMPORTANT: do not forget to modify the __init__.py under StrippingArchive/python/StrippingArchive/StrippingXX as explained in readme.txt
 

Prepare CommonParticlesArchive

Revision 212015-09-22 - AndreaContu

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 110 to 110
  Then add StrippingXX to __init__.py, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.
Added:
>
>
IMPORTANT: do not forget to modify the init.py as explained in readme.txt
 

Prepare CommonParticlesArchive

CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified. To prepare the package a folder with the name StrippingXX needs to be created in CommonParticlesArchive and the contents of the *.py files in CommonParticles needs to be copied. if the full folder is copied, remember to cancel the hidden file .svn, otherwise will prevent the commit to happen.

Revision 202015-09-07 - AndreaContu

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 121 to 121
 

Check lost tags in tag collector

Tag collector page is here. Go to "Lost tags" and add them if needed.
Added:
>
>

Prepare stripping cache (Phys/StrippingCache is a DaVinci Package!)

Update stripping list in Phys/StrippingCache/CMakeLists.txt and add options under Phys/StrippingCache/options
 

Tag packages in Stripping project

The easiest way to tag packages is to use tag collector. Go to "Tag package" and tag each package of the Stripping project which was modified since the previous release (no need to tag packages without changes).

Revision 192015-06-29 - AndreaContu

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 145 to 145
 

Add Stripping project to lhcb-prerelease slot in the nightlies

Go to nightlies page, open "Configuration Editor", choose the slot needed (most likely lhcb-prerelease), double-click on Stripping tag and change the STRIPPING version there. Make sure "disabled" flag is "false".
Added:
>
>
If you do not want to wait for the nightlies, follow the instructions here ProjectRelease#For_Release_Managers
 

Prepare AppConfig options

The stripping options are in AppConfig/options/DaVinci/DV-StrippingXX-Stripping.py. In most cases the DaVinci options for stripping can be copied from the previous round, with minor changes:

Revision 182015-06-09 - RicardoVazquezGomez

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 102 to 102
  mv stripping.tmp ../../dbase/strippingXX
Changed:
<
<

Prepare stripping archive

>
>

Prepare StrippingArchive

  StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections which are used only for development. Once the stripping selections are fixed, copy the contents of the StrippingSelections into StrippingArchive in the appropriate subdirectory (StrippingXX). It's better to do it using svn functionality to keep track of where the selections came from:
Line: 110 to 110
  Then add StrippingXX to __init__.py, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.
Added:
>
>

Prepare CommonParticlesArchive

CommonParticlesArchive is the package all versions of released common particles are stored. The CommonParticles package is rarely modified. To prepare the package a folder with the name StrippingXX needs to be created in CommonParticlesArchive and the contents of the *.py files in CommonParticles needs to be copied. if the full folder is copied, remember to cancel the hidden file .svn, otherwise will prevent the commit to happen.

  svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/CommonParticles/python/CommonParticles/
svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/CommonParticlesArchive/python/CommonParticlesArchive/StrippingXX/ -m "Added archive for StrippingXX"
 

Check lost tags in tag collector

Tag collector page is here. Go to "Lost tags" and add them if needed.

Revision 172015-05-29 - AndreaContu

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 113 to 113
 

Check lost tags in tag collector

Tag collector page is here. Go to "Lost tags" and add them if needed.
Added:
>
>

Tag packages in Stripping project

The easiest way to tag packages is to use tag collector. Go to "Tag package" and tag each package of the Stripping project which was modified since the previous release (no need to tag packages without changes).

Tag StrippingSys

Use tag collector, modify cmt/requirements to use the correct versions of the Stripping packages tagged at the previous step, add sensible release notes (summary of changes from all packages).
 

Update dependencies in Stripping project

  cd <where you have plenty of space>
Line: 128 to 134
 svnCheckTags StrippingSys vXrY --diffs
Deleted:
<
<

Tag packages in Stripping project

The easiest way to tag packages is to use tag collector. Go to "Tag package" and tag each package of the Stripping project which was modified since the previous release (no need to tag packages without changes).

Tag StrippingSys

Use tag collector, modify cmt/requirements to use the correct versions of the Stripping packages tagged at the previous step, add sensible release notes (summary of changes from all packages).
 

Add Stripping project to lhcb-prerelease slot in the nightlies

Go to nightlies page, open "Configuration Editor", choose the slot needed (most likely lhcb-prerelease), double-click on Stripping tag and change the STRIPPING version there. Make sure "disabled" flag is "false".

Revision 162015-03-13 - AndreaContu

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 119 to 119
  getpack -P -H STRIPPING HEAD cd STRIPPING/STRIPPING_HEAD/ emacs cmt/project.cmt # Update dependencies here
Changed:
<
<
cd cmt/
>
>
emacs CMakeLists.txt # Update dependencies
  svn commit -m vXrY
Deleted:
<
<
cd ..
  tag_package -P Stripping vXrY
Added:
>
>
Check you picked up the correct packages by issuing
svnCheckTags StrippingSys vXrY --diffs
 

Tag packages in Stripping project

The easiest way to tag packages is to use tag collector. Go to "Tag package" and tag each package of the Stripping project which was modified since the previous release (no need to tag packages without changes).

Revision 152014-09-11 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 163 to 163
  > SetupProject DaVinci vVVrRpP --build-env
Changed:
<
<
>
>
 
        > getpack Hlt/HltDAQ
Deleted:
<
<

  > SetupProject DaVinci vVVrRpP > cd Hlt/HltDAQ/cmt > cp $LHCBRELEASES/HLT/HLT_v20r1p1/Hlt/HltCommon/src/HltGenConfig.* ../src/ > cmt make
Changed:
<
<
>
>
These steps are NOT needed for new DaVinci (v36r0 and more recent) because HltGenConfig became available in DaVinci.
 
  • Create an empty directory in a place with plenty of space, copy some scripts used for stripping configuration dump:

Revision 142014-08-21 - LaurenceCarson

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 48 to 48
 

Run line independence tests

Added:
>
>
These should be run once the stripping selections are frozen. Assuming that the lines for 2011 and 2012 data are the same as each other, only one year needs to be checked. The scripts to run these tests are attached to this TWiki page. (The TWiki software has appended ".txt" to all of these filenames for some reason, this should be removed obviously)

The DaVinci scripts DV-Stripping20r0p3-Indep.py and DV-Stripping20r0p3-Indep-all.py run the stripping and save the stripping decisions in a ROOT NTuple (these scripts are set up to run S20r0p3, but changing to other strippings should be straightforward). The Indep-all.py runs all of the lines in a particular stripping, while Indep.py runs only a single line. Of course each script must be run over the same input events.

The ganga script ganga_all.py runs DV-Stripping20r0p3-Indep-all.py in ganga. This only needs to be run once, producing a single ganga job. The ganga script ganga_line.py runs DV-Stripping20r0p3-Indep.py in ganga, for a certain stripping line. This needs to be run as many times as there are lines in the given stripping. This produces many ganga jobs. To run these many jobs, there is a small script called submit_jobs.py. This reads in the various line names from a plain text file, which can be produced by running a short test of DV-Stripping20r0p3-Indep-all.py locally. Running submit_jobs.py to submit a lot of ganga jobs can take many hours.

Once the NTuples have been created, a script called check_tuples.py is used to run over the various single-line NTuples, and compare the stripping decisions with the NTuple from the whole-stripping job. The script can deal with incomplete or failed subjobs by skipping over the missing data. The only part of check_tuples.py that should need changing is the job numbers for the ganga jobs, and the location of the gangadir directory holding the output of the jobs.

If there is a mismatch between the stripping decision of a given line for a certain event, you will see a mesage like this in the stdout of check_tuples.py:

Checking line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision (job 724) subjob 0 checked, 48501 events, 37 selected Stripping decision is different: line StrippingLb2DpKPiD2HHHBeauty2CharmLineDecision, event 29990 (job 724, subjob 1): all=0, single=1

This indicates that this line gave a positive decision on this event when the lines was run on its own, but a negative decision when ran as part of the whole stripping. Such mismatches can be caused by a known feature of the way that the stripping deals with events where the maximum number of allowed combinations is reached in an algorithm such as CombineParticles or SubPIDMMFilter (details of this can be found in the Stripping Coordinator Report here: https://indico.cern.ch/event/201535/session/3/contribution/29/material/slides/1.pdf). This feature has a tendency to affect lines in the Beauty2Charm module, but occasionally affects other lines as well. To check whether this feature is the cause of a mismatch, grep the log files of the single-line stripping job for "Maximum number of allowed particles reached". You are looking for a line like:

D+2HHHSubPIDSelBeauty2Charm WARNING SubPIDMMFilter:: Maximum number of allowed particles reached

If one (or more) of these warnings is found at an event number matching that given in the output of check_tuples.py, then this feature is to blame and (as this is a known issue) no action needs to be taken. If no such warning is found, then there is some other, new issue with this line that requires further investigation.

 

Release Stripping project

Help on release procedure is at PrepareProjectRelease page.

Line: 176 to 196
 

  • Copy html/strippingXX to /afs/cern.ch/lhcb/software/releases/DOC/stripping/config/ and edit /afs/cern.ch/lhcb/software/releases/DOC/stripping/maindesc.html.
Added:
>
>
META FILEATTACHMENT attachment="check_tuples.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="check_tuples.py.txt" path="check_tuples.py.txt" size="2137" user="lcarson" version="1"
META FILEATTACHMENT attachment="DV-Stripping20r0p3-Indep.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="DV-Stripping20r0p3-Indep.py.txt" path="DV-Stripping20r0p3-Indep.py.txt" size="5494" user="lcarson" version="1"
META FILEATTACHMENT attachment="DV-Stripping20r0p3-Indep-all.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="DV-Stripping20r0p3-Indep-all.py.txt" path="DV-Stripping20r0p3-Indep-all.py.txt" size="5324" user="lcarson" version="1"
META FILEATTACHMENT attachment="ganga_all.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="ganga_all.py.txt" path="ganga_all.py.txt" size="799" user="lcarson" version="1"
META FILEATTACHMENT attachment="ganga_line.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="ganga_line.py.txt" path="ganga_line.py.txt" size="962" user="lcarson" version="1"
META FILEATTACHMENT attachment="submit_jobs.py.txt" attr="" comment="Scripts to run Independence Tests" date="1408639878" name="submit_jobs.py.txt" path="submit_jobs.py.txt" size="239" user="lcarson" version="1"

Revision 132013-10-08 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 8 to 8
  Stripping coordinators need the following rights:
  • Have lhcb_tech role in DIRAC to be able to modify production steps
Changed:
<
<
  • Have to be members of lhcb-release-managers e-group to be able to tag projects in tag collector
>
>
  • Be members of lhcb-release-managers e-group to be able to tag projects in tag collector
  • Be members of z5:doclibrarians AFS group to be able to publish stripping lines documentation on the web pages
 

Actions before stripping selections deadline

Line: 162 to 163
  > cp ~poluekt/public/stripping20r1/strip_tck/* .
Changed:
<
<
  • Edit run.sh and tck.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation).
>
>
  • Edit run.sh and make_html.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation).
 
  • Run the ./run.sh script which will run 10000 events (to make sure most on-demand services are initialised) and produce the stripping config in the file config.tar
        > ./run.sh
Changed:
<
<
  • Run ./utils.py script which will create web documantation in the "html" folder using config.tar
>
>
  • Run ./make_html.py script which will create web documantation in the "html" folder using config.tar (note that utils.py module is required by this script)
 
Changed:
<
<
> python utils.py
>
>
> ./make_html.sh
 

  • Copy html/strippingXX to /afs/cern.ch/lhcb/software/releases/DOC/stripping/config/ and edit /afs/cern.ch/lhcb/software/releases/DOC/stripping/maindesc.html.
\ No newline at end of file

Revision 122013-06-26 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 165 to 162
  > cp ~poluekt/public/stripping20r1/strip_tck/* .
Changed:
<
<
  • Edit run.sh and tck.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc.
>
>
  • Edit run.sh and tck.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc. (including the text strings in HltGenConfig initialisation).

  • Run the ./run.sh script which will run 10000 events (to make sure most on-demand services are initialised) and produce the stripping config in the file config.tar
        > ./run.sh

  • Run ./utils.py script which will create web documantation in the "html" folder using config.tar
        > python utils.py
 
Changed:
<
<
>
>
  • Copy html/strippingXX to /afs/cern.ch/lhcb/software/releases/DOC/stripping/config/ and edit /afs/cern.ch/lhcb/software/releases/DOC/stripping/maindesc.html.
 \ No newline at end of file

Revision 112013-06-26 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 134 to 134
 

Run production

Prepare web documentation

Added:
>
>
Reference documentation about all lines (their cuts, sequence of algorithms, streams, output locations) and standard particles is available at the Stripping project webpages. These pages are generated automatically using the HltGenConfig algorithm (the same one that is used to generate HltTCK). Here are the instructions on generating these pages:

  • Prepare build environment for DaVinci version used in stripping production:

        > SetupProject DaVinci vVVrRpP --build-env

        > getpack Hlt/HltDAQ

        > SetupProject DaVinci vVVrRpP
        > cd Hlt/HltDAQ/cmt
        > cp $LHCBRELEASES/HLT/HLT_v20r1p1/Hlt/HltCommon/src/HltGenConfig.* ../src/
        > cmt make

  • Create an empty directory in a place with plenty of space, copy some scripts used for stripping configuration dump:
        > mkdir strippingXX
        > cd strippingXX
        > cp ~poluekt/public/stripping20r1/strip_tck/* .

  • Edit run.sh and tck.py to use the correct AppConfig options, DaVinci version, Stripping version, data type, test run etc.

Revision 102013-06-14 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Added:
>
>

Prerequisites

Stripping coordinators need the following rights:

  • Have lhcb_tech role in DIRAC to be able to modify production steps
  • Have to be members of lhcb-release-managers e-group to be able to tag projects in tag collector
 

Actions before stripping selections deadline

The deadline for submitting new or modifying the existing stripping selections is decided by the PPG, taking into account the time of conferences where the new results have to be presented, processing time and the time to prepare the stripping and DaVinci release after the deadline. The latter can take up to one month and depends on the number of iterations required to reach the limits on bandwidth and/or timing.

Line: 47 to 53
 

Check nightlies and correct if needed

Changed:
<
<
Nightlies page is here.
>
>
Nightlies page is here. Nightlies have to be checked not only before the release of Stripping, but also from time to time during the development of the stripping selections by the analysts to spot problems early. In case of problems, inform the responsible person. In some cases, the nightly tests failures are expected because of changes introduced, and updating the test reference is needed to fix them.
  To run the test of a package:
Line: 59 to 65
  cmt TestPackage davinci.mytest
Changed:
<
<
To update test reference:
>
>
To update test reference (after runnung the test):
 
   cd tests/refs
   mv ***.ref.new ***.ref
Line: 99 to 105
 

Tag packages in Stripping project

Changed:
<
<
Use tag collector
>
>
The easiest way to tag packages is to use tag collector. Go to "Tag package" and tag each package of the Stripping project which was modified since the previous release (no need to tag packages without changes).
 

Tag StrippingSys

Changed:
<
<
Use tag collector, add sensible release notes
>
>
Use tag collector, modify cmt/requirements to use the correct versions of the Stripping packages tagged at the previous step, add sensible release notes (summary of changes from all packages).
 

Add Stripping project to lhcb-prerelease slot in the nightlies

Added:
>
>
Go to nightlies page, open "Configuration Editor", choose the slot needed (most likely lhcb-prerelease), double-click on Stripping tag and change the STRIPPING version there. Make sure "disabled" flag is "false".
 

Prepare AppConfig options

Revision 92013-06-04 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 108 to 108
 

Prepare AppConfig options

Added:
>
>
The stripping options are in AppConfig/options/DaVinci/DV-StrippingXX-Stripping.py. In most cases the DaVinci options for stripping can be copied from the previous round, with minor changes:
  • Change stripping version used to retrieve configuration from StrippingSettings and StrippingArchive.
  • Stripping TCK has to be changed. Current convention is the following: TCK = 0xVVVVSSSS, where VVVV is the DaVinci version, and SSSS is the Stripping number (e.g. for Stripping20r1p1 running on DaVinci v32r2p5 TCK = 0x32252011).
  • Make sure DaVinci().DataType is correct ('2011', '2012', etc.).
If the stripping configuration is expected to run in MC, the corresponding MC stripping options (DV-StrippingXX-Stripping-MC*.py) have to be updated.
 

Ask for release in savannah

Ask Stripping, DaVinci and AppConfig to be released using Savannah. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).

Revision 82013-05-23 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 106 to 106
 

Add Stripping project to lhcb-prerelease slot in the nightlies

Deleted:
<
<

Ask for release in savannah

 

Prepare AppConfig options

Added:
>
>

Ask for release in savannah

Ask Stripping, DaVinci and AppConfig to be released using Savannah. Stripping should be released "without online or cvmfs" since it will be installed as dependency, DaVinci and AppConfig have to be released "with online and cvmfs" (online farm is also used for stripping!).

 

Prepare production steps

Added:
>
>
Go to https://lhcbweb.pic.es/DIRAC/, login with your GRID certificate, choose lhcb_tech role at the bottom of the page, and go to Production/Step Manager in the menu. Choose Application=DaVinci in the panel on the right, duplicate one of the previous Stripping and Merging steps. Modify DaVinci and AppConfig versions, database tags (CondDB and DDDB). Correct CondDB and DDDB tags can be taken from the corresponding reconstruction (Brunel) steps, or check with the DB expert.
 

Run validation

Run production

Revision 72013-05-22 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 6 to 6
 

Actions before stripping selections deadline

Changed:
<
<
The deadline for submitting new or modifying the existing stripping selections is decided by the PPG, taking into account the time of conferences where the new results have to be presented, processing time and the time to prepare the stripping and DaVinci release after the deadline. The latter can take up to one month and depends on the number of iteration required to reach the limits on bandwidth and/or timing.
>
>
The deadline for submitting new or modifying the existing stripping selections is decided by the PPG, taking into account the time of conferences where the new results have to be presented, processing time and the time to prepare the stripping and DaVinci release after the deadline. The latter can take up to one month and depends on the number of iterations required to reach the limits on bandwidth and/or timing.
 

Prepare test samples

Changed:
<
<
Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed. It can make sense to take (parts of) several runs as the test sample if the running conditions or trigger configuration was changing significantly. One the run for the test sample has been identified, inform bookkeeping manager and ask to copy it to "freezer". The python options have to be prepared and put to Phys/StrippingSelections/tests/data/.
>
>
Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed. It can make sense to take (parts of) several runs as the test sample if the running conditions or trigger configuration was changing significantly. Once the run for the test sample has been identified, inform bookkeeping manager and ask to copy it to "freezer". The python options have to be prepared and put to Phys/StrippingSelections/tests/data/.
 

Actions after stripping selections deadline

Line: 18 to 18
 

Run memory use tests

Changed:
<
<
To test the memory consumption, a single, reasonably long (100k-200k events) stripping job is needed. Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory as a function of event number:
>
>
To test the memory consumption, a single, reasonably long (100k-200k events) stripping job is needed. Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory use as a function of event number:
 
  ./analyseDaVinciLog.py logfile.txt
  display plot.png
Line: 31 to 31
 

Run bandwidth tests

Added:
>
>
The total bandwidth for the stripping should be agreed with the bookkeeping manager and the PPG. Currently the limit is 60 Mb/s for the full stripping. The maximum bandwidth of the incremental restripping should be "small" compared to it (not more than 20% of the full stripping, e.g. 12 MB/s).

Bandwidth and timing tests are run on the test sample. Scripts are provided (files ganga.py, submit_jobs.sh). Since large number of events is required for reliable calculation, the test jobs are submitted to the GRID (use submit_jobs.py after setting up Ganga environment).

Script retentions.py can be used after the test jobs finish to calculate the retentions and timing of individual lines, streams and WGs and generate tables in the twiki format. One needs to edit its header to change the job IDs of the test jobs, DaVinci version and, possibly, the trigger rate.

After the tables with bandwidth and timing are ready, advertise them on DaVinci and Stripping mailing lists. It is very important that all WGs check that the lines they need are included, and report back if anything is missing.

 

Run line independence tests

Release Stripping project

Line: 76 to 84
 Then add StrippingXX to __init__.py, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.

Check lost tags in tag collector

Changed:
<
<
Tag collector page is here
>
>
Tag collector page is here. Go to "Lost tags" and add them if needed.
 

Update dependencies in Stripping project

Revision 62013-05-17 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 10 to 10
 

Prepare test samples

Changed:
<
<
Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed. It can make sense to take (parts of) several runs as the test sample if the running conditions or trigger configuration was changing significantly. One the run for the test sample has been identified, inform bookkeeping manager and ask to copy it to "freezer". The python options have to be prepared and put to Phys/StrippingSelections/tests/data/.
>
>
Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed. It can make sense to take (parts of) several runs as the test sample if the running conditions or trigger configuration was changing significantly. One the run for the test sample has been identified, inform bookkeeping manager and ask to copy it to "freezer". The python options have to be prepared and put to Phys/StrippingSelections/tests/data/.
 

Actions after stripping selections deadline

Changed:
<
<
After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings.
>
>
After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings.
 

Run memory use tests

Changed:
<
<
To test the memory consumption, a single, reasonably long (100k-200k events) stripping job is needed. Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory as a function of event number:
>
>
To test the memory consumption, a single, reasonably long (100k-200k events) stripping job is needed. Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory as a function of event number:
 
  ./analyseDaVinciLog.py logfile.txt
  display plot.png
Changed:
<
<
To run safely in the production, jobs don't have to take more than 4 Gb of memory. Make sure there are no big jumps in memory use after ~50-100k events. They can indicate memory leaks. In case of doubt, use valgrind to find memory leaks. Memory auditor can also help to find an algorithm with memory leak. To enable it, add the following to the DaVinci options:
>
>
To run safely in the production, jobs don't have to take more than 4 Gb of memory. Make sure there are no big jumps in memory use after ~50-100k events. They can indicate memory leaks. In case of doubt, use valgrind to find memory leaks. Memory auditor can also help to find an algorithm with memory leak. To enable it, add the following to the DaVinci options:
 
  ApplicationMgr().ExtSvc += [ 'AuditorSvc' ]
  AuditorSvc().Auditors += [ 'MemoryAuditor' ]
Line: 46 to 46
  cd cmt cmt TestPackage
Changed:
<
<
or to run a single test, assuming the tests are in a directory davinci.qms and the test is called mytest.qmt:
>
>
or to run a single test, assuming the tests are in a directory davinci.qms and the test is called mytest.qmt:
 
   cmt TestPackage davinci.mytest
Line: 69 to 69
 

Prepare stripping archive

Changed:
<
<
StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections which are used only for development. Once the stripping selections are fixed, copy the contents of the StrippingSelections into StrippingArchive in the appropriate subdirectory (StrippingXX). It's better to do it using svn functionality to keep track of where the selections came from:
>
>
StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections which are used only for development. Once the stripping selections are fixed, copy the contents of the StrippingSelections into StrippingArchive in the appropriate subdirectory (StrippingXX). It's better to do it using svn functionality to keep track of where the selections came from:
 
  svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/python/StrippingSelections/ svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingArchive/python/StrippingArchive/StrippingXX/ -m "Create StrippingXX"
Changed:
<
<
Then add StrippingXX to __init__.py, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.
>
>
Then add StrippingXX to __init__.py, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.
 

Check lost tags in tag collector

Tag collector page is here

Revision 52013-05-17 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 10 to 10
 

Prepare test samples

Changed:
<
<
Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed.
>
>
Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed. It can make sense to take (parts of) several runs as the test sample if the running conditions or trigger configuration was changing significantly. One the run for the test sample has been identified, inform bookkeeping manager and ask to copy it to "freezer". The python options have to be prepared and put to Phys/StrippingSelections/tests/data/.
 

Actions after stripping selections deadline

Added:
>
>
After the deadline, ask stripping liaisons in WGs to prepare the configuration dictionaries in Phys/StrippingSettings.
 

Run memory use tests

Added:
>
>
To test the memory consumption, a single, reasonably long (100k-200k events) stripping job is needed. Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory as a function of event number:
  ./analyseDaVinciLog.py logfile.txt
  display plot.png
To run safely in the production, jobs don't have to take more than 4 Gb of memory. Make sure there are no big jumps in memory use after ~50-100k events. They can indicate memory leaks. In case of doubt, use valgrind to find memory leaks. Memory auditor can also help to find an algorithm with memory leak. To enable it, add the following to the DaVinci options:
  ApplicationMgr().ExtSvc += [ 'AuditorSvc' ]
  AuditorSvc().Auditors += [ 'MemoryAuditor' ]
 

Run bandwidth tests

Run line independence tests

Line: 55 to 68
 

Prepare stripping archive

Added:
>
>
StrippingArchive is the package where all versions of released stripping selections are stored. Production uses selections from StrippingArchive rather than from StrippingSelections which are used only for development. Once the stripping selections are fixed, copy the contents of the StrippingSelections into StrippingArchive in the appropriate subdirectory (StrippingXX). It's better to do it using svn functionality to keep track of where the selections came from:
 
  svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/python/StrippingSelections/ svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingArchive/python/StrippingArchive/StrippingXX/ -m "Create StrippingXX"
Changed:
<
<
Then add StrippingXX to __init__.py, to the list of _known_strippings_ and to _stripping_help_
>
>
Then add StrippingXX to __init__.py, to the list of _known_strippings_ and to _stripping_help_. Sometimes, the new stripping uses the same selections as the older one without changes, and the only difference is in StrippingSettings. In this case, it is unnecessary to duplicate selections in StrippingArchive. Instead, use _duplicate_strippings_ dictionary in __init__.py to tell which archive to use.
 

Check lost tags in tag collector

Tag collector page is here
Line: 92 to 107
 

Run validation

Run production

\ No newline at end of file
Added:
>
>

Prepare web documentation

Revision 42013-05-17 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 6 to 6
 

Actions before stripping selections deadline

Added:
>
>
The deadline for submitting new or modifying the existing stripping selections is decided by the PPG, taking into account the time of conferences where the new results have to be presented, processing time and the time to prepare the stripping and DaVinci release after the deadline. The latter can take up to one month and depends on the number of iteration required to reach the limits on bandwidth and/or timing.
 

Prepare test samples

Added:
>
>
Test sample should be provided to authors of stripping lines to tune their selections. It has to be done well in advance, ~one month before the deadline. Typically the test sample is one run containing about 10M events, taken with the running conditions typical to the data sample to be processed.
 

Actions after stripping selections deadline

Run memory use tests

Revision 32013-04-23 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Line: 6 to 6
 

Actions before stripping selections deadline

Added:
>
>

Prepare test samples

 

Actions after stripping selections deadline

Run memory use tests

Revision 22013-04-22 - AntonPoluektov

Line: 1 to 1
 
META TOPICPARENT name="LHCbStripping"
Changed:
<
<

Stripping coordinator's cheatsheet

>
>

Stripping coordinator's cheatsheet

 
Changed:
<
<

Stripping release procedure

>
>

Actions before stripping selections deadline

Actions after stripping selections deadline

Run memory use tests

Run bandwidth tests

Run line independence tests

Release Stripping project

  Help on release procedure is at PrepareProjectRelease page.
Line: 64 to 74
 

Tag StrippingSys

Use tag collector, add sensible release notes
Added:
>
>

Add Stripping project to lhcb-prerelease slot in the nightlies

Ask for release in savannah

Prepare AppConfig options

Prepare production steps

Run validation

Run production

Revision 12013-04-19 - AntonPoluektov

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LHCbStripping"

Stripping coordinator's cheatsheet

Stripping release procedure

Help on release procedure is at PrepareProjectRelease page.

Check nightlies and correct if needed

Nightlies page is here.

To run the test of a package:

   cd cmt
   cmt TestPackage
or to run a single test, assuming the tests are in a directory davinci.qms and the test is called mytest.qmt:
   cmt TestPackage davinci.mytest

To update test reference:

   cd tests/refs
   mv ***.ref.new ***.ref

Prepare StrippingSettings database

  cd Phys/StrippingSettings
  cd cmt
  cmt make
  cd ../python/StrippingSettings
  python makeDB.py StrippingXX
  mv stripping.tmp ../../dbase/strippingXX

Prepare stripping archive

  svn cp svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingSelections/python/StrippingSelections/ svn+ssh://svn.cern.ch/reps/lhcb/Stripping/trunk/Phys/StrippingArchive/python/StrippingArchive/StrippingXX/ -m "Create StrippingXX"
Then add StrippingXX to __init__.py, to the list of _known_strippings_ and to _stripping_help_

Check lost tags in tag collector

Tag collector page is here

Update dependencies in Stripping project

  cd <where you have plenty of space>
  getpack -P -H STRIPPING HEAD
  cd STRIPPING/STRIPPING_HEAD/
  emacs cmt/project.cmt # Update dependencies here
  cd cmt/
  svn commit -m vXrY
  cd ..
  tag_package -P Stripping vXrY

Tag packages in Stripping project

Use tag collector

Tag StrippingSys

Use tag collector, add sensible release notes
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback