Stripping project description

The stripping project has been moved to gitlab: https://gitlab.cern.ch/lhcb/Stripping and contains the following packages:

• CommonParticlesArchive
• DisplVertices
• IncTopoVert
• IsolationTools
• StandardParticles
• StrippingAlgs
• StrippingArchive
• StrippingConf
• StrippingNeuroBayes
• StrippingSelections
• StrippingSettings
• StrippingTCK
• StrippingUtils
• TMVASelections

Please note that CommonParticles has been moved to Phys.

Typically, the liaison acts only on (in order of frequence) StrippingSelections and StrippingSettings. If you think a modification of some other package is needed, please discuss it first with the coordinators.

Some notes on good practice

• Run tests periodically, TestMyWGfromSelections.py every couple of days before the deadline for commits. Start using TestMyWGfromSettings.py when the LineConfig can be filled up (usually after the folder python/StrippingSettings/StrippingXY is created and announced by the coordinators)
• Inform the coordinators about errors and suspicious warnings in the logfiles
• Share the final LineConfig with your WG to make sure all the lines are there and properly configured

How to start

Create a directory where you have lots of space and check which DaVinci version is required for the tests (here it's DaVinci v41r4p1). Both the DaVinci version and Stripping branch will be provided

mkdir MyStrippingDir
cd MyStrippingDir
lb-dev DaVinci v41r4p1
cd DaVinciDev_v41r4p1
git lb-use Stripping
git lb-checkout Stripping/<branch> Phys/StrippingSelections
git lb-checkout Stripping/<branch> Phys/StrippingSettings
make

Sometimes you might need to checkout and build other packages, which include fixes to eg. RelatedInfoTools as the tests detailed below might fail due to bugs in those packages/tools.

StrippingSettings

StrippingSettings is the package where the configuration that is run in production is defined. Although the line configuration may be copied from the default_config in StrippingSelections, the two are independent. Therefore the default_config are ignored in production, all that matters is what is in StrippingSettings. During stripping campaigns it is therefore normal that some fine tuning is performed directly on this package although it is good practice to keep the settings in line with the default_config in StrippingSelections. For each XY Stripping, each liaison should take care of the WG configurations, stored at python/StrippingSettings/StrippingXY/LineConfigDictionaries_.py

IMPORTANT: Check carefully what enters in your LineConfig, usually not all what is in StrippingSelections has to run (this is the case for restrippings, where we usually have a reduced number of lines).

Overview of tests

There are several test scripts to be run by the liaisons in the folder Phys/StrippingSelections/tests/liaisons/

* TestMyWGfromSelections.py: test all the lines from the corresponding Working Group using StrippingSelections. Please note that the final tests of the Stripping should be done from Settings.

* TestMyWGfromSettings.py: test all the lines from the corresponding Working Group using StrippingSettings.

These tests produce:

1. A (large) log file that needs to be inspected carefully.
2. DST and mDST files.

In the log file, all the errors related to the stripping should not exist. All the Warnings related to the stripping are potential sources of problems, so should be treated carefully as well.

In the (m)DST all the information of the relevant lines should be present and readable.

* analyseDaVinciLog.py: test the memory consumption of all the lines. It takes as input the log file produced by the step above and returns a printout and an image with the memory usage vs events processed.

IMPORTANT: when running tests, please pass the option "-T" to gaudirun.py so that one has a production-like memory usage.

Prepare StrippingSettings database

As anticipated, to propagate changes to the LineConfigs, a simple recompilation is not sufficient. Here are the steps to follow:

1. Modify some lines
2. Recompile by executing

   make purge && make

   in DaVinciDev _v41r4p1
3. To collect all the information in the dictionaries of the modules of your Working Group, do

   ./run python Phys/StrippingSelections/tests/liaisons/collect_dicts.py 'your_WG'

   , which will create dictionaries.py. For an incremental re-stripping edit dictionaries.py as required.

4. mv dictionaries.py Phys/StrippingSettings/python/StrippingSettings/StrippingXY/LineConfigDictionaries_your_WG.py

5. Go to $STRIPPINGSETTINGSROOT/python/StrippingSettings
6. Issue the following commands (case sensitive):

      ./run python makeDB.py StrippingXY
      mv stripping.tmp ../../dbase/strippingXY
      

7. Navigate back to DaVinciDev _v41r4p1 and copy Phys/StrippingSelections/tests/liaisons/TestMyWGfromSettings.py to this folder.
8. Edit your copy of TestMyWGfromSettings.py: at the beginning of the file, change my_wg and stripping. Make sure you are running over the correct input data located in $STRIPPINGSELECTIONSROOT/tests/data and that you are using the proper CondDB and DDDB tags.
9. Run the test

   ./run gaudirun.py -T TestMyWGfromSettings.py > log_file 

   and then inspect the output carefully.

In case you want to test from StrippingSelections, you do not need to build the dictionary and can simply execute

./run gaudirun.py -T TestMyWGfromSelections.py > log_file 

after changing the required inputs (see above for the test from StrippingSettings). Inspect the output carefully.

To test the global memory consumption, a single, reasonably long (100k-200k events) stripping job is needed (see step above). Store the log of this job in the file, then use analyseDaVinciLog.py script to plot the memory use as a function of event number.

  python(2) analyseDaVinciLog.py logfile.txt -p name_of_your_plot.png
  display name_of_your_plot.png

The options -c and -l are currently not working.

To run safely in the production, jobs don't have to take more than 4 Gb of memory. Make sure there are no big jumps in memory use after ~50-100k events. They can indicate memory leaks. If you are unsure if the plot indicates a memory leak, please contact the coordinators.

Some guidelines using B2CC as example

Here are some guidelines prepared by Carlos Vazquez Sierra to help future liaisons, which rely still on SetupProject.