Test a Stripping line with git

Foreword

If you are completely unfamiliar with git, a good place to start is the Git4LHCb web page.

When testing a stripping line, there are two possibilities. Either you want to test an existing stripping line that has been run already in a stripping campaign, or you want to develop or modify a stripping line to be run in a future stripping campaign. We consider both cases, where we assume the existing stripping line is run in S34 in DaVinci v44r4, and the line being developed is in 2018-patches.

Setup the environment

Before you can test a stripping line you need to set up the environment. For an existing stripping line, the DaVinci version is well defined with version vXXrYYpZZ. For a campaign under development, the DaVinci version is the name of the development branch, for example, 2018-patches (the other possible branches are 2017-patches, stripping21-patches, stripping21-firstpass-patches or lhcb-reco14-patches). If you are unsure, check with the Stripping liaison of your WG.

To check which platforms can be used with a particular DaVinci version, run the following command, e.g. for DaVinci v44r4:

lb-sdb-query listPlatforms DaVinci v44r4

then set the platform with:

lb-set-platform x86_64-centos7-gcc7-opt

Note that this doesn't work for campaigns under development. In this case, you can check with the Stripping liaison or the instructions for the campaign, or check the platforms that appear in the nightly builds, e.g. for 2018-patches

After you set the platform, then you are ready to set up DaVinci. If you are testing on an existing DaVinci version, you can do the following:

lb-dev DaVinci/vXXrYYpZZ
cd DaVinciDev_vXXrYYpZZ

If it's a campaign under development, then you will take the latest nightly build:

lb-dev --nightly lhcb-2018-patches/latest DaVinci/2018-patches
cd DaVinciDev_2018-patches

Checkout the relevant version of Phys/StrippingSelections

First set the project to be used, in this case "Stripping"

git lb-use Stripping

then checkout and compile StrippingSelections from the tagged branch or the currently used branch for the development of a certain Stripping campaign (check with liaisons if unsure). In this example, we assume you are using DaVinci v44r4, in which case the Stripping tag (as shown in the DaVinci v44r4 release notes) is v12r3p1:

git lb-checkout Stripping/v12r3p1 Phys/StrippingSelections
make configure
make

If the campaign is under development, then stripping won't be tagged yet. In this case, we assume that the development branch is 2018-patches:

git lb-checkout Stripping/2018-patches Phys/StrippingSelections
make configure
make

Now you can modify the content of StrippingSelections by adding or modifying a line; every time you do it, the package must be recompiled.

make purge
make

Add a new line

If you are modifying an existing line you can skip this part. The lines in StrippingSelections are organised in WGs. Suppose we want to add the module StrippingMyCharmDecay.py to the lines of the Charm WG. We need to copy StrippingMyCharmDecay.py to the Phys/StrippingSelections/python/StrippingSelections/StrippingCharm/ directory

cp <path_of_your_file/StrippingMyCharmDecay.py Phys/StrippingSelections/python/StrippingSelections/StrippingCharm/>

Since this is a new line, it has to be added to Phys/StrippingSelections/python/StrippingSelections/StrippingCharm/__init__.py in the, open the init.py file with your favourite editor and add your line to the end of the "_selections" list at the beginning of the file (omit the .py extension)

_selections = [ "StrippingSomeCharm","StrippingSomeOtherCharm", "StrippingMyCharmDecay"]

Now you are ready to recompile StrippingSelections, so as usual do

make purge
make

If you get compilation errors or a warning due to your line please fix them and then recompile. If get an error/warning NOT due to your line, please inform the stripping coordinators.

Modify an existing line

To modify a an existing module is sufficient to apply the changes using your favourite editor and then compile again

make purge
make

Test a stripping line

To set up the correct environment variables, issue the command

./run bash --norc

This will bring you to a new bash shell with the correct environment settings (e.g. you will be able to use gaudirun.py). You will need to do this every time that you recompile the package.

The test scripts for users can be found under $STRIPPINGSELECTIONSROOT/tests/users. Among others you can find one called TestMyStrippingLine.py. To test your line by simply editing the file to set the variable 'confname' with the value of the key 'NAME' in your default_config (or directly the key identifying your specific configuration as explained here). So if the name of your configuration is "MyCharmLines" you have simply to set

confname='MyCharmLines'

At this point you can finally test your line by doing, from any location

gaudirun.py -T Phys/StrippingSelections/tests/users/TestMyStrippingLine.py >& someOutputDir/StripTest.log & 

the -T option allows for a better handling of the memory of the process and it's not strictly necessary for testing purposes but it's recommended.

The output of the script will be in the log file, and contains the usual printout of StrippingReport with retention and timing of your lines. In addition, the output DST and/or micro-DST will be produced according to the settings you specified in the default_config (MDST.DST included). The TES location at which your lines are saved together with all related info and flavour tagging info (if requested) are the same as the stripping that will eventually run in production so that you can run your ntuple-maker script on these DSTs to check that all the information is stored correctly. If for some reason you are unhappy with the configuration or implementation of your lines, just go back issuing the command

exit

This will bring you to the original bash session where you can apply your modifications, recompile and start again testing as in explained in this section.

If you are returning with a "fresh" lxplus session, return to the

DaVinciDev_vXXrYY.pZZ

directory, and you can run as follows:

./run gaudirun.py -T Phys/StrippingSelections/tests/users/TestMyStrippingLine.py >& someOutputDir/StripTest.log & 

IMPORTANT

Please do not import anything explicitly from the StrippingSelections package, which is used for development. In production the LineBuilders are copied into StrippingArchive to freeze them. Explicit imports from StrippingSelections break this scheme and may cause your line to crash if there are differences with what is put in the StrippingArchive and what is in StrippingSelections.

Before I commit my stripping line

Once you have successfully tested your line inform your WG group liaison about the results of the testing and ask for permission to commit to git, specifically we want to know the rate of the new line and the timing information. For MDST lines, the rate should be <0.5% and for DST lines, <0.05%. For new lines, please inform also the coordinators before committing the new code. Add the results of the tests as comment to the merge request.

Commit a new or modified line

*You must commit to a new branch and NEVER to the master of the project. The branch will be created in the moment of pushing the changes. Immagine that the name of the line is my_new_branch (please use something that you can identify afterwards), and that I need to change a couple of python files called file1.py and file2.py.

If the files are new first add them to the repository

git add file1.py file2.py

Then, issue the following two commands

git commit -m "message that identifies my commit"
git lb-push Stripping my_new_branch

Prior to making a merge request, merge changes from the master branch into your updated branch with

git fetch --all
git lb-checkout Stripping/master Phys/StrippingSelections

If there are any conflicts, resolve them and test again. To mark a conflicting file as resolved do

git add filename

then commit and push again to the same branch

git commit -m "Merging updates from master & resolving conflicts"
git lb-push Stripping my_new_branch

Were is my commit?

In the gitlab web page https://gitlab.cern.ch/lhcb/Stripping you can find all of the existing branches under Repository/Branches. Once you are satisfied with the results of your work you can ask to merge your branch with the master, so the changes or new lines will be executed in the next stripping campaign. To do so, click on the box Merge Request that follows the name of the branch, and the coordinators will check your changes and eventually accept it. You will be contacted by the coordinators in case that they found something to be changed.

REMEMBER ALWAYS TO ASK FOR THE MERGE REQUEST ONCE YOU ARE SATISFIED WITH THE CHANGES. AND CHECK HOW THAT YOUR MERGE REQUEST CONTAINS ONLY YOUR OWN CHANGES!