GaudiDiff, for intelligent auto-diffing of two gaudi jobs

Introduction User guide What other options are there. Examples of use Comparing your local build with the release build Compare your local changes with the head from yesterday Compare your changes in the morning with your changes in the afternoon Compare results on two different platforms Compare the same options between two different people Compare two slightly different sets of options I would like to do the diffing myself, can I do that? Can I use this in my qmtest? (the result of task #50644 )

.......

Introduction

A common annoyance with LHCb jobs is the huge amount of stdout they throw at you whenever you run. It is quite difficult to pick out of the large number of changes small differences which really matter. The large amount of printout often requires piping to a file, and then when you are looking for the changes of your latest modification often they are buried.

Another issue arises when we consider our automated testing framework. Due to small variations in printout and modifications made by others each day much lower down the stack, our tests can be quite volatile. Although we ask our developers to check the nightlies, it is difficult for a them to really know if it is they who caused a certain set of test failures.

The parsing and diffing of stdout, to understand if certain modifications were due to you, and whether they are important, is a complicated task for which every devoloper has been used to needing to set up their own little test bench, when we're all doing the same things really, so a common script would be better.

gaudidiff.py is a wrapper around gaudirun.py which creates local reference files as a baseline against which to test your future or existing changes, and compares them for you, intelligently and automatically.

User guide

Prerequisites:

• you will have a Gaudi job which you want to use as your baseline for comparison.
• you will have write access somewhere to store the various reference files
• ideally you will be running within a significantly up-to-date software stack, LHCb >= 37r1 is preferred so that you get the newest Timing Table features (to see what stack you depend on, try svnProjectDeps -P SomeProject vXrYpZ

Help?:

• gaudidiff.py with no options will print a load of help information and exit.

#setup some project and check your options to gaudirun result in what you expect. For example:
$ SetupProject Moore
$ gaudirun.py  $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt
 
#run the same thing with gaudidiff to store the reference files
$ gaudidiff.py $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

...
#make whatever changes you wer thinking of, in this example we will compare two different options files:

#run a second time and the results of this second run will be automatically compared with the first
$ gaudidiff.py $MOOREROOT/tests/qmtest/moore.qms/database.qms/2011.qms/default.qmt

What other options are there.

There are a lot of options:

• rename the reference files (before or after)
• re-run the checks without changing the reference files
• reset force recreation of the reference files

... and even more options!!

Best way to see this all the options is to add --help, or run gaudidiff.py with no options or arguments.

Examples of use

Comparing your local build with the release build

#setup the release build with no local changes at all
$ SetupProject Moore --no-user-area
$ gaudidiff.py  $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

#setup with your local area
$ SetupProject Moore
$ gaudidiff.py  $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

Compare your local changes with the head from yesterday

#setup the release build with no local changes at all
$ SetupProject Moore --no-user-area HEAD --nightly lhcb-head
$ gaudidiff.py  $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

#setup with your local area
$ SetupProject Moore HEAD --nightly lhcb-head
$ gaudidiff.py  $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

Compare your changes in the morning with your changes in the afternoon

$ SetupProject Moore 
$ gaudidiff.py  --reset --initial-run morning $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt
...
$ make
$ gaudidiff.py  --test-run afternoon $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

Compare results on two different platforms

$ LbLogin -c x86_64-slc6-gcc46
$ SetupProject Moore
$ gaudidiff.py  --initial-run x86_64-slc5-gcc46 $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

$ bash -l
$ LbLogin -c x86_64-slc6-gcc48
$ SetupProject Moore
$ gaudidiff.py  --initial-run x86_64-slc6-gcc46  --test-run x86_64-slc6-gcc48 $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

Compare the same options between two different people

#setup the release build with no local changes at all
#person 1
$ cd ~/public/forPerson2
$ SetupProject Moore 
$ gaudidiff.py  $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

#person 2
$ SetupProject Moore
$ gaudidiff.py  --initial-run ~person1/public/forPerson2/before $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt

Compare two slightly different sets of options

$ SetupProject Moore 
$ gaudidiff.py  $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt --option="from Configurables import Moore; Moore().EvtMax=11;"

$ gaudidiff.py  $MOOREROOT/tests/qmtest/moore.qms/database.qms/2010.qms/default.qmt --option="from Configurables import Moore; Moore().EvtMax=12;"

I would like to do the diffing myself, can I do that?

• YES: the script will tell you what files it has produced, running the script with no options will tell you what is in those files, and then you can diff them however you want to.

Can I use this in my qmtest?

• YES: once it is integrated into Lbscripts this will be easy.
• you can store the reference files in the usual .ref directory and then use --initial-run ../refs to make the comparison.

-- RobLambert - 26 Mar 2014