Tau Validation Automation

Introduction

The automatic Tau Validation submition package serves to simplify the submition of the validation tasks. The Validation Group tests various updates and configurations to the MC production algorithims. When a request is sent from The Validation group, it usually requires one to test over many different configurations. The Validation group requests are divided into Tasks, each containing a test sample ( The new updated configuration of a package), and 1 to 2 reference samples (a previously validated configuration as an example). The validation requires one to run over various datasets, each with its own configuration, and running the whole validation can be quite cumbersome. For this purpouse, the Tau Validation package has been automised in order to leviate the time consuming validation tasks. The automation uses a combination of Python and Bash scrips in order to create the nessesary Pathena commands, set up Athena and other packages, submit the jobs, download the resuls, compare refence and test samples and upload the results to the Tau Validation page.

There are two sections to this automatic validation, each of which is documented below with the respective scrips in the package, and explenations on how they can be edited for future development (Work In Progress).

Checking out the validation package

The validation package needs to be checked out. The steps to follow are documented in the Manual Validation documentation here.

Creating a New User

Once the package has been checked out you need to update variables in 2 different scripts. These are the share/jobSettings.py and python/userSettings.sh .

One needs to edit the following lines of code:

jobSettings.py

user='ghamity'
workDir='/afs/cern.ch/user/g/ghamity/testarea2/TauValidation/'
validationDir='/afs/cern.ch/user/g/ghamity/testarea2/TauValidation/PhysicsAnalysis/TauID/TauValidation/share/'
dummyFileDir='/afs/cern.ch/atlas/groups/validation/Tau/DummyFile/group10.perf-tau.62855_008179.EXT1._00008.AOD.pool.root'
setupFileDir='./SetupFiles/'

• user: Put your lxplus username here.
• workDir: This should be the path of your test area TauValidation directory.
• validationDir: This is the path to your share/ directory where the validation part 1 files are found.
• dummyFileDir: The validation makes use of a dummy file which can be found on the Tau Validation group directory (/afs/cern.ch/atlas/groups/validation/Tau/DummyFile/). You can copy this file to your local directory and call it from there, or call it straight from its location in the group directory.
• setupFileDir: This is the folder to which the preconfig file and configuration folders will be saved (directory needs to exist). One could make this the python/ directory and hence would eliminate the need to copy the configuration folders before starting part 2 of the validation.

do_validation.sh

user='ghamity'
amiUser='ghamity'
author='Guillermo Hamity'
institute='The University of the Witwatersrand'
storageDir='/afs/cern.ch/work/g/ghamity/TauValidationResults/'
packageDir='/afs/cern.ch/user/g/ghamity/testarea2/TauValidation/PhysicsAnalysis/TauID/TauValidation/python/'
tempDir='/tmp/ghamity/'
uploadDir='/afs/cern.ch/atlas/groups/validation/Tau/'

• user: lxplus username.
• amiUser: Your ami username
• author: Write the name you wish to display in the cration of a beemer presentation.
• institute: Your current institute
• storageDir: A location where the results will be stored. We suggest using your work directory on afs as it has more space.
• packageDir: The directory on your afs profile of the python folder in the tau validation package
• tempDir: A temporary directory location where all the work is done. Makesure it is an empty or non-existent directory..
• uploadDir: Path to the tau validation group's directory. Only change this if the afs location we are uploading to changes.

Remember to have a / at the end of all path names.

Part 1:

In Part 1 of the validation all the samples for all configurations are configured to be run over by the Tau Validation analysis package and the jobs are then submitted to the CERN computing grid. The tau analysis package histograms the necessary variables that need validation and produces output
datasets.

A set up file ( valSetup.config ) is used to set up the tasks, specifying athena release, jobOption files, etc...
Instructions on how to set up the valSetup.config can be found here. This file is found in the share/ directory, and setting it up is the most important part of the automation process. New users should familirise themselves with setting it up.

The jobs then need to be set up and submitted. The datasetests required are attained using dq2, which needs to be set up. Then the jobs and datasets are all sent off to the grid. This is all taken care of by doing:

python RunSumbmission.py option

The options are:

without creating preconfig file (old way)
--t Testing the submission
--s Submit

creating preconfig file
--ts Test the submission with printing out the pathena command
--gp Generate a preconfiguration file
--ts Test the configuration using the preconfiguration file
--sp Submit using the preconfiguration file

The most up to date procedure would be to follow the following steps:

1. Set up dq2
2. Run the RunSubmission script by typing: python RunSubmission.py --gp
   This sets up a preconfiguration file which is used to find the samples in the next step.
3. Start a new shell (this can be done by using 'screen').
   The reason one does this is that the Python bindings of dq2 and Athena clash, which caused problems when submiting from SLC6 machines. The preconfig file fixes the issue by ommiting the need to have dq2 and athena set up simultaneously.
4. python RunSubmission.py --ts
   A test is done which checks that the samples exist, and that the Athena versions are set up correctly.
5. python RunSubmission.py --sp
   Sumbition of the jobs is finally done.

If all goes correctly though, a folder containing (same name as whatever was under the VERSION section) all the configuration needed for part 2 will be created. The entire folder needs to be copied to the python/ directory for the second half of the validation.

---

Example 1:

If we submit a validation which we lable 2013-11-15-V2 with 2 tasks, then the directry will look like this:

• 2013-11-15-V2/
  • T1-2013-11-15-V2/
    • taskSetup0.config
  • T2-2013-11-15-V2/
    • taskSetup0.config

The taskSetup.config files have been automatically generated and contain information about the samples used in the task, as well as the output sample names. An example of a task 1 configuration file for user ghamity is given below:

NAME=Validation.T1-2013-11-15-0.e1574_s1820_r4973.vs.e1574_s1737_r4800
DIR=2013-11-15/Results/Task1a
TEVGEN=e1574
TSIM=s1820
TREC=r4973
REVGEN=e1574
RSIM=s1737
RREC=r4800
ZTAUTAU=147808
TTBAR=105200
ZEE=147806
D=user.ghamity.valid1.147808.PowhegPythia8_AU2CT10_Ztautau.recon.AOD.e1574_s1820_r4973.T1-2013-11-15-V2
D=user.ghamity.valid1.105200.McAtNloJimmy_CT10_ttbar_LeptonFilter.recon.AOD.e1574_s1820_r4973.T1-2013-11-15-V2
D=user.ghamity.valid1.147806.PowhegPythia8_AU2CT10_Zee.recon.AOD.e1574_s1820_r4973.T1-2013-11-15-V2
D=user.ghamity.valid1.147808.PowhegPythia8_AU2CT10_Ztautau.recon.AOD.e1574_s1737_r4800.T1-2013-11-15-V2
D=user.ghamity.valid1.105200.McAtNloJimmy_CT10_ttbar_LeptonFilter.recon.AOD.e1574_s1737_r4800.T1-2013-11-15-V2
D=user.ghamity.valid1.147806.PowhegPythia8_AU2CT10_Zee.recon.AOD.e1574_s1737_r4800.T1-2013-11-15-V2

The variables keep track of different information that is needed for the second part of the validation. NAME is the name given to the sample and is used throught the second part of the validation. DIR is the name of the directory where the results will be stored and will be created in the storageDir variable location set previously in the userSettings.py script. The other variables keep track of the tag and dataset information.

---

Before proceeding to part 2, you need to wait for your grid jobs to finish running. If some jobs fail on the grid, try resubmitting them using ’pbook’ first. If this fails you can just resubmit the entire validation (maybe need to exclude some sites by editing the ’prun’ command in grid submit auto.sh), or comment out the sections of the tasks that succeeded and use the resubmission feature.

Part 2:

The second part of the validation involves using the scripts under the python directory in the Tau Validation package. Make sure that you have already edited the python/userSettings.py script as explained in the 'Creating a New User' section above. Once a user is set up we can look at the next steps of the validation. This is broken down into a couple parts which are all done using one script and different command line arguments.

Downloading the Data

First we need to download the data from the tasks we submitted. Make sure you have copied the folder containing all the
configuration files to the python directory. Now set up dq2. Once dq2 is done, run the following command:

. do_validation.sh -d task_configuration_folder/

The -d option tells the script we want to download the data files. The task configuration folder is the name of the folder containing the configuration files (this would be 2013-11-15-V2 for Example 1 above). Make sure to include the / at the end of the folder name or things will break. This takes a while to run so it may be a good idea to leave it to run in a screen session.

Get the ami tags

Once all the data is downloaded we need a new shell for the next part, so logout and back in (or close your screen session and start another). In the new shell, set up athena release 17.2.0.2 in your test area (asetup 17.2.0.2,here or asetup 17.2.0.2,slc5 if running on slc6). Any athena release will do actually, this is just the standard that we have been using. You can find different Athena releases documented here. Now cd back to the python directory. We now want to get the ami information for the tags used in the validation. To do this run the following command:

. do_validation.sh -a task_configuration_folder/

The -a option tells the script we want to download the ami information for our different tags. The task configuration folder is the same as previously. You are prompted for you ami password before it runs. This should not take long to run.

Do the validation

Once we have the ami tags we wish to do the actual validation. To do this run the following command:

. do_validation.sh -s task_configuration_folder/

The -s option tells the script we want to do the actual validation. The task configuration folder is the same as previously. This takes a very long time to run and I suggest running in a screen session again (setup screen before the ami tags part so that you don’t need to run the asetup again). Once this is done, go to your results directory and you should find a folder containing all the results of the validation for the given date. If you browse the results directory under the given date folder for a given task you can find what will be uploaded to the website. Opening the index.html files allows you to give a description in the allocated section if you wish before uploading.

Upload the results

The final step is to upload the results to the validation area. This is just doneusing the following command:

. do_validation.sh -u task_configuration_folder/

The -u option tells the script we want to upload our results. The task configuration folder is the same as previously. No athena setup is required for this step.

Updating the Validation website

Once the validation has been completed the results need to be uploaded onto the validation page. Inside the tau validation directory (/afs/cern.ch/atlas/groups/validation/Tau) you will find various subfolders and files used to create this validation page.

/afs/cern.ch/atlas/groups/validation/Tau/
         2013-10-4-ghamity/
         2013-11-02-kbristow/
         2013-11-13-kbristow/
         2013-11-15-ghamity/
              index.html
              Task1a/
              Task2a/
              Task4Part1a/
              Task4Part2a/
              valPres.tex
         DummyFile/
         index.html

The index.html file takes care of the main page in the TauValidation. The 2013-* directories have been created for previous validations in the upload results part of the validation. They contain an index.html file which creates a main page linking to different taskes, a valPres.tex file which can be used to create a beamer presentation, and subfolders containing the pages of the different tasks.

Once the results have been uploaded one needs to link the index.html page inside the validation directory , to this index page. The results can then be seen on the validation page.

Special Cases:

Instructions for comparing same samples

Occasionally it is necessary to compare the exact same samples against one another. There have been 2 previous occasions where this has occurred. The automatic validation framework has not been built to handle these situations so a manual trick is required to fool the validation software into working. The 2 occassions where this was required were when we were comparing new tau ID to old tau ID in the validations and comparing 2 different versions of athena to each other. Since these are not data dependant checks (we are checking changes in software only) we need to run on the exact same samples. Below is the task description for the second case:

(2) Task : Validation of digi+reco with 17.8.0.2 (64bits) and gcc47. No differences expected with respect to the gcc46 compiler. Evgen tag e1574

Test : s1469_r4847 (17.2.0.2,17.8.0.2) - gcc47 samples (valid2)
Reference a) s1469_r4847 (17.2.0.2,17.8.0.2) - gcc46 samples (valid1)

This is the more complicated of the 2 cases we have had to do this since 17.8.0.2 itself requires a few special settings so we will look at this one.

The problem comes in because data sets are named based on their task they fall under. So here, irrelevant of the athena versions the jobs would both be sent to the grid with the same output dataset name. This would fail. The trick comes in by splitting the job into 2 different jobs. Each with a fake counter part data set. The first job must have the gcc47 validation as the test with a fake reference. The second part can be done however we like. The valSetup.config file looks like this in the tag section in the way I have set it up:

##T2Part1
e1574_s1469_r4847;AtlasProduction-17.8.0.2-slc6-64-gcc47;jobo:64bit;cmt:x86_64-slc6-gcc47-opt
e1574_s1469_r4764;17.2.11.1-slc5-gcc43;jobo:OldTauID

##T2Part2
e1574_s1469_r4764;17.2.11.1-slc5-gcc43;jobo:OldTauID
e1574_s1469_r4847;AtlasProduction-17.8.0.2-slc6-64-gcc46;jobo:64bit;cmt:x86_64-slc6-gcc46-opt

Note that anything we enter into the Athena setup section (the second space between the :'s) get all -'s changed to ,'s and is used as the input for asetup.
E.g. AtlasProduction-17.8.0.2-slc6-64-gcc47 means that asetup is called: "asetup AtlasProduction,17.8.0.2,slc6,64,gcc47"
For this case since 17.8.0.2 is a 64 bit release, we want it compiled with gcc47 and it is only an slc6 release, we need to call it as above (Information provided by Jose Garcia). Since this is slc6 we had trouble running this on slc5 lxplus machines so slc6 machines are suggested. There is also an issue with pathena when trying to submit to the grid with an slc5 release even though we have an slc6 release locally. So we need to specify the cmtConfig for the pathena command. This is the very last option in the tag string which means we need all options prior to it to be initialised. The following is the current structure of the tag string:

tags;athena;joboptionsfile;extracodeindex;cmtconfig

The reference is just a dataset from task1 in this validation set and could be any dataset we want. For T2Part2 I have used the reference as the reference with a fake test but that could be switched around.
You should be able to submit this job now. We now continue the normal validation process until after we get the data for second part of the validation. Once we have downloaded the data, we rename T2Part2 reference to the exact corresponding names of the reference data in T2Part1 and copy them to replace the reference in the first part. This way we are fooling the validation package into thinking it is using different data sets and it will work. You can then run the validation. For aesthetic reasons, you could manually go into the html file produced and rename the title of the validation and put a note about what happened but the validation will run which is the important part.

-- GuillermoHamity - 28 Nov 2013