Simultaneous setup of MPI and SUSY group software

Current SUSYTools version: SUSYTools-00-00-63

This page details the steps currently needed to set up the MPI software for SUSY analyses, and run the default 4 lepton analysis. The key problems are that the MPI software runs in athena, while the SUSY group software (SUSYTools and the packages it depends on) are optimised for RootCore use, and often have limited athena compatibility.

For the moment, therefore, there is a lot of code editing etc needed in order to get everything working. The plan is to contact the developers of the various packages and try to persuade them to include the changes in future SVN versions, to make things easier in the future.

Max has pointed out that there is a slightly easier way of setting some things up by setting ROOTCOREDIR manually, even when running in athena. This makes some of the code changes unnecessary, and forms a valid alternative to some of the instructions on this page.

General note

Most lines can be copied and pasted, either onto the command line or into a file. There is one major exception: whenever you see svn+ssh://, this needs to be changed to svn+ssh:// if your CERN user name is different to your username on the local system.

If there are questions, or you have trouble locating any of the files mentioned, please contact MichaelFlowerdew.


11th June: Update to recommendations for DGTriggerReweight, removing the unnecessary linking in the InstallArea.

Initial setup

Every time you log in, set up athena using your favourite method. For example, using Mike's script:

source -t,AtlasProduction -a MPIsetup/

A lot of packages need to be checked out. To make this easier you can create an ssh key to give you password-less access to SVN. This procedure is stolen from here.

First, execute these lines on your local system:

ssh-keygen -t dsa       # Don't provide a password
scp ~/.ssh/
Then, log in to lxplus and do
cat ~/ >> ~/.ssh/authorized_keys

Still on lxplus, create a file ~/.ssh/config, with the following contents:

Host lxplus 
Protocol 2 
PubkeyAuthentication no 
PasswordAuthentication yes

Host svn 
GSSAPIAuthentication yes 
GSSAPIDelegateCredentials yes 
Protocol 2 
ForwardX11 no

Package checkout

Execute all commands in this section from $TestArea.

Check out the MPI analysis (This should now no longer require a password)

cmt co -r VersatileUserAnalysis17-00-00-18 svn+ssh://

Fix up usage of SVNGRP, before checking out the other MPI packages

export SVNGRP=svn+ssh://
cmt co -requirements VersatileUserAnalysis17/cmt/requirements

Check out SUSYTools and the packages it depends on. The package versions should match those in SUSYTools/python/, but some are skipped either because they are unnecessary or already in the athena release.

cmt co -r SUSYTools-00-00-63 PhysicsAnalysis/SUSYPhys/SUSYTools
cmt co -r PileupReweighting-00-02-02 PhysicsAnalysis/AnalysisCommon/PileupReweighting
cmt co -r JetSelectorTools-00-00-16 PhysicsAnalysis/JetMissingEtID/JetSelectorTools
cmt co -r GoodRunsLists-00-00-96 DataQuality/GoodRunsLists
cmt co -r MuonMomentumCorrections-00-05-03 PhysicsAnalysis/MuonID/MuonIDAnalysis/MuonMomentumCorrections
cmt co -r MuonEfficiencyCorrections-01-00-10 PhysicsAnalysis/MuonID/MuonIDAnalysis/MuonEfficiencyCorrections
cmt co -r egammaAnalysisUtils-00-02-76 Reconstruction/egamma/egammaAnalysis/egammaAnalysisUtils
cmt co -r egammaEvent-03-06-20 Reconstruction/egamma/egammaEvent
cmt co -r JetResolution-01-00-00 Reconstruction/Jet/JetResolution
cmt co -r JetUncertainties-00-05-01 Reconstruction/Jet/JetUncertainties
cmt co -r MissingETUtility-00-02-14 Reconstruction/MissingETUtility
cmt co -r ReweightUtils-00-02-06 PhysicsAnalysis/AnalysisCommon/ReweightUtils
cmt co -r CalibrationDataInterface-00-01-02 PhysicsAnalysis/JetTagging/JetTagPerformanceCalibration/CalibrationDataInterface
cmt co -r ApplyJetCalibration-00-01-01 svn+ssh://
cmt co -r ObjectSelectorCore-00-00-17 PhysicsAnalysis/AnalysisCommon/ObjectSelectorCore
cmt co -r TrigRootAnalysis-00-00-07 Trigger/TrigAnalysis/TrigRootAnalysis
cmt co -r DGTriggerReweight-00-00-04 svn+ssh://

Finally, check out the analysis packages

cmt co svn+ssh://
cmt co svn+ssh://

Code fixing.

The above code will not compile (or run) out of the box. This section contains the (near-to-minimal) changes that need to be made. All paths are relative to $TestArea.



Remove some SUSYTools source files that do not compile in athena:

for f in HforToolD3PD.cxx Reco2Lep.cxx Reco_dpd2.cxx TJVFReweighting.cxx TopTag.cxx 
mv ${f} ${f}_backup


Replace [ptbins] with [9999] throughout.

        double x[9999]; double xerr[9999];
        double y[9999]; double yerr[9999];


After this block

  if (tmparea != NULL) {
    maindir = tmparea;
    maindir = maindir + "/../";
insert these lines:
  else { // Athena
    tmparea = getenv("TestArea");
    if (tmparea != NULL) {
      maindir = tmparea;
      maindir = maindir + "/";

Then, comment out these two lines (they're near, but not next to, each other):

if (muon_sf_dir == "") muon_sf_dir = maindir + "MuonEfficiencyCorrections/share/"; // Default path
if (muon_momentum_dir == "") muon_momentum_dir = maindir + "MuonMomentumCorrections/share/"; // Default path

Next, add the "PhysicsAnalysis/SUSYPhys/" bits to these lines:

  if (multijes_file == "") multijes_file = maindir + "PhysicsAnalysis/SUSYPhys/SUSYTools/share/MultijetJES.config";
  if (jes_file == "") jes_file = maindir + "PhysicsAnalysis/SUSYPhys/SUSYTools/share/InsituJES2011_AllNuisanceParameters.config";
Finally, add "Reconstruction/Jet/" to this line:
  if (jer_file == "") jer_file = maindir + "Reconstruction/Jet/JetResolution/share/JERProviderPlots.root";


In InsituJES2011_AllNuisanceParameters.config, replace the line for the JESUncertaintyRootFile with this:

JESUncertaintyRootFile:      ${TestArea}/Reconstruction/Jet/JetUncertainties/share/JESUncertainty.root

Similarly, for MultijetJES.config:

MultijetJESUncertaintyRootFile:      ${TestArea}/Reconstruction/Jet/JetUncertainties/share/MJESUncertainty.root
AnalysisRootFile:                    ${TestArea}/Reconstruction/Jet/JetUncertainties/analysisPlots/UnknownFlavourComp.root


Remove the final parts of the paths from these two lines:

use JetResolution JetResolution-* Reconstruction/Jet/JetResolution
use MissingETUtility MissingETUtility-* Reconstruction/MissingETUtility
so that they look like this:
use JetResolution JetResolution-* Reconstruction/Jet
use MissingETUtility MissingETUtility-* Reconstruction

Uncomment the "use CalibrationDataInterface" line in cmt/requirements, and add these packages:

use egammaEvent egammaEvent-* Reconstruction/egamma
use ObjectSelectorCore ObjectSelectorCore-* PhysicsAnalysis/AnalysisCommon
use TrigRootAnalysis TrigRootAnalysis-* Trigger/TrigAnalysis
use DGTriggerReweight DGTriggerReweight-* 
use ApplyJetCalibration ApplyJetCalibration-* 
Important: the last two lines have a space at the end of the line!

A little further down, under public, add these lines:

library  SUSYTools ../src/*.cxx ../Root/*.cxx
apply_pattern installed_library

Optional: Reflex dictionary generation

If you want to use the SUSYTools classes in standalone ROOT/pyROOT, you can achieve this with a few more modifications. For further details, see this page. This example just adds access to one class (CrossSectionDB), it is trivial to add in more.

Create a file SUSYTools/SUSYToolsDict.h:


#include "SUSYTools/SUSYCrossSection.h"


Then, create a file SUSYTools/selection.xml:

<class name="SUSY::CrossSectionDB" />

Then, add these lines to cmt/requirements, just before the RTT section:

### dict generation ###
use AtlasReflex  AtlasReflex-*  External -no_auto_imports

apply_pattern lcgdict dict=SUSYTools \
 selectionfile=selection.xml \


To use the dictionaries in pyROOT (after compilation, of course!), do the following:

>>> import PyCintex
>>> PyCintex.Cintex.Enable()
>>> import ROOT
>>> mydb = ROOT.SUSY.CrossSectionDB('../data/susy_crosssections.txt') # Of course, adjust this to the correct path

Note that the middle "SUSY" is the namespace of the class.

Packages that SUSYTools depends on


Remove the "../src/*.cxx" bit from this line:

library CalibrationDataInterfaceLib "../Root/*.cxx ../src/*.cxx"
to leave just this:
library CalibrationDataInterfaceLib "../Root/*.cxx"


This file needs to be created, with this content:
use AtlasPolicy         AtlasPolicy-* 
use AtlasROOT           AtlasROOT-*           External
use PathResolver        PathResolver-*        Tools
use GaudiInterface      GaudiInterface-*      External

apply_tag ROOTMathLibs
apply_tag ROOTBasicLibs

library ApplyJetCalibration "../src/*.cxx ../Root/*.cxx"
apply_pattern installed_library

## This is so that the JERProviderPlots.root file gets copied to the share area
apply_pattern declare_runtime files="*.root"


Find this block:

  if (RootCoreDir != "") {
    if (gSystem->AccessPathName(_basePath+fn)==false) return _basePath+fn;
    if (gSystem->AccessPathName(_basePath+fn)==false) return _basePath+fn;
and add this immediately after
  else { // Fix for athena
    RootCoreDir = gSystem->Getenv("TestArea");
    printf("Looking for file in %s\n", (_basePath+fn).Data());
    if (gSystem->AccessPathName(_basePath+fn)==false) return _basePath+fn;
    printf("Looking for file in %s\n", (_basePath+fn).Data());
    if (gSystem->AccessPathName(_basePath+fn)==false) return _basePath+fn;


This file also needs to be created, with this as contents:

use AtlasPolicy         AtlasPolicy-* 
use AtlasROOT           AtlasROOT-*      External
use PathResolver        PathResolver-*      Tools
use GaudiInterface      GaudiInterface-*   External
use ReweightUtils   ReweightUtils-*      PhysicsAnalysis/AnalysisCommon

apply_tag ROOTMathLibs
apply_tag ROOTBasicLibs

library DGTriggerReweight "../Root/*.cxx"
apply_pattern installed_library

## This is so that the root files get copied to the share area
apply_pattern declare_runtime files="*.root"


Comment out this line:

  ClassDef(triggerReweight, 2); 


Comment out these two lines in initialize():

This fix will be unnecessary when we switch to MissingETUtility-01-00-05.

MPI core code


In the clean() method, after "delete m_my_met_composition", the pointer should be set to NULL:

      delete m_my_met_composition;
      m_my_met_composition = NULL;

SUSY analysis code


Change the MyAnalysisUtilities line to this:

use MyAnalysisUtilities MyAnalysisUtilities-* 
Note that this has a space at the end of the line.

Comment out all the "use" statements under "ATLAS packages" except SUSYTools, which should be changed to look like this:

use SUSYTools SUSYTools-* PhysicsAnalysis/SUSYPhys


MPI core packages

cd NtupleReaderAlg/cmt
cmt bro cmt config
cmt bro cmt make -j4

The rest of the code

Everything else can be compiled from SUSYAnalysis.
cd SUSYAnalysis/cmt/
cmt bro cmt config
cmt bro source
cmt bro cmt make -j4


cd $TestArea/SUSYAnalysis/run

I recommend creating a working directory within run/ and copying your job options there. For example:

mkdir SUSYtest
cd SUSYtest
cp ../ .

File lists

To run, you need to provide a list of files. If you are running at RZG, you can do the following:

cp /ptmp/mpp/flowerde/MPIsetup/FileLists.tar.gz .
tar -xzf FileLists.tar.gz 
This creates a directory called VUA18_MC11c, containing the text files you need

Additional input files

You will need a couple of files from the multilepton twiki, to provide additional scaling and correction factors (and these will occasionally need to be updated). Scroll right down to the bottom (where it says "Attachments") and download these files:


You will also need a "Good Runs List" (GRL) from this page. At the moment, we are using

Right-click to select "Save As" or similar, depending on your browser.

Job options editing

To run a test job, you need to edit the job options in a few places. First, replace

NtupleReaderAlg.fileList = "filelist.txt"
NtupleReaderAlg.fileList = "VUA18_MC11c/filelist_mc11_7TeV.105986.ZZ_Herwig.merge.AOD.e825_s1310_s1300_r3043_r2993.txt"

Then, set the number of events to a small number for testing (a few lines further down)

NtupleReaderAlg.numberOfEvents = 20

In addition, near the bottom (where the analysis tool properties are set), add this line:

analysisTool.GRL = "data11_7TeV.periodAllYear_DetStatus-v36-pro10_CoolRunQuery-00-04-08_Susy.xml"

Run it!

At last we get to run a test job:

athena > firstrun.out 2>&1
The "2>&1" makes sure that both stdout and stderr end up in firstrun.out, so you can examine it at your leisure. An example file (run over 2000 events) is provided in /ptmp/mpp/flowerde/MPIsetup/firstrun.out.

Batch farm running

For this you will need four more files, all under /ptmp/mpp/flowerde/MPIsetup/: Note these will be slightly different to those sent around by Max, to make it easier to work at RZG.

  •, the script you call to submit the analysis.
  • batch_jobdcap2_v3.exe, the script that's executed on each worker node. It makes (and eventually removes) a temporary directory, runs the code and manages all necessary file transfers.
  • merge_job.exe does exactly what the filename says, merging outputs into a single, final file.
  • copy_files_dcap2.exe is used to actually copy the input files to the local batch node.
  • is a wrapper for the shell script with the same name, just to ensure it's executed in a different shell (this should be tidied up at some point).

These files need to be made executable:

chmod u+x *.exe

Necessary edits

As with the code, these scripts will need a little tweaking for each user.

File locations

The batch jobs will need access to the files descibed earlier here. Relative to where you submit the jobs from, you will need to create these directories:

  • GRL for the Good Runs List file(s) (*.xml)
  • rootfiles for all other files (electron maps etc)

If you are running at RZG, the edits below are optional. If you want to submit from MPI, some variables describing input and output locations will need to be set according to the needs of each user:

  • MyOutDir: Somewhere on your local system with space for output to be stored, for example under.
  • MyTempDir: Similar, but this is only used temporarily by the jobs, so must be local to RZG. The default is probably OK for most users.
  • MyBatchLogs: Directory to store the job log files, could be in /ptmp or your home area at RZG.
  • MyEmail: You will receive an email if a job fails or is aborted.
  • q: "short" is OK most of the time. If your jobs take longer than two hours, you will need to change this to "standard".

If you are submitting from MPI, you will need to create output_dir remotely; uncomment this line and edit it appropriately:

# os.system(" ssh mkdir -p "+output_dir)
Don't forget to remove the line after it!


Check that the variables ATHENAWORKDIR and INITSCRIPT are set correctly for you.

TMPDIR is set to be in /tmp by default (that is, on the worker node itself). This can apparently cause job crashes if other users' jobs do not remove their temporary files, so there is a suggested alternative under /ptmp that you could use instead.


Make sure that ATHSETUP does what you want it to do. Also, check the variable errorlocation, which is where logfiles will be put if the copy fails (should never happen if you're running at RZG). If you're submitting from MPI, you will need to uncomment all the lines referring to scp.

Job options

The input and output are given by the script executed on the batch node. To pick this up correctly, make sure your input/output lines look like this:

NtupleReaderAlg.fileList = "filelist.txt"
NtupleReaderAlg.numberOfEvents = -1
analysisTool.outputFileName = outputfilename


The syntax for the submission script is this:

./ <filelist> <joboptions file> <name for the job == output folder> [<files per job>] [<job number>]
The first two arguments are your file list and job options file, the third is the desired output location, which doubles as the job name. The last two arguments are optional: the first of these specifies how many files are processed in each job (default is 20), while the final argument tells the script to only submit the job with that particular number, obviously less than or equal to the length of the file list divided by the number of files per job. If it is the string "test", a local test job will be run instead.

An example command for running one local test (from SUSYAnalysis/run):

./ SUSYtest/VUA18_MC11c/filelist_mc11_7TeV.105986.ZZ_Herwig.merge.AOD.e825_s1310_s1300_r3043_r2993.txt firstZZtest 1 "test"

A test batch job:

./ SUSYtest/VUA18_MC11c/filelist_mc11_7TeV.105986.ZZ_Herwig.merge.AOD.e825_s1310_s1300_r3043_r2993.txt firstZZtest 10

This will create 5 jobs (plus a merge job) because there are 50 ZZ files.

The batch backend is Sun Grid Engine (SGE), meaning that you can use the commands qstat and qdel to monitor and remove jobs, respectively. Refer to the command man pages for more details.

Alternative: UCI Ntuples + common software

This is apparently very quick and easy to install, let's see if it's true! The main packages are:


Assuming you only want to read the ntuples, the following is sufficient (based on the README file). You should search for the latest tag, or else check out the trunk.

svn co svn+ssh:// SusyNtuple
# IMPORTANT: Edit, commenting or modifying the athena setup as appropriate
source SusyNtuple/scripts/
# At the end, this will complain about package Mt2 being missing. You can fix that like this:
svn co svn+ssh:// Mt2

If you want to produce the ntuples, you will need the SusyCommon package, which has a similar install script. Please add notes if you try it!

Before you build, you need to fix one missing dependency in TriggerMatch/cmt/Makefile.RootCore:


Regular use

Every time you start a new session, you need to do this, in addition to setting up ROOT:

cd RootCore
source scripts/   
cd ..

To compile, run this script noting that this is RootCore compilation, not athena:


You can also compile an individual package if you need to:

cd ../cmt
make -f Makefile.RootCore

-- MichaelFlowerdew - 01-Jun-2012

Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r12 - 2012-08-16 - unknown
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback