Running Delphes 3.0.8 on ATLAS Distributed Computing Resources


ATLAS computing resources in the U.S. are available to run Delphes as a fast-simulation package for study of LHC and VLHC detectors. Instructions provided by Sanjay Padhi for the Snowmass Energy Frontier page detail how to get started... and this present documentation will describe how to run simulations on the ATLAS Grid.

Setting Up

You can use either lxplus, BNL acas or local resources to submit your jobs on the grid, as long as you have a correct version of Asoka da Silva's excellent ATLASLocalRootBase system installed. If this is not yet the case, please use a resource that is either set up with fully updated PanDA client tools (to provide prun) or install ATLASLocalRootBase, something that will not take an excess of time or skill.

On a well-configured system, you will use the standard ATLASLocalRootBase method of getting the ATLAS environment set up:


You are now ready to check your credentials. If you have not already done so, please copy your Grid certificate into your $HOME/.globus folder, and change the permissions of your certificate to -rw-r--r-- via:

chmod 644 ~/.globus/*.p12 ~/.globus/*.pem



to get Grid tools, then

voms-proxy-init -voms atlas

to see if your Grid environment will allow submission. If not, please see the ATLAS Workbook for more suggestions on how to get your setup running correctly.

If this is all functional, you are ready for the next stage -- getting files checked in to the Grid and ready to run.

Checking In Data Files

You will need to add your inputs to the ATLAS Distributed Data Management (DDM) system. There will be three classes of input file: Pileup, Cards and MC input.

The card files are small, and will be included in the prun jobs themselves. The others are weighty and harder to manage, so we'll put them in DDM.

First up are files that you have created yourself. Make sure they are in a filesystem accessible to your Grid login, and get them loaded into a dataset.

There are many rules, tricks and best practices for naming the datasets, but the basic practicalities are:

  • Start the dataset name with user.myusername.
  • Name the dataset something useful. For example, user.myusername.myGenerator_physicsChannel.20130522.hepmc. I added the date string because since dataset names cannot be reused, and it helps me keep track of what was uploaded when.
  • Be consistent, since it will help simplify your submissions. This applies to both datasets and input files.

Leave the files in uncompressed format, since some sites will be using direct access to the files, making a decompression step more difficult.

Choose a space token from the list found in the AGIS database. Something with _USERDISK, and close to your home institute would be a good choice. You can always replicate the data elsewhere later with DaTRI as well.

Start up the DDM client as before. With a dataset name chosen and the files in a single directory, run the following. Remember that it will take some time to complete, especially for abundant data. Items like "=myusername=" and "=SITE_USERDISK=" are placeholders for your dataset, site and user names.

dq2-register-dataset user.myusername.datasetName.20130522.hepmc dq2-register-location user.myusername.datasetName.20130522.hepmc SITE_USERDISK dq2-put -d -L SITE_USERDISK -f datasetDirectory user.myusername.datasetName.20130522.hepmc

Once the datasets have been uploaded and checked, you can look at a list of them using

dq2-ls -f user.myusername.datasetName.20130522.hepmc

The pre-generated pileup files may already be in the system. If not, they are easy to add. Download and uncompress the files listed on the Snowmass Energy Frontier page (MinBias_13TeV.tar.gz, for example) as follows:

curl -O
curl -O
curl -O

tar xzf MinBias13TeV.tar.gz
tar xzf MinBias33TeV.tar.gz

mkdir MinBias13TeV
mkdir MinBias33TeV
mkdir MinBias100TeV

mv MinBias13TeV.pileup MinBias13TeV
mv MinBias33TeV.pileup MinBias33TeV
mv MinBias100TeV.pileup MinBias100TeV

Use the same process as above to put the MinBias files into datasets. Name them along these lines:


There will also be replicas of the minbias files under user.stradlin.* in dq2. You may use them in your jobs.

Getting Ready To Run

Open a new shell, run setupATLAS, and then setup the prun client as follows:

localSetupPandaClient --noAthenaCheck

With the PanDA client set up, create a run directory.

cd ~; mkdir -p Snowmass/run cd Snowmass/run

Create the file as follows:

cat << END >
#! /usr/bin/env bash
export VERSION=3.0.8
export PATH=$PATH:$ROOTSYS/bin:/bin:/usr/bin:/usr/local/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROOTSYS/lib:/lib:/lib64:/usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64
/usr/bin/curl -O$VERSION.tar.gz
/bin/gunzip Delphes-$VERSION.tar.gz
/bin/tar xf Delphes-$VERSION.tar
cd Delphes-$VERSION

Also create the file

cat << END >
#! /usr/bin/env bash
export VERSION=3.0.8
export PATH=$PATH:$ROOTSYS/bin:/bin:/usr/bin:/usr/local/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROOTSYS/lib:/lib:/lib64:/usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64

Now download the card files to the run directory (as tcl files):

curl -O
curl -O
curl -O
curl -O
rename tcl?revision=1.2 tcl *
rename delphes_card_Snowmass_ "" *

These are small, and will come as the payload for all your runs.

Running Delphes using PanDA

You will use the following environment variables to make your command line submission more flexible and easier to debug.

CARD=50PileUp.tcl #(For example)
MINBIAS=user.myusername.MinBias13TeV.pileup #(For example)
OUTFILE=user.myusername.myGenerator_physicsChannel.MinBias13TeV.50PileUp #(For example)

Now you can submit your jobs to the grid with prun command. If the system asks you for a password to your certificate, type it in.

prun --exec " $CARD $OUTFILE.root %IN %PILEUP" --inDS $MYDATA --secondaryDSs PILEUP:1:$MINBIAS --outputs $OUTFILE.root --outDS $OUTDS --athenaTag= --rootVer=5.32.00 --bexec

If you want to do multiple runs, you can continue by changing the variables and submitting again using the following command (to avoid having to rebuild Delphes):

prun --exec " $CARD $OUTFILE.root %IN %PILEUP" --inDS $MYDATA --secondaryDSs PILEUP:1:$MINBIAS --outputs $OUTFILE.root --outDS $OUTDS --athenaTag= --rootVer=5.32.00 --libDS LAST

If the jobs begin to fail because the local directory size gets too large, break up the input files into smaller chunks and try again.

Jobs have run successfully on ANALY_SWT2_CPB. Other sites have had individual issues, but can be debugged. Please feel free to contact Alden about it.

A successful job example can be found here.

All jobs may be monitored from PanDA user job monitoring pages.

-- AldenStradling - 22-May-2013

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2013-05-24 - KaushikDe
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback