This page is intended to share solutions and tips to common Athena problems when running on RZG. Please add new solutions when they come up!

Using the Tier2 software for local analysis. Athena Release example.

It is possible to use Tier2 software and to compile your favorite packages in release 16 without any cmthome/requirement and without any modification in PATH's etc... Just do following:
  • source /t2/sw/profile.d/
  • source /t2/sw/vo/ag/
  • mkdir ~/testarea/AtlasOffline-16.0.0
  •  source /t2/sw/vo/ag/software/16.0.0/ AtlasProduction, --testarea=~/testarea/AtlasOffline-16.0.0 
Known issues: last command, as well as any make command in any package will give plenty of warning like
#CMT---> Error: syntax error : ParseRequirements> 
#CMT---> Error: bad syntax in requirements of GaudiPolicy v10r7  line #1240 
#CMT---> Error: bad syntax in requirements of AtlasPolicy AtlasPolicy-01-06-92  
However, everything seems working.

Using the Tier2 software for local analysis

Sometimes it may be convenient to make use of the Tier2 automatic software installation.

Release sourcing:

The code can be source by:
source /t2/sw/vo/ag/software/15.6.10/ -tag=AtlasProduction,,runtime,setup

Package development and user code on top of the ATLAS installation

If you need to compile your favorite package on top of a Tier2 kit installation.
  • Create you favorite cmthome-t2 directory
  • Pick up the requirements file from that kit (for example: /t2/sw/vo/ag/prod/releases/rel_15-22/cmtsite/requirements)
  • modify it according to your test area, additional prepend... for example:
        macro ATLAS_TEST_AREA ${HOME}/TopPhysTest/athena
        path_prepend PYTHONPATH "/afs/ipp/home/g/gcortian/extra_python_lib_15.6.3"
    or look into /afs/ipp/home/g/gcortian/public/tier2sw/requirements for an example.

  • source /t2/sw/vo/ag/prod/releases/rel_15-22/CMT/v1r21/mgr/
  • cmt config
  • then simply proceed the usual way from your ATLAS_TEST_AREA (cmt co mypackage... gmake ... run)

Athena Release solving compiler problems

In order to be able to select the right environment for compiling in this release one needs to export the following variables after sourcing the release set up:



Running ARA/C++ jobs on the grid

The standard setup for submitting ARA applications to the the grid, using ganga, j.application.atlas_exetype='PYARA' does not work for ARA/C++ packages. To be able to have the proper setup on the worker node use the following, in your ganga submission script:

j.application.option_file=['' ]

Athena will be ran instead of python in this case. The option file,, should look like the following:

# Job options file

import os

theApp.EvtMax = 1

# call here the ARA job options:


As in the case of PYARA, Ganga will provide a list ('input.txt') of the input files on the worker node that can be loaded into the ARA job option.

Athena Releases 15.6.3: solving inputFilePeeker problems

if after setup /afs/ -tag=15.6.3,opt,runtime by issuing file.AOD you get error messages containing the following:
ImportError: No module named _md5

change your setup by putting some python libs into a directory that you append to the PYTHONPATH. And add into your cmthome/requirements file before the use AtlasLogin line the statement:

path_prepend PYTHONPATH "/afs/ipp/home/g/gcortian/public/extra_python_lib_15.6.3"

Setting up Top-Phys in Garching RZG:

At the moment (Mon Feb, 15, 2010), the AtlasLogin package installed at RZG does not allow to setup TopPhys-

One possible work-around is to setup AtlasProduction- and checkout on top of it the packages related to the phys cache and compiling them:

From tag collector the list of the packages neeed are as follows:

cmt co -r CoolConvUtilities-00-03-26  Database/CoolConvUtilities 
cmt co -r PhysicsElementLinks-00-00-05 PhysicsAnalysis/PhysicsElementLinks 
cmt co -r SingleTopDPDDumper-00-00-19  PhysicsAnalysis/TopPhys/SingleTopDPDDumper 
cmt co -r TopPhysD2PDMaker-00-02-07    PhysicsAnalysis/TopPhys/TopPhysD2PDMaker 
cmt co -r TopPhysRunTime-00-00-01     Projects/TopPhysRunTime 
cmt co -r HighPtTopJetFinder-00-00-10  Reconstruction/Top/HighPtTopJetFinder 
cmt co -r TopChi2Fitter-00-01-16       Reconstruction/Top/TopChi2Fitter 
cmt co -r TopCommissioning-00-00-11    Reconstruction/Top/TopCommissioning 
cmt co -r TopEDM_Test-00-00-18         Reconstruction/Top/TopEDM_Test 
cmt co -r TopEvent-00-00-33            Reconstruction/Top/TopEvent 
cmt co -r TopEventAthenaPool-00-00-14  Reconstruction/Top/TopEventAthenaPool 
cmt co -r TopEventTPCnv-00-00-18       Reconstruction/Top/TopEventTPCnv 
cmt co -r TopInputs-00-01-03           Reconstruction/Top/TopInputs 
cmt co -r TopInputsAlgs-00-00-22       Reconstruction/Top/TopInputsAlgs 
cmt co -r TopInputsAthenaPool-00-01-02 Reconstruction/Top/TopInputsAthenaPool 
cmt co -r TopInputsSelection-00-01-17  Reconstruction/Top/TopInputsSelection 
cmt co -r TopInputsTPCnv-00-01-02      Reconstruction/Top/TopInputsTPCnv 
cmt co -r TopTools-00-00-15            Reconstruction/Top/TopTools 
cmt co -r TopUtils-00-00-24            Reconstruction/Top/TopUtils          

To compile them using cmt br gmake.

ARA_Example_Top (14.5.0)

If you get the following message while performing ARA_Example_Top setup
#CMT> Warning: package egammaEvent egammaEvent-* Reconstruction not 
> found (requested by ARA_Examples_Top)
Substitute the following line in the requirements

use egammaEvent         egammaEvent-*       Reconstruction

with this one:

use egammaEvent         egammaEvent-*       Reconstruction/egamma

Local chi2 alignment (14.5.0)

cmt co -r InDetRecExample-01-16-05 InnerDetector/InDetExample/InDetRecExample
cmt co InnerDetector/InDetAlignAlgs/InDetLocalChi2AlignAlgs (00-00-47)
cmt co InnerDetector/InDetAlignTools/InDetLocalChi2AlignTools (00-00-53)
cmt co InnerDetector/InDetAlignEvent/InDetLocalChi2AlignEvent (00-00-11)

#compile this in reverse order

#libg2c missing:
#create links to in your InstallArea:

ln -s /afs/ipp/@sys/soft/gcc/gcc-3.4.6/lib/   ~/athena/AtlasOffline-14.5.0/InstallArea/i686-slc4-gcc34-opt/lib/.
ln -s /afs/ipp/@sys/soft/gcc/gcc-3.4.6/lib/ ~/athena/AtlasOffline-14.5.0/InstallArea/i686-slc4-gcc34-opt/lib/.

cd InnerDetector/InDetAlignEvent/InDetLocalChi2AlignEvent/cmt/ ; source ; gmake ; cd -

cd InnerDetector/InDetAlignTools/InDetLocalChi2AlignTools/cmt/ ; source ; gmake ; cd -

cd InnerDetector/InDetAlignAlgs/InDetLocalChi2AlignAlgs/cmt/ ; source ; gmake ; cd -

Database access via Replicas

Please see this link to setup Replica DB access.

Athena on SLES10

Update: AtlasSite is configured such that SLES10 nodes automatically setup gcc 3.4.6 (as the SLES9 nodes) for Athena. Everything should compile smoothly.

Question: Is the Fortran setup (see below) affected by this?

14.2.20/14.2.21 no AOD from release 13 can be used within ARA

Using ARA versions 14.2.20/14.2.21 it is not possible to read AOD produced with version 13 See related hyperNews: here and following messages

Athena 14.1.0 (GX2 alignment)

These are instruction to run the ID Global chisquare Alignemnt

Release setup

As always create cmt14.1.0 and athena/AtlasOffline-14.1.0

source /afs/ipp/mpp/atlas/software/dist/14.1.0/CMT/v1r20p20080222/mgr/

Then check out the following packages:

    cmt co -r SiGlobalChi2AlgebraUtils-00-00-18 InnerDetector/InDetAlignTools/SiGlobalChi2AlgebraUtils
    cmt co -r InDetAlignGenAlgs-00-02-15 InnerDetector/InDetAlignAlgs/InDetAlignGenAlgs
    cmt co -r SiGlobalChi2AlignTools-00-00-29 InnerDetector/InDetAlignTools/SiGlobalChi2AlignTools
    cmt co -r SiGlobalChi2Algs-00-01-43 InnerDetector/InDetAlignAlgs/SiGlobalChi2Algs
    cmt co -r InDetAlignGlobalChi2Monitoring-00-00-13 InnerDetector/InDetMonitoring/InDetAlignGlobalChi2Monitoring
    cmt co -r InDetRecExample-01-08-51 InnerDetector/InDetExample/InDetRecExample
    cmt co -r InDetAlignExample-00-00-39 InnerDetector/InDetExample/InDetAlignExample
    cmt co -r InDetCosmicRecExample-00-03-65 InnerDetector/InDetExample/InDetCosmicRecExample

then for each package: cmt config && source && gmake

Pathes to run at RZG

There are few changes needed to be able to run at RGZ.

Database access

In InDetAlignExample/share/ you need to add the following lines in the files in order to access the DB replica at GridKa
   # adding database replica
   print "Setting up DB replica"
   from DBReplicaSvc.DBReplicaSvcConf import DBReplicaSvc
   DBReplicaSvc = DBReplicaSvc(UseCOOLSQLite=False)
   svcMgr += DBReplicaSvc
   print "DB replica setup done"

Data Access

Copies of the FDR2 data samples are provided in


However, since some files are very big (3.0G), dcache have problems in accessing them, and in any case could be very slow. A possible solution, is to modify the python files which specify the input, making them copy in the /ltmp/ the files before accessing them. In the case of the frd2_idstream a possible implementation is as follows:

INPUTDIR = "/pnfs/"

FileName+=[ "" ] 
FileName+=[ "" ] 
FileName+=[ "" ] 
# the list continues....
# end of the file list

intFile = InDetAlignExampleFlags.FirstNumberFile
FilesToRead = InDetAlignExampleFlags.FilesToRead

myTMP = "/ltmp/mpp/cortiana/"
print 'checking dir '+myTMP
if not os.path.isdir(myTMP):
    print 'creating dir '+myTMP
    print 'dir '+myTMP+' already exists! using it!'
print 'Files to process:'
while intFile < FilesToRead+InDetAlignExampleFlags.FirstNumberFile:
    if intFile < len(FileName):
        os.system("dccp "+INPUTDIR+FileName[intFile]+" "+myTMP)       
        print myTMP+FileName[intFile]
        ServiceMgr.ByteStreamInputSvc.FullFileName += [myTMP+FileName[intFile]]
    intFile += 1

However, if you what to use dcache (works only for small input files), you need to:

export LD_LIBRARY_PATH=/opt/LCG/d-cache/dcap/lib:/opt/d-cache/dcap/lib/:$LD_LIBRARY_PATH

Queue system

RGZ uses Grid Engine - a facility for executing UNIX jobs on remote machines, as a queue management Jobs can be run in batch mode using qsub. For example, adding the following line to

os.system("qsub -m ea -q %s -o %s -e %s %s" % (self.QUEUE, OutputLogFile, OutputLogFile, self.SCRIPTNAME))
allows to lunch your script (SCRIPTNAME) in the queue (QUEUE). The log files OutputLogFile will contain both errors and stdoutput reports. You will also receive an email after job completition or in case of errors.

Fortran Libraries

If you get some compilation errors regarding fortran libraries, please consult: 14.2.0 problems

Athena 14.2.0


standard requirements file

Q: can you give me a standard req file to start with?

A: take e.g. this one (updated 14.2.10 version):

checkout from our groups area

Q: How to check out a package in our groups area?

A: don't use cmt (as for the Athena packages), but cvs:

cd ~/athena/AtlasOffline-14.2.0
cvs co -d directory_to_put_in groups/MPP/Package_Name
cvs co -d ARA_Examples_Top groups/MPP/ARA_Examples_Top
After that, the usual cvs update and cvs commit work as normal

compilation options

Q: Are there some handy options for gmake?

A: There's e.g.

gmake QUICK=1
usable after the first compilation, skips setting all links in InstallArea and is much faster
gmake clean
"uninstalls" the package from InstallArea (slow)
gmake binclean
only removes binaries for a fresh recompilation (fast)
gmake install_includes
this target only installs the header files in the InstallArea, so that other packages can compile against this one without a full compilation
cmt broadcast
prepend this to every command that you want to execute to the package and all it's dependencies, e.g. cmt br cmt config, cmt br gmake, etc.

Problems missing

Q: I get a compile error like
g++: /usr/lib/ No such file or directory
A: Patch the AtlasFortranPolicy and create links in your InstallArea:
cmt co AtlasFortranPolicy
In AtlasFortranPolicy/cmt/requirements patch line 145: From
144      HAS_GFO    " $(GFO)/$(GFO_LIBS)/libgfortranbegin.a $(GFO)/$(GFO_LIBS)/libgfortran.a" \
145      HAS_GFO&slc4    " $(GFO)/$(GFO_LIBS)/" \
146      HAS_G95    " $(G95)/lib/gcc-lib/i686-pc-linux-gnu/4.0.1/libf95.a"
144      HAS_GFO    " $(GFO)/$(GFO_LIBS)/libgfortranbegin.a $(GFO)/$(GFO_LIBS)/libgfortran.a" \
145      HAS_GFO&slc4    " -lgfortran" \
146      HAS_G95    " $(G95)/lib/gcc-lib/i686-pc-linux-gnu/4.0.1/libf95.a"
then create links in InstallArea:
ln -s /afs/ ~/athena/AtlasOffline-14.2.0/InstallArea/i686-slc4-gcc34-opt/lib/ 
ln -s /afs/ ~/athena/AtlasOffline-14.2.0/InstallArea/i686-slc4-gcc34-opt/lib/
ln -s /afs/ ~/athena/AtlasOffline-14.2.0/InstallArea/i686-slc4-gcc34-opt/lib/
and recompile (? needed ?)

A2: This topic was quite old and since we are already in release 17, I think that this answer doesn't work anyway.

I am writing here because it is the first link in google search.

The problem should be solved if the setup of athena is done properly as dq2 and asetup write in same env. variables.

1st: source ~/scripts/ that has

source /afs/
export LFC_HOST=`lcg-infosites --vo atlas lfc`
source /afs/

2nd: source ~/scripts/ that has

export AtlasSetup=/afs/
alias asetup='source $AtlasSetup/scripts/'
echo " "
echo "Now cd to WorkArea/1X.Y.ZZ and do something like:"
echo "$ asetup 1X.Y.ZZ,here"
echo " "

3th: get packages (svn co ....)

4rd: setup athena:

cd 17.X.Y/
asetup 17.X.Y,here
cd PhysicsAnalysis/StandardModelPhys/xxx/xxx/cmt/

5th: config setup make:

cmt config
source setup.[c]sh
cmt make

6th (for pathena): source ~/scripts/ that has

source /afs/

I hope it works now!

libg2c missing

Q: I get a compile error like:
/usr/lib64/gcc/x86_64-suse-linux/4.1.2/../../../../x86_64-suse-linux/bin/ld: cannot find -lg2c 
A: create links to in your InstallArea:
ln -s /afs/ipp/@sys/soft/gcc/gcc-3.4.6/lib/   ~/athena/AtlasOffline-14.2.0/InstallArea/i686-slc4-gcc34-opt/lib/.
ln -s /afs/ipp/@sys/soft/gcc/gcc-3.4.6/lib/ ~/athena/AtlasOffline-14.2.0/InstallArea/i686-slc4-gcc34-opt/lib/.

ERROR: /lib/ version `GCC_3.4' not found

Q: I get the following error:
ERROR: /lib/ version `GCC_3.4' not found (required by /afs/ 
A: create the following link:
ln -s /afs/ipp/@sys/soft/gcc/gcc-3.4.6/lib/  ~/ali/athena/AtlasOffline-14.1.0/InstallArea/i686-slc4-gcc34-opt/lib/ 

Athena 14.2.10

Fortran libs setup

Here the setup for the Fortran libraries is a bit more tricky: Check out & modify AtlasFortranPolicy as described for 14.2.0, then create the following links in your InstallArea:
cd ~/athena/AtlasOffline-14.2.10/InstallArea/i686-slc4-gcc34-opt/lib/
mkdir amd64_sles10
mkdir amd64_sles9
mkdir i386_sles9
ln -s /afs/ipp/amd64_sles10/soft/gcc/gcc-4.2.4/lib/ amd64_sles10/
ln -s /afs/ipp/amd64_sles9/soft/gcc/gcc-4.2.3/lib/ amd64_sles9/
ln -s /afs/ipp/i386_sles9/soft/gcc/gcc-4.2.3/lib/ i386_sles9/
ln -s @sys/
ln -s
ln -s
ln -s
ln -s /afs/ipp/@sys/soft/gcc/gcc-3.4.6/lib/
ln -s /afs/ipp/@sys/soft/gcc/gcc-3.4.6/lib/

Ganga @ RZG

First setup your Athena version, e.g. source cmt14.2.10/

On a SLES10 node (e.g. dt01), source /afs/, on SLES9 nodes, it should be OK without...

Run /afs/ in the run dir of the package you want to work with

for a Ganga tutorial, have a look at Atlas.GangaTutorial44

if your jobs terminate with NameError: name 'EventSelector' is not defined, prepend the line svcMgr.EventSelector.InputCollections ... with the following line: EventSelector = svcMgr.EventSelector

Local submission works, grid submission not (SLES9 has too old libcurl, SLES10 has no edg-job-submit)

Update: if you setup the correct grid environment source /t2/sw/glite-3.1.17-UI/external/etc/profile.d/grid-env, grid submission should be possible on at01 and dt01

running @ RZG

In Ganga j.backend.requirements.sites is the space token name of the input dataset:



For Tier1s there are shortcuts such as FZK or TRIUMF

qsub @ RZG

TIP: if all Your batch jobs are failing (and Your account has recently been prolonged) try:


If You want to run batch jobs needing a grid proxy, which will also work when submitting batch jobs, then do:

[at01]:~> source /afs/

[at01]:~> voms-proxy-init -voms atlas -out $HOME/.globus/gridproxy.cert

[at01]:~> export X509_USER_PROXY=${HOME}/.globus/gridproxy.cert

dq2 @ RZG

dq2-get at RZG needs the seting of:


dCache @ RZG

old (User) dcap://

new (Grid) dcap://

ROOT Access to dCache files at RZG and MPPMU

at RZG you need to setup athena and dCache.

[dt01:]~> source ~/cmthome15.6.10/ -tag=AtlasProduction,15.6.10,opt,setup

[dt01:~> source /t2/sw/profile.d/

[dt01:~> cd root

[dt01:~/root> root

[root]: TFile * tf = TFile::Open("dcap://")

[root]: tf->ls()

The important part is probably the static TFile::Open instead of a TFile ctor ...

The same should work on your laptop (or desktop) at MPPMU provided you have an athena release installed ...

Access to dCache @ RZG from anywhere

you need libdcap-1.9.3-6 or higher - for example from here.

Install the libraries and make sure you export LD_LIBRARY_PATH with the location of these libs pre-pended (i.e. after the setup of athena).

Then from another window execute:

/afs/ [LoginAtRZG]

for access to the grid-based dCache or

/afs/ [LoginAtRZG]

for the access to the old user dCache.

LoginAtRZG is your user name at RZG. If not given it defaults to the local user name. Both commands accept several more arguments like:

PortRange (default is "50010:50011")

MySSHPort (default is 8022)

MyDCAPPort (default is 22125)

where PortRange is the port numbers occupied by your dCache tunnel - so it's usually good to specify your own range in order not to interfere with others MySSHPort is the port used for the master ssh tunnel and MyDCAPPort is the local port for your dCache control line. Note that all arguments before the one you'd like to alter have to be specified ...

The scripts will print 3 environment variables you need to export. Just copy the lines and paste them into the window where you plan to use the tunnel. In both cases a little dCache status icon will appear on your screen and you can then start accessing the files using the syntax:


for the old user dCache and


for the new grid-based dCache. To close all ssh tunnels close the icon with the dCache bird.

misc grid @ RZG

informations about CEs such as ATHENA versions names etc:

[at01]:~>lcg-infosites --vo atlas [tag,sitenames]

-- HansVonDerSchmitt - 24 Jun 2008 -- TobiasGoettfert - 24 Jun 2008 -- GiorgioCortiana - 27 Jun 2008-- GiorgioCortiana - 24 Jul 2008 -- EmanuelRauter - 19 Jan 2009 -- SvenMenke - 23-Feb-2010 -- SvenMenke - 15-Nov-2010

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatext requirements r2 r1 manage 0.6 K 2008-09-02 - 18:34 TobiasGoettfert Requirements file for 14.2.10
Edit | Attach | Watch | Print version | History: r29 < r28 < r27 < r26 < r25 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r29 - 2011-11-16 - MarioSousa
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback