Running ATLAS software at Edinburgh

This page describes the steps needed to access and run ATLAS software from all local desktops in the Edinburgh group.

Software access via AFS

AFS (and the corresponding kinit and aklog credential tools) are now available as part of the standard desktop build. Once authenticated the main ATLAS software directory can be accessed by navigating to: /afs/cern.ch/atlas/software

Using DQ2, ROOT, Ganga, Panda and Athena Locally.

These packages can be used through ATLASLocalRootBase, this is separate to the particular releases installed below. To use these:

export ATLAS_LOCAL_ROOT_BASE=/Disk/speyside4/atlas/ddm/ATLASLocalRootBase

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase

source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh

and follow the instructions given on screen. Or at https://twiki.atlas-canada.ca/bin/view/AtlasCanada/ATLASLocalRootBase The former location on speyside is not up to date and the recommendation is to use the CVMFS setup.

For example: For DQ2 (to download data)

localSetupDQ2Client

voms-proxy-init --voms atlas

dq2-get [DATASET]

To use standalone root (to run on d3pds for example):

localSetupROOT

root

To run atlas software:

asetup -r 15.6.12

To use a nightly (you may need to check for the correct path

asetup --releasesarea=/Disk/speyside4/atlas/SW/nightlies/16.4.X rel_3 here

Currently for releases that aren't already around ask us - but instructions for adding them automatically will follow....

D3PDs can be stored in /Disk/speyside7/Grid/grid-files (not backed up)

Local installation

A copy of all the official releases since 15.4.0 are available on our dedicated group storage. For any local analysis and simulation work it is preferred to work with the local copy rather using AFS. This will be become more important once ATLAS jobs can be submitted to the Edinburgh condor batch system.

Current List of Installed Releases

All AtlasProduction kits from 15.4.0 onwards are available on local storage. The latest version available is 15.6.0.

These kits are currently available in the following directory structure:

/Disk/speyside4/atlas/SW/[arch]/[SL-version]/AtlasProduction/[Release-version]/

(e.g. /Disk/speyside4/atlas/SW/x86_64/SL5/AtlasProduction/15.4.0)

All new releases will be installed once the availability announcement has been made via the atlas-releaseKit-announce mail list. Please let me know if anyone needs a local copy of any atlas software for releases before 15.4.0.

Nightly builds

The following nightly development releases are available:

  • 15.X.0-SLHC

These are currently available in the following directory structure:

/Disk/speyside4/atlas/SW/[arch]/[SL-version]/AtlasProduction/nightlies/[Dev-release]/rel[0-6]

(e.g. /Disk/speyside4/atlas/SW/i686/SL4/AtlasProduction/nightlies/15.X.0-SLHC/rel0)

The nightly development software is rotated on a weekly basis through directories rel0 to rel6. A daily cron script (not finalised) will run the necessary pacman commands to retrieve the latest dev releases. A symlink is also available in the [Dev-release] directory to point to the latest release available (also updated daily)

Other nightly releases can be added on request. Please let me know if you need other development releases added to the local filesystem.

SL5 compatibility

SL5 RPMs

The following list of rpms allow SLC4/32bit binaries to be run on SLC5 machines.

  • compat-gcc-34
  • compat-gcc-34-c++
  • compat-gcc-34-g77
  • compat-glibc
  • compat-glibc-headers
  • compat-libf2c-34
  • compat-libgcc-296
  • compat-libstdc++-296
  • compat-libstdc++-33
  • compat-readline43
  • libgfortran-4.1.2
  • lapack-3.0-37.el5
  • ghostscript-8.15.2
  • libXpm-3.5.5
  • glibc-devel-2.5
  • giflib
  • compat-openldap
  • openssl097a-0.9.7a-9.el5_2.1
  • compat-db-4.2.52-5.1

These are all available as part of the standard build.

GCC issues

A different version of gcc is required for the ATLAS software than the one available in the local standard build (which is currently gcc-4.1). The required gcc versions are available in:

/Disk/speyside4/atlas/SW/[arch]/[SL-version]/[gccversion]

e.g. /Disk/speyside4/atlas/SW/i686/SL5/gcc43

To use this version of gcc you will need to source the setup file in the directory:

source /Disk/speyside4/atlas/SW/i686/SL5/gcc43/setup.sh

Submitting local batch jobs

This is currently work in progress. Once the software has been validated for local interactive jobs a request will be put in to central support to attach the ATLAS software area to all workstations running condor. At the present time over 300 nodes are available through this batch system.

Jobs requiring high memory

We are due to have a dedicated ATLAS server in place by the end of October. This machine has up to 48GB available for simulation studies.

Data output

There is a generous amount of local storage available (>10TB) for offline data on our present and new dedicated storage servers. Once the local batch system is attached any job output should be stored on these servers rather than in a home directory or the desktop scratch area. The output location is yet to be defined (it is likely to be atlas/data/*) so suggestions are welcome for how to best order the directory hierarchy.

Accessing files from ECDF_LOCALGROUPDISK locally

TFile::Open("root://srm.glite.ecdf.ed.ac.uk//dpm... ")

e.g.

TFile::Open("root://srm.glite.ecdf.ed.ac.uk//dpm/ecdf.ed.ac.uk/home/atlas/atlaslocalgroupdisk/mc10_7TeV/NTUP_HSG5WH/e701_s933_s946_r2302_r2300_p766/mc10_7TeV.116591.WH120lnubb_pythia.merge.NTUP_HSG5WH.e701_s933_s946_r2302_r2300_p766_tid563989_00/NTUP_HSG5WH.563989._000001.root.1")

This is slow for interactive use, but should be about as fast as having the files locally if making use of TTreeCache in your code. Using TTree::Draw() seems to be extremely fast.

Alternatively, it might be faster to copy the file locally. This can be done with rfcp:

export DPM_HOST=srm.glite.ecdf.ed.ac.uk

export DPNS_HOST=srm.glite.ecdf.ed.ac.uk

rfcp rfio:////dpm/ecdf.ed.ac.uk/home/atlas/atlaslocalgroupdisk/mc12_8TeV/NTUP_HSG2/e1622_s1581_s1586_r3658_r3549_p1344/mc12_8TeV.167892.PowhegPythia8_AU2CT10_ggH125_ZZ4lep_noTau.merge.NTUP_HSG2.e1622_s1581_s1586_r3658_r3549_p1344_tid01225391_00/NTUP_HSG2.01225391._000034.root.1 .

Listing datasets on ECDF

First, to generate a list of datasets, do e.g.:

dq2-list-dataset-site UKI-SCOTGRID-ECDF_LOCALGROUPDISK -n mc12*HSG2*

where the last parameter is a search string for your datasets. Then, given a dataset name, to generate a list of the paths to the files:

dq2-ls -f -p -L UKI-SCOTGRID-ECDF_LOCALGROUPDISK mc12_8TeV.167892.PowhegPythia8_AU2CT10_ggH125_ZZ4lep_noTau.merge.NTUP_HSG2.e1622_s1581_s1586_r3658_r3549_p1344/ | grep srm | sed 's/srm:\/\/srm.glite.ecdf.ed.ac.uk/rfio:\/\/\//g'

These can be accessed locally as described above.

Suggestions/Requests

This is an ongoing task so please let me know if everything (or anything!) is working.

Current task List

  • Interactive validation of local software
  • SL5 32-bit/64-bit issues
  • Documentation on local user setup, testing interactive job
  • Check for kit validation for SL5
  • Local cron script tests to retrieve nightlies
  • Determine best mode of file transfer (rsync or pacman package diff)
  • Associate nightly cron script to central service
  • Request software area be visible from condor
  • Test run of batch system
  • Add more development nightlies (on request)
  • Autoinstall method for new official releases
  • Possible data migration
  • Permanent mount point for /Disk/speyside4

Resources

-- AndrewWashbrook - 2009-09-30

Edit | Attach | Watch | Print version | History: r19 | r17 < r16 < r15 < r14 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r15 - 2013-05-15 - NickEdwards
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback