ScotGrid-Edinburgh dCache

Accessing the dCache

This section describes how to use ROOT and DaVinci to access files on the Edinburgh Grid storage. It is a dCache, so it is necessary to use custom protocols to get access to the data; in this case we will use dcap (dCache access protocol). If you want to write data to the dCache (unlikely, but possible), you should use GSIdcap, which invokes the Grid security layer for authentication and authorisation.

DCAP

Environment variables

You should ensure that the following environment variables are set. This should happen automatically for you if you source the LHCb environment (typically happens automatically when you log in). If you are having problems, the first thing to check is that these variables are defined.

export DCACHE_CLIENT_ACTIVE=1
export DCACHE_CLIENT_LIB=/Disk/lochnagar0/lhcb/lhcb-soft/lcg/external/dcache_client/1.8.0/slc4_ia32_gcc34/dcap/lib
export LD_LIBRARY_PATH=$DCACHE_CLIENT_LIB:$LD_LIBRARY_PATH

Using dcap when running on ECDF

To get access to the data on dCache when running jobs on ECDF, it is necessary to apply the following workaround. This is required since the ECDF worker nodes have both public and private hostnames and naturally the worker node will use it's private hostname for communications. This means that prior to performing a TFile::Open() you need to tell dCache the public hostname so that it knows where to send the data. If you are trying to read data from the dCache, it is essential that you have the following line in any script that you submit to ECDF. Note that you must use exactly this line, otherwise the hostname will not be set correctly.

export DCACHE_CLIENT_LIB=/exports/work/physics_ifp_ppe/lhcb/lhcb-soft/lcg/external/dcache_client/1.8.0/slc4_ia32_gcc34/dcap/lib/
export LD_LIBRARY_PATH=$DCACHE_CLIENT_LIB:$LD_LIBRARY_PATH
export DCACHE_REPLY="eddie`/bin/hostname|/bin/cut -d\"e\" -f2`.ecdf.ed.ac.uk"

Using dcap with Ganga on Condor

No extra environment variables are needed apart from those in your bashrc.

Using dcap with Ganga on ECDF

Ganga 5.X is automatically configured by using gu.configure() and the central ganga ini as given here with Greigs Ganga Goodies.

Ganga 4.X is not centrally maintained so you need extra stuff. The environment variables should be set up in the SGE backend in Ganga 4:

[SGE]
kill_str = qdel %s
submit_str = cd %s;qsub -cwd -l h_vmem=2500M -l h_rt=4:00:00 -V %s %s %s %s
preexecute = os.chdir(os.environ["TMPDIR"])
        os.environ["PATH"]+=":."
        os.environ["LD_LIBRARY_PATH"]+=":/exports/work/physics_ifp_ppe/lhcb/lhcb-soft/lcg/external/dcache_client/1.8.0/slc4_ia32_gcc34/dcap/lib/"
        os.environ["DCACHE_REPLY"]="".join(("eddie",os.uname()[1][4:],".ecdf.ed.ac.uk"))
        os.environ["DCACHE_CLIENT_ACTIVE"]="1"

For more information see LHCbEdinburghGroupDPM, LHCbEdinburghGroupGanga, LHCbEdinburghGroupECDF

ROOT

To open the file:

TFile* f = TFile::Open("dcap://pool1.epcc.ed.ac.uk:22125/pnfs/epcc.ed.ac.uk/data/lhcb/production/DC06/phys-v3-lumi2/00001883/DST/0000/00001883_00002826_2.dst")

where the dcap://... stuff is the full path to the file on the dCache server. Of course, opening a DST file in ROOT doesn't make too much sense, but it gives you an idea of what is possible.

DaVinci

If you want to use DaVinci, in addition to the above environment variables you need to specify that you want to use the dCache client in your DaVinci requirements file:

use dcache_client v* LCG_Interfaces

And remember to cmt config; source setup.sh; cmt gmake afterwards {j.application.cmt(br cmt config), j.application.make(), in Ganga}. It appears that this only works from DaVinci v19r8 onwards.

DaVinci opts

DATASET='dcap://pool1.epcc.ed.ac.uk:22125/pnfs/epcc.ed.ac.uk/data/lhcb/production/DC06phys-v3-lumi2/00001883/DST/0000/00001883_00002826_2.dst'

GSIDCAP

XROOTD

GFAL

GFAL is the Grid File Access Library. It is used so hide users from the different storage system implementations, meaning that they can use the same syntax to access data on a variety of systems (think dCache, CASTOR and DPM). It also means that you do not need to know anything about the backend storage configuration. For example, the above only works as you have been told to point DaVinci or ROOT to pool1.epcc.ed.ac.uk.

DaVinci

In addition to the above, you need to have this in your requirements file:

use gfal v* LCG_Interfaces

It should then be possible to access files in this way:

DATASET='gfal:srm://srm.epcc.ed.ac.uk/pnfs/epcc.ed.ac.uk/data/lhcb/production/DC06phys-v3-lumi2/00001883/DST/0000/00001883_00002826_2.dst'

LHCbEdinburghGroupDataFiles

DC06 data has been stored on the dCache as is listed in LHCbEdinburghGroupDataFiles . The simplest way to access it is through ganga using Greigs Ganga Goodies

To check everything works

To check that you can see the dCache try:

dccp -d 15 dcap://pool1.epcc.ed.ac.uk:22125/pnfs/epcc.ed.ac.uk/data/lhcb/production/DC06/v1r0/00002000/DST/0000/00002000_00000004_2.dst /tmp/aFile

Or do this for any file listed in the dCache as in LHCbEdinburghGroupDataFiles

The simplest way to check everything is through ganga using Greigs Ganga Goodies.

gu.testjob('dCache')

Monitoring plots (regarding activity on ed.ac.uk domain)

The dCache has been retired. Here is a plot of all the activity that it has seen over the past year:

-- GreigCowan - 18 Dec 2007