ScotGrid-Edinburgh dCache
Accessing the dCache
This section describes how to use
ROOT and DaVinci to access files on the Edinburgh Grid storage. It is a dCache, so it is necessary to use custom protocols to get access to the data; in this case we will use dcap (dCache access protocol). If you want to write data to the dCache (unlikely, but possible), you should use GSIdcap, which invokes the Grid security layer for authentication and authorisation.
DCAP
Environment variables
You should ensure that the following environment variables are set. This should happen automatically for you if you source the LHCb environment (typically happens automatically when you log in). If you are having problems, the first thing to check is that these variables are defined.
export DCACHE_CLIENT_ACTIVE=1
export DCACHE_CLIENT_LIB=/Disk/lochnagar0/lhcb/lhcb-soft/lcg/external/dcache_client/1.8.0/slc4_ia32_gcc34/dcap/lib
export LD_LIBRARY_PATH=$DCACHE_CLIENT_LIB:$LD_LIBRARY_PATH
Using dcap when running on ECDF
To get access to the data on dCache when running jobs on ECDF, it is necessary to apply the following workaround. This is required since the ECDF worker nodes have both public and private hostnames and naturally the worker node will use it's private hostname for communications. This means that prior to performing a TFile::Open() you need to tell dCache the public hostname so that it knows where to send the data. If you are trying to read data from the dCache, it is
essential that you have the following line in any script that you submit to ECDF. Note that you must use
exactly this line, otherwise the hostname will not be set correctly.
export DCACHE_CLIENT_LIB=/exports/work/physics_ifp_ppe/lhcb/lhcb-soft/lcg/external/dcache_client/1.8.0/slc4_ia32_gcc34/dcap/lib/
export LD_LIBRARY_PATH=$DCACHE_CLIENT_LIB:$LD_LIBRARY_PATH
export DCACHE_REPLY="eddie`/bin/hostname|/bin/cut -d\"e\" -f2`.ecdf.ed.ac.uk"
No extra environment variables are needed apart from those in your bashrc.
Using dcap with Ganga on ECDF
Ganga 5.X is automatically configured by using gu.configure() and the central ganga ini as given here with
Greigs Ganga Goodies.
Ganga 4.X is not centrally maintained so you need extra stuff. The environment variables should be set up in the SGE backend in Ganga 4:
[SGE]
kill_str = qdel %s
submit_str = cd %s;qsub -cwd -l h_vmem=2500M -l h_rt=4:00:00 -V %s %s %s %s
preexecute = os.chdir(os.environ["TMPDIR"])
os.environ["PATH"]+=":."
os.environ["LD_LIBRARY_PATH"]+=":/exports/work/physics_ifp_ppe/lhcb/lhcb-soft/lcg/external/dcache_client/1.8.0/slc4_ia32_gcc34/dcap/lib/"
os.environ["DCACHE_REPLY"]="".join(("eddie",os.uname()[1][4:],".ecdf.ed.ac.uk"))
os.environ["DCACHE_CLIENT_ACTIVE"]="1"
For more information see
LHCbEdinburghGroupDPM,
LHCbEdinburghGroupGanga,
LHCbEdinburghGroupECDF
To open the file:
TFile* f = TFile::Open("dcap://pool1.epcc.ed.ac.uk:22125/pnfs/epcc.ed.ac.uk/data/lhcb/production/DC06/phys-v3-lumi2/00001883/DST/0000/00001883_00002826_2.dst")
where the dcap://... stuff is the full path to the file on the dCache server. Of course, opening a DST file in
ROOT doesn't make too much sense, but it gives you an idea of what is possible.
DaVinci
If you want to use DaVinci, in addition to the above environment variables you need to specify that you want to use the dCache client in your DaVinci requirements file:
use dcache_client v* LCG_Interfaces
And remember to cmt config; source setup.sh; cmt gmake afterwards {j.application.cmt(br cmt config), j.application.make(), in
Ganga
}. It appears that this only works from
DaVinci v19r8 onwards.
DaVinci opts
DATASET='dcap://pool1.epcc.ed.ac.uk:22125/pnfs/epcc.ed.ac.uk/data/lhcb/production/DC06phys-v3-lumi2/00001883/DST/0000/00001883_00002826_2.dst'
GSIDCAP
XROOTD
GFAL
GFAL is the Grid File Access Library. It is used so hide users from the different storage system implementations, meaning that they can use the same syntax to access data on a variety of systems (think dCache, CASTOR and DPM).
It also means that you do not need to know anything about the backend storage configuration. For example, the above only works as you have been told to point DaVinci or
ROOT to pool1.epcc.ed.ac.uk.
DaVinci
In addition to the above, you need to have this in your requirements file:
use gfal v* LCG_Interfaces
It should then be possible to access files in this way:
DATASET='gfal:srm://srm.epcc.ed.ac.uk/pnfs/epcc.ed.ac.uk/data/lhcb/production/DC06phys-v3-lumi2/00001883/DST/0000/00001883_00002826_2.dst'
DC06 data has been stored on the dCache as is listed in
LHCbEdinburghGroupDataFiles . The simplest way to access it is through ganga using
Greigs Ganga Goodies
To check everything works
To check that you can see the dCache try:
dccp -d 15 dcap://pool1.epcc.ed.ac.uk:22125/pnfs/epcc.ed.ac.uk/data/lhcb/production/DC06/v1r0/00002000/DST/0000/00002000_00000004_2.dst /tmp/aFile
Or do this for any file listed in the dCache as in
LHCbEdinburghGroupDataFiles
The simplest way to check
everything is through ganga using
Greigs Ganga Goodies.
gu.testjob('dCache')
Monitoring plots (regarding activity on ed.ac.uk domain)
The dCache has been retired. Here is a plot of all the activity that it has seen over the past year:
--
GreigCowan - 18 Dec 2007