TWiki> ArdaGrid Web>DataTransferForTheoryQCD (revision 7)EditAttachPDF

Data Transfer Tools for Theory QCD Application

Grid Tools

We investigate xrootd and FTS for this purpose.

xrootd

lxplus.cern.ch : the tools are already installed in: /usr/bin

SLC4,SLC5: yam install xrootd-client

SL4,SL5: find and install rpm in linuxsoft.cern.ch extras repository

Alternatively see below to install "from scratch".

Install and setup xrootd client tools "from scratch" (latest development version)

Minimal steps (should be fine for SLC4,SLC5,RHEL):

Get the installer from the xrootd homepage

I installed the latest CVS development version which should in few weeks become the production version.

=bash xrd-installer --install

By default the client tools get installed in ~/xrdserver. You need this:

export LD_LIBRARY_PATH=~/xrdserver/lib:$LD_LIBRARY_PATH
export PATH=~/xrdserver/bin:$PATH

You are ready to go.

Extra steps in case of problems (what I did to get it running on ubuntu 9.10)

If installer fails to compile xrootd packages then check if you have all needed packages on the system (including the dev versions):

To authenticate via Kerberos 5 make sure that you have krb5 package installed and configured to include CERN.CH:

If needed, add CERN.CH to kerberos realms in [realms] section in the configuration file /etc/krb5.conf:

[realms]
     CERN.CH = {
         default_domain = cern.ch
         kpasswd_server = afskrb5m.cern.ch
         admin_server = afskrb5m.cern.ch
         kdc = afsdb2.cern.ch
         kdc = afsdb3.cern.ch
         kdc = afsdb1.cern.ch

         v4_name_convert = {
            host = {
              rcmd = host
            }
         }
     }

I also made it default:

[libdefaults]
        default_realm = CERN.CH

Transfer your files

kinit user@CERN.CH

You are ready to play with Castor transfer:

  • upload: xrdcp /etc/hosts root://castorpublic.cern.ch//castor/cern.ch/user/m/moscicki/tmp.test
  • download and dump on screen: xrdcp root://castorpublic.cern.ch//castor/cern.ch/user/m/moscicki/tmp.test -
  • download in verbose mode and pipe to /dev/null: xrdcp -d 3 -f root://castorpublic//castor/cern.ch/theory/pcqcd/L12T12_b5.8458_id1/cond/rome/L12T12_b5.8458_cond_run1.tar /dev/null
  • for WAN transfers use -S15 option to use up to 15 parallel streams to speedup transfers

Interactive command line client:

  • xrd castorpublic.cern.ch
  • browse the tree with dirlist and cd

Some transfer tests done by me

Create a big random file on a local disk:

pcarda75: dd if=/dev/urandom of=20GB.RANDOM.TEST bs=20M count=1000
1000+0 records in
1000+0 records out
20971520000 bytes (21 GB) copied, 5908.11 s, 3.5 MB/s

Copy it to Castor using xrootd (intranet):

time xrdcp 20GB.RANDOM.TEST root://castorpublic.cern.ch//castor/cern.ch/user/m/moscicki/20GB.RANDOM.TEST
Disabling apmon monitoring since env variable APMON_CONFIG was not found
[xrootd] Total 20000.00 MB	|====================| 100.00 % [10.9 MB/s]

real	32m6.181s
user	0m9.925s
sys	2m37.810s

Setting up proxy server at CERN with public access

Instructions are here:

/afs/cern.ch/sw/arda/install/theory/xrootd

Grid tools at CERN

Enable debug output for SOAP clients (srmcp,lcg-cp):

export CGSI_TRACE=1

Setup environment:

source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh

GridFTP

Example:

grid-proxy-init
globus-url-copy gsiftp://lxfsrk5801.cern.ch:2811///castor/cern.ch/user/m/moscicki/tmp.test file:///tmp/tmp.test

srmcp from dcache

Here is the trick:

srmcp -srm_protocol_version 2 srm://srm-public.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/user/m/moscicki/tmp.test file:///./tmp.test

Version mismatch if v1 (default) is used:

Possible reason for this error is an outdate client SOAP protocol which is not understood by the server deployed at CERN:

> srmcp srm://srm-public.cern.ch/castor/cern.ch/user/m/moscicki/tmp.test file:///tmp/tmp.txt
WARNING: SRM_PATH is defined, which might cause a wrong version of srm client to be executed
WARNING: SRM_PATH=/afs/cern.ch/project/gd/LCG-share/3.1.38-0/d-cache/srm
SRMClientV1 : Method 'ns1:get' not implemented: method name or namespace not recognized
SRMClientV1 : get : try # 0 failed with error
SRMClientV1 : Method 'ns1:get' not implemented: method name or namespace not recognized
srm copy of at least one file failed or not completed

lcg-cp

lcg-cp srm://srm-public.cern.ch/castor/cern.ch/user/m/moscicki/tmp.test file:///tmp/tmp.txt

-- JakubMoscicki - 12-Jan-2010

Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r7 - 2010-02-25 - JakubMoscicki
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ArdaGrid All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback