Xrootd 3rd Party Copy (TPC)

Variations of Xrootd TPC Implementations

Xrootd Proxy as a DTN

Hardware and OS

You need a machine with a fast network connection (10Gbps or faster recommended). This machine should allow inbound traffic to xrootd port 1094, allow unrestricted outbound traffic, and run SL6/SL7 x86_64.

Installation

Grid environment:

Make sure a standard EGI or OSG environment is installed, including
  • host certificate (default /etc/grid-security/{hostcert.pem, hostkey.pem})
  • CA certificates (default /etc/grid-security/certificates)
  • voms certificates (default /etc/grid-security/vomsdir)

copy /etc/grid-security/{hostcert.pem, hostkey.pem} to /etc/grid-security/xrd/{xrdcert.pem, xrdkey.pem}. Keep the permission bits and change ownership of these files to whoever runs the xrootd and cmsd process.

Set up YUM repositories

Please enable the EPEL or OSG repo, and the WLCG repo (see http://linuxsoft.cern.ch/wlcg/).

Install the RPMs

  • Install or upgrade Xrootd rpms
yum install xrootd. This will also added user "xrootd" and group "xrootd" to your system if they don't already exist. Make sure you install Xrootd 4.11.1 or above.

  • Install VOMS authorization rpm
yum install vomsxrd. This rpm requires voms rpm, which is available from EPEL repo and OSG repo.

  • Files to check
From time to time, you may need to check or change the following files for configuration changes or debugging:

  • /etc/xrootd/xrootd-clustered.cfg : Main Xrootd configuration file
  • /etc/sysconfig/xrootd (SL6): System level configuration to set up runtime environment for Xrootd
  • /etc/xrootd/Environment (SL7): Setup environment variables before xrootd and cmsd start. This file by default does not exist (see below)
  • /etc/xrootd/auth_file : Xrootd Authentication / Authorization file
  • /var/log/xrootd/xrootd.log : xrootd log file
  • /var/log/xrootd/cmsd.log : cmsd log file

Configuration

System level configuration

SL6 platform
/etc/sysconfig/xrootd is a shell script that setup the runtime environment for xrootd and cmsd.
  • Adjust XROOTD_USER and XROOTD_GROUP if needed. This is the owner that will run xrootd and cmsd processes.
  • X509_CERT_DIR and X509_VOMS_DIR can also be defined in this file if they are in non-default locations.

If XROOTOD_USER and/or XROOTD_GROUP is changed, run the following command as root once:

service xrootd setup
to update the directory permissions (and do this every time xrootd rpms are updated).

SL7 platform
run the following command
systemctl enable xrootd@clustered
# systemctl enable cmsd@clustered
The user and group that runs xrootd and cmsd are defined in /etc/systemd/system/multi-user.target.requires/{xrootd,cmsd}@clustered.service, so you need to adjust in these two files. Once you make change to user/group, make sure owner/group of /var/log/xrootd, /var/spool/xrootd /var/spool/xrootd and their sub-directories are all updated.

One can also add a line

EnvironmentFile=/etc/xrootd/EnvironmentFile
to the [Service] session in the above systemd service files. Then add X509_CERT_DIR and X509_VOMS_DIR definitions to /etc/xrootd/Environment (this file will be "sourced" as a shell script). Please make the permission of this file is 755.

Xrootd configuration

/etc/xrootd/xrootd-clustered.cfg is the main xrootd configuration file. The following is a template for a single server configuration. The first few lines are site specific.
## "redirector" should be full qualified DNS name (e.g. hostname -f). 
## If you are running a single xrootd service, leave the "set redirector" line below as it is.
#set redirector = www.google.com
xrd.port 1094
all.export <my.storage.path>

pss.origin <mybackend.storage.xrootd.door>:<port>

# the rest of this configuration file doesn't need to be touched except to add/remove VOs from the "sec.protparm" line
# when that happens, also remember to update the access control file /etc/xrootd/auth_file

#all.manager proxy $(redirector):1213
all.adminpath /var/spool/xrootd/var/spool
all.pidpath   /var/run/xrootd/var/run

ofs.tpc fcreds gsi =X509_USER_PROXY ttl 60 70 xfr 20 autorm pgm /etc/xrootd/xrdcp-tpc.sh
xrootd.chksum adler32 /etc/xrootd/xrdadler32-tpc.sh

#if $(redirector)
#    all.role proxy manager
#    cms.dfs lookup distrib redirect immed
#    cms.delay startup 30
#    pss.ckslib adler32 /usr/lib64/libXrdPss.so
#else
#    all.role proxy server
    ofs.osslib /usr/lib64/libXrdPss.so
    xrootd.seclib /usr/lib64/libXrdSec.so
    sec.protparm gsi -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:certfmt=pem|vos=atlas,cms,dteam|grps=/atlas,/cms,/dteam|grpopt=10|dbg
    sec.protocol /usr/lib64 gsi -dlgpxy:1 -exppxy:=creds -ca:1 -crl:3 -gridmap:/dev/null
    acc.audit deny
    acc.authdb /etc/xrootd/auth_file
    acc.authrefresh 60
    ofs.authorize
#fi

Access control file

The access control file, /etc/xrootd/auth_file should have the following lines:
g /atlas <my.storage.path> rwild
g /cms <my.storage.path> rwild
g /dteam <my.storage.path>/dteam/doma rwild

Other files

create /etc/xrootd/xrdcp-tpc.sh and /etc/xrootd/xrdadler32-tpc.sh with the following contents
$ cat /etc/xrootd/xrdcp-tpc.sh
#!/bin/sh
/usr/bin/xrdcp --server -f $1 root://$XRDXROOTD_PROXY/$2

$ cat /etc/xrootd/xrdadler32-tpc.sh
#!/bin/sh
/usr/bin/xrdadler32 root://$XRDXROOTD_PROXY/$1 | awk '{print $1}'
Both scripts need permission 755.

Start/Stop services

On SL6 platform, the command to start and stop xrootd and cmsd services are:
service xrootd start (or stop)
# service cmsd start (or stop)
On SL7 platform, the command to start and stop xrootd and cmsd services are:
systemctl start xrootd@clustered (or stop)
# systemctl start cmsd@clustered (or stop)
Please refer to this page for detail.

To setup a cluster of xrootd proxy, both xrootd and cmsd services need to run on redirector and individual proxies. To setup a single xrootd proxy, only xrootd service is needed.

Credential delegation

The client will delegate its x509 credential to the destination in order to fetch data from the source. (Currently xrootd proxy can not use delegated user credential to access remote file on user's behave. This is still a work in progress. Until that is completed, a credential with read access to VO's files is needed by the xrootd proxy. In the case of ATLAS and CMS, this credential is a X509 proxy with VOMS attributes. The credential needs to be periodically refreshed before it expired).

Test the setup

To verify the TPC setup, use the following command:
/usr/bin/xrdcp --tpc -f only root://myhost:port//dir/myfile root://myhost:port//dir/myfile.new
Make sure /usr/bin/xrdcp --version shows version 4.11.1 or above. The above command test both TPC read and TPC write to the site.

Containerization

Temporary GitHub repo of (preliminary) container receipts: https://github.com/wyang007/Xrootd-Containers/tree/master/tpc - for a single DTN or a cluster of DTNs

Native Xrootd TPC on Posix Storage Systems

Configuration for Native Xrootd TPC is similar to those in Xrootd Proxy, with the following changed configuration files:

/etc/xrootd/xrootd-clustered.cfg

## "redirector" should be full qualified DNS name (e.g. hostname -f). 
## If you are running a single xrootd service, leave the "set redirector" line below as it is.
#set redirector = www.google.com
xrd.port 1094
all.export <my.storage.path>

# the rest of this configuration file doesn't need to be touched except to add/remove VOs from the "sec.protparm" line
# when that happens, also remember to update the access control file /etc/xrootd/auth_file

#all.manager $(redirector):1213
all.adminpath /var/spool/xrootd/var/spool
all.pidpath   /var/run/xrootd/var/run

ofs.tpc fcreds gsi =X509_USER_PROXY ttl 60 70 xfr 20 autorm pgm /etc/xrootd/xrdcp-tpc.sh
xrootd.chksum adler32 /etc/xrootd/xrdadler32-tpc.sh

#if $(redirector)
#    all.role manager
#    cms.dfs lookup distrib redirect immed
#    cms.delay startup 30
#    pss.ckslib adler32 /usr/lib64/libXrdPss.so
#else
#    all.role server
    xrootd.seclib /usr/lib64/libXrdSec.so
    sec.protparm gsi -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:certfmt=pem|vos=atlas,cms,dteam|grps=/atlas,/cms,/dteam|grpopt=10|dbg
    sec.protocol /usr/lib64 gsi -dlgpxy:1 -exppxy:=creds -ca:1 -crl:3 -gridmap:/dev/null
    acc.audit deny
    acc.authdb /etc/xrootd/auth_file
    acc.authrefresh 60
    ofs.authorize
#fi

/etc/xrootd/xrdcp-tpc.sh

$ cat /etc/xrootd/xrdcp-tpc.sh
#!/bin/sh
/usr/bin/xrdcp --server -f $1 $2

/etc/xrootd/xrdadler32-tpc.sh

$ cat /etc/xrootd/xrdadler32-tpc.sh
#!/bin/sh
/usr/bin/xrdadler32 $1 | awk '{print $1}'
Both scripts need permission 755.

In the checksum script, one may use other ways to store and calculate checksum. Xrootd only cares that the checksum script xrdadler32-tpc.sh returns a single word (the checksum)

TPC with dCache

TPC with DPM

For DPM (>= 1.10) DOME, please refer to DPM deployment guide. For more technical detail on DPM Xrootd TPC configuration, including x509 user proxy delegation, check this DPM manual installation Twiki

TPC with EOS

TPC with CEPH

TPC with Storm

If the storage systems at a Storm site is posix (such as GPFS or Lustre), please refer to the Posix storage section.

Deployment Status

EOS & CASTOR

Both EOS and CASTOR supports Xrootd TPC. X509 robot certificates are not available so transferring data to EOS and CASTOR from outside will not work, if the outside endpoints requires X509 authentication. Transfer between EOS and CASTOR does not require X509 authentication, and therefore works well.

Xrootd & Posix

  • SLAC: a cluster of (containerized) Xrootd DTNs to move data in and out of the SLAC xrootd storage.
  • NERSC: a single Xrootd DTN to move data in and out of their Lustre filesystem.
  • Uni-Bonn: seven Xrootd DTNs with one redirector, data on shared CephFS storage

Integration with FTS and RUCIO

Completed. ATLAS has been able to use RUCIO to drive, through FTS to move data between Xrootd endpoints at EOS, CASTOR, SLAC and NERSC.

Open Tickets / Issues

  • gfal2-xrootd plugin
    • Implement performance markers logic as in gridftp plugin (DMC-1016)
  • Xrootd
    • The tpcTimeout passed to the XrdCl::CopyProcess is not honored, so the default 1800 sec timeout is always used. ( no ticket open yet but confirmed by the devs)

-- AlessandraForti - 2018-08-01

Edit | Attach | Watch | Print version | History: r16 < r15 < r14 < r13 < r12 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r16 - 2020-05-07 - WeiYang
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback