Joining a POSIX-based cluster to the Xrootd service.

This document covers joining a POSIX-based filesystem (Lustre, GPFS, NFS, etc) the USCMS Xrootd service based on the redirector xrootd.unl.edu. We have also integrated HDFS and dCache into Xrootd, but those are covered on other pages; if you have another file system you'd like to use, please let us know.

Installation

First, install the OSG software repository. For SL6:

rpm -Uhv http://repo.grid.iu.edu/osg-el6-release-latest.rpm

For SL5:

rpm -Uhv http://repo.grid.iu.edu/osg-el5-release-latest.rpm

Then, install Xrootd using yum. Two notes before doing this:

• This will add the xrootd user if it does not already exist - ROCKS users might want to create this user beforehand.
• This will install certificates into /etc/grid-security/certificates. If you want to handle certificates on your own, doing the following will satisfy the dependency:

yum install empty-ca-certs

For the installation, you should actually use the cms-xrootd meta-RPM:

yum install xerces-c-3.0.1 # It is in the epel repo
yum install cms-xrootd --disablerepo=* --enablerepo=osg-testing,osg-contrib
yum install  lcmaps-plugins-verify-proxy.x86_64 lcmaps-plugins-tracking-groupid.x86_64

The version of cms-xrootd should be at least 1.0, and the version of xrootd should be at least 3.2.2.

Copy the template config file, /etc/xrootd/xrootd.sample.posix.cfg to /etc/xrootd/xrootd-clustered.cfg.

If your CMS namespace is not truly trivial (i.e., if the CMS top-level directory in Lustre/GPFS is not /store), copy your site's storage.xml to /etc/xrootd/storage.xml. If you are unsure of what this means, please contact your site's CMS representative. Uncomment and update the following line in xrootd.cfg:

#oss.namelib /usr/lib64/libXrdCmsTfc.so file:/etc/xrootd/storage.xml?protocol=direct

You need to update the protocol you use; most Lustre/GPFS sites will use protocol=posix or protocol=direct.

Finally, create a copy of the host certs to be xrootd service certs:

mkdir -p /etc/grid-security/xrd
cp /etc/grid-security/hostcert.pem /etc/grid-security/xrd/xrdcert.pem
cp /etc/grid-security/hostkey.pem /etc/grid-security/xrd/xrdkey.pem
chown xrootd: -R /etc/grid-security/xrd
chmod 400 /etc/grid-security/xrd/xrdkey.pem # Yes, 400 is required

Integrating with GUMS, SCAS, or Argus

In order to integrate xrootd with GUMS (v1.3 or higher), Argus, or SCAS, install the following RPM:

yum install xrootd-lcmaps
wget http://repository.egi.eu/sw/production/umd/2/sl5/x86_64/updates/lcmaps-plugins-c-pep-1.2.2-1.el5.x86_64.rpm
wget  http://repository.egi.eu/sw/production/umd/2/sl5/x86_64/base/argus-pep-api-c-2.1.0-3.sl5.x86_64.rpm 
yum localinstall argus-pep-api-c-2.1.0-3.sl5.x86_64.rpm lcmaps-plugins-c-pep-1.2.2-1.el5.x86_64.rpm

This will bring in several dependencies, including Globus libraries, from the OSG. These do not appear to conflict with gLite installs of these libraries, but please be careful.

Next, copy/paste the following line from /etc/xrootd/lcmaps.cfg into /etc/xrootd/xrootd-clustered.cfg:

# sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfun:libXrdLcmaps.so -authzfunparms:--osg,--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0|useglobals -gmapopt:10 -gmapto:0

Uncomment the line in xrootd-clustered.cfg, of course.

For GUMS or SCAS, update the /etc/xrootd/lcmaps.cfg provided in the RPM so the endpoint properly references your server's XACML endpoint. For Argus, use the attached lcmaps.cfg.

Change the path on lcmaps.cfg file (from path = /usr/lib64/modules to path = /usr/lib64/lcmaps )

If this is a brand new host, you may need to run fetch-crl to update CRLs before starting Xrootd.

Operating xrootd

There are two init services, xrootd and cmsd, which must both be working for the site to participate in the xrootd service:

service xrootd start
service cmsd start

Everything is controlled by a proper init script (available commands are start, stop, restart, status, and condrestart).

Log files are kept in /var/log/xrootd/{cmsd,xrootd}.log, and are auto-rotated.

After startup, the xrootd and cmsd daemons drop privilege to the xrootd user.

Port usage:

The following information is probably needed for sites with strict firewalls:

• The xrootd server listens on TCP port 1094.
• The cmsd server needs outgoing TCP port 1213 to xrootd-itb.unl.edu.
• Usage statistics are sent to xrootd.unl.edu on UDP ports 9930 and 9931.

Testing the install.

The newly installed server can be tested directly using:

xrdcp xroot://local_hostname.example.com//store/foo/bar /tmp/bar

You will need a grid certificate installed in your user account for the above to work

You can then see if your server is participating properly in the xrootd service by checking:

xrdcp root://xrootd-itb.unl.edu//store/foo/bar /tmp/bar2

where /store/foo/bar is unique to your site