Joining a POSIX-based cluster to the Xrootd service.

This document covers joining a POSIX-based filesystem (Lustre, GPFS, NFS, etc) the USCMS Xrootd service based on the redirector We have also integrated HDFS and dCache into Xrootd, but those are covered on other pages; if you have another file system you'd like to use, please let us know.


First, install the OSG software repository. For SL6:

rpm -Uhv

For SL5:

rpm -Uhv

Then, install Xrootd using yum. Two notes before doing this:

  • This will add the xrootd user if it does not already exist - ROCKS users might want to create this user beforehand.
  • This will install certificates into /etc/grid-security/certificates. If you want to handle certificates on your own, doing the following will satisfy the dependency:
    yum install empty-ca-certs

For the installation, you should actually use the cms-xrootd meta-RPM:

yum install cms-xrootd --disablerepo=* --enablerepo=osg-testing,osg-contrib
The version of cms-xrootd should be at least 1.0, and the version of xrootd should be at least 3.2.2.

Copy the template config file, /etc/xrootd/xrootd.sample.posix.cfg to /etc/xrootd/xrootd-clustered.cfg.

If your CMS namespace is not truly trivial (i.e., if the CMS top-level directory in Lustre/GPFS is not /store), copy your site's storage.xml to /etc/xrootd/storage.xml. If you are unsure of what this means, please contact your site's CMS representative. Uncomment and update the following line in xrootd.cfg:

#oss.namelib /usr/lib64/ file:/etc/xrootd/storage.xml?protocol=direct

You need to update the protocol you use; most Lustre/GPFS sites will use protocol=posix or protocol=direct.

Finally, create a copy of the host certs to be xrootd service certs:

mkdir -p /etc/grid-security/xrd
cp /etc/grid-security/hostcert.pem /etc/grid-security/xrd/xrdcert.pem
cp /etc/grid-security/hostkey.pem /etc/grid-security/xrd/xrdkey.pem
chown xrootd: -R /etc/grid-security/xrd
chmod 400 /etc/grid-security/xrd/xrdkey.pem # Yes, 400 is required

Integrating with GUMS, SCAS, or Argus

In order to integrate xrootd with GUMS (v1.3 or higher), Argus, or SCAS, install the following RPM:

yum install xrootd-lcmaps
yum localinstall argus-pep-api-c-2.1.0-3.sl5.x86_64.rpm lcmaps-plugins-c-pep-1.2.2-1.el5.x86_64.rpm
This will bring in several dependencies, including Globus libraries, from the OSG. These do not appear to conflict with gLite installs of these libraries, but please be careful.

Next, copy/paste the following line from /etc/xrootd/lcmaps.cfg into /etc/xrootd/xrootd-clustered.cfg:

# sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfunparms:--osg,--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0|useglobals -gmapopt:10 -gmapto:0
Uncomment the line in xrootd-clustered.cfg, of course.

For GUMS or SCAS, update the /etc/xrootd/lcmaps.cfg provided in the RPM so the endpoint properly references your server's XACML endpoint. For Argus, use the attached lcmaps.cfg.

Change the path on lcmaps.cfg file (from path = /usr/lib64/modules to path = /usr/lib64/lcmaps )

If this is a brand new host, you may need to run fetch-crl to update CRLs before starting Xrootd.

Operating xrootd

There are two init services, xrootd and cmsd, which must both be working for the site to participate in the xrootd service:

service xrootd start
service cmsd start

Everything is controlled by a proper init script (available commands are start, stop, restart, status, and condrestart).

Log files are kept in /var/log/xrootd/{cmsd,xrootd}.log, and are auto-rotated.

After startup, the xrootd and cmsd daemons drop privilege to the xrootd user.

Port usage:

The following information is probably needed for sites with strict firewalls:
  • The xrootd server listens on TCP port 1094.
  • The cmsd server needs outgoing TCP port 1213 to
  • Usage statistics are sent to on UDP ports 9930 and 9931.

Testing the install.

The newly installed server can be tested directly using:
xrdcp xroot:// /tmp/bar
You will need a grid certificate installed in your user account for the above to work

You can then see if your server is participating properly in the xrootd service by checking:

xrdcp root:// /tmp/bar2
where /store/foo/bar is unique to your site
Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r11 - 2013-07-05 - BrianBockelman
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback