THIS PAGE IS BEING DECOMMISSIONED - PLEASE DON'T EDIT - USE ITS SUCCESSOR HERE
Joining a POSIX-based cluster to the Xrootd service.
This document covers joining a POSIX-based filesystem (Lustre, GPFS, NFS, etc) the USCMS Xrootd service based on the redirector xrootd.unl.edu. We have also integrated HDFS and dCache into Xrootd, but those are covered on other pages; if you have another file system you'd like to use, please let us know.
Installation
First, install the OSG software repository. For SL6:
rpm -Uhv http://repo.grid.iu.edu/osg-el6-release-latest.rpm
For SL5:
rpm -Uhv http://repo.grid.iu.edu/osg-el5-release-latest.rpm
Then, install Xrootd using yum. Two notes before doing this:
- This will add the
xrootd
user if it does not already exist - ROCKS users might want to create this user beforehand.
- This will install certificates into
/etc/grid-security/certificates
. If you want to handle certificates on your own, doing the following will satisfy the dependency:
yum install empty-ca-certs
For the installation, you should actually use the
cms-xrootd
meta-RPM:
yum install xerces-c-3.0.1 # It is in the epel repo
yum install cms-xrootd --disablerepo=* --enablerepo=osg-testing,osg-contrib
yum install lcmaps-plugins-verify-proxy.x86_64 lcmaps-plugins-tracking-groupid.x86_64
The version of cms-xrootd should be at least 1.0, and the version of xrootd should be at least 3.2.2.
Copy the template config file,
/etc/xrootd/xrootd.sample.posix.cfg
to
/etc/xrootd/xrootd-clustered.cfg
.
If your CMS namespace is not truly trivial (i.e., if the CMS top-level directory in Lustre/GPFS is not /store), copy your site's storage.xml to
/etc/xrootd/storage.xml
. If you are unsure of what this means, please contact your site's CMS representative. Uncomment and update the following line in
xrootd.cfg
:
#oss.namelib /usr/lib64/libXrdCmsTfc.so file:/etc/xrootd/storage.xml?protocol=direct
You need to update the protocol you use; most Lustre/GPFS sites will use
protocol=posix
or
protocol=direct
.
Finally, create a copy of the host certs to be xrootd service certs:
mkdir -p /etc/grid-security/xrd
cp /etc/grid-security/hostcert.pem /etc/grid-security/xrd/xrdcert.pem
cp /etc/grid-security/hostkey.pem /etc/grid-security/xrd/xrdkey.pem
chown xrootd: -R /etc/grid-security/xrd
chmod 400 /etc/grid-security/xrd/xrdkey.pem # Yes, 400 is required
Integrating with GUMS, SCAS, or Argus
In order to integrate xrootd with GUMS (v1.3 or higher), Argus, or SCAS, install the following RPM:
yum install xrootd-lcmaps
wget http://repository.egi.eu/sw/production/umd/2/sl5/x86_64/updates/lcmaps-plugins-c-pep-1.2.2-1.el5.x86_64.rpm
wget http://repository.egi.eu/sw/production/umd/2/sl5/x86_64/base/argus-pep-api-c-2.1.0-3.sl5.x86_64.rpm
yum localinstall argus-pep-api-c-2.1.0-3.sl5.x86_64.rpm lcmaps-plugins-c-pep-1.2.2-1.el5.x86_64.rpm
This will bring in several dependencies, including Globus libraries, from the OSG. These do not appear to conflict with gLite installs of these libraries, but please be careful.
Next, copy/paste the following line from
/etc/xrootd/lcmaps.cfg
into
/etc/xrootd/xrootd-clustered.cfg
:
# sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfun:libXrdLcmaps.so -authzfunparms:--osg,--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0|useglobals -gmapopt:10 -gmapto:0
Uncomment the line in
xrootd-clustered.cfg
, of course.
For GUMS or SCAS, update the
/etc/xrootd/lcmaps.cfg
provided in the RPM so the endpoint properly references your server's XACML endpoint. For Argus, use the
attached lcmaps.cfg.
Change the path on lcmaps.cfg file (from path = /usr/lib64/modules to path = /usr/lib64/lcmaps )
If this is a brand new host, you may need to run
fetch-crl
to update CRLs before starting Xrootd.
Operating xrootd
There are two init services,
xrootd
and
cmsd
, which must both be working for the site to participate in the xrootd service:
service xrootd start
service cmsd start
Everything is controlled by a proper init script (available commands are start, stop, restart, status, and condrestart).
Log files are kept in
/var/log/xrootd/{cmsd,xrootd}.log
, and are auto-rotated.
After startup, the xrootd and cmsd daemons drop privilege to the xrootd user.
Port usage:
The following information is probably needed for sites with strict firewalls:
- The xrootd server listens on TCP port 1094.
- The cmsd server needs outgoing TCP port 1213 to xrootd-itb.unl.edu.
- Usage statistics are sent to xrootd.unl.edu on UDP ports 9930 and 9931.
THIS PAGE IS BEING DECOMMISSIONED - PLEASE DON'T EDIT - USE ITS SUCCESSOR HERE
Testing the install.
The newly installed server can be tested directly using:
xrdcp xroot://local_hostname.example.com//store/foo/bar /tmp/bar
You will need a grid certificate installed in your user account for the above to work
You can then see if your server is participating properly in the xrootd service by checking:
xrdcp root://xrootd-itb.unl.edu//store/foo/bar /tmp/bar2
where
/store/foo/bar
is unique to your site