Joining a dCache-based SE to the Xrootd service.

This document covers joining a dCache-based storage element to the CMS Xrootd service based on the redirector The architecture setup is diagrammed below:


This architecture uses the built-in dCache Xrootd door and adds a "proxy host". The proxy host gives the following missing functionality:

  1. GSI security
  2. Namespace translation (from site namespace to CMS namespace and vice-versa)
  3. Integration with the global federation.
It additionally allows a site to keep its dCache pools behind a firewall. Note that future versions of dCache provide (1); it may be possible to serve data directly from pools in the future.

After being redirected to the site, a user will contact the xrootd daemon of the Xrootd proxy host. The filesystem plugin for this host is; this library is simply a wrapper around the xrootd client. The proxy then acts as a client inside the network boundary; it first contacts the dCache door to locate the file (or other metadata operations), then contacts a data pool directly to receive the file.


First, install the OSG software repository. For SL6:

rpm -Uhv

For SL7:

rpm -Uhv

Then, install Xrootd using yum. Two notes before doing this:

  • This will add the xrootd user if it does not already exist - ROCKS users might want to create this user beforehand.
  • This will install certificates into /etc/grid-security/certificates. If you want to handle certificates on your own, doing the following will satisfy the dependency:
    yum install empty-ce-certs

Then, install Xrootd using yum. This will add the xrootd user if it does not already exist - ROCKS users might want to create this user beforehand.

yum install --enablerepo=osg-testing xrootd xrootd-cmstfc
The version should be at least 3.1.0.

Warning: The CMS transition to 3.1.0 from previous versions is not a clean upgrade (as we switched to the CERN-based packaging). We believe this is a one-time-only event. Unfortunately, folks will need to remove all local copies of xrootd, lcmaps, lcas, xrootd-lcmaps, xrootd-cmstfc, and lcas-lcmaps-gt4-interface before installing.

If the node does not already have CA certificates and fetch-crl installed, you can also do this from the OSG Xrootd repo:

yum install fetch-crl osg-ca-certs

Copy the template config file, /etc/xrootd/xrootd.sample.dcache.cfg to /etc/xrootd/xrootd.cfg.

Copy your site's storage.xml to /etc/xrootd/storage.xml. If you are unsure of what this means, please contact your site's CMS representative. Uncomment and update the oss.namelib line in xrootd.cfg to read:

oss.namelib /usr/lib64/ file:/etc/xrootd/storage.xml?protocol=direct

When using the protocol direct with the storage.xml, it should map CMS filenames starting with /store to something starting with /pnfs. Adjust the protocol name accordingly for your site.

Finally, create a copy of the host certs to be xrootd service certs:

mkdir -p /etc/grid-security/xrd
cp /etc/grid-security/hostcert.pem /etc/grid-security/xrd/xrdcert.pem
cp /etc/grid-security/hostkey.pem /etc/grid-security/xrd/xrdkey.pem
chown xrootd: -R /etc/grid-security/xrd
chmod 400 /etc/grid-security/xrd/xrdkey.pem # Yes, 400 is required

Integrating with GUMS or SCAS

In order to integrate xrootd with GUMS (v1.3 or higher), Argus, or SCAS, install the following RPM:

yum install xrootd-lcmaps
This will bring in several dependencies, including Globus libraries, from the OSG. These do not appear to conflict with gLite installs of these libraries, but please be careful.

Next, copy/paste the following line from /etc/xrootd/lcmaps.cfg into /etc/xrootd/xrootd-clustered.cfg:

# sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfunparms:--osg,--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0|useglobals -gmapopt:10 -gmapto:0
Uncomment the line in xrootd-clustered.cfg, of course.

For GUMS or SCAS, update the /etc/xrootd/lcmaps.cfg provided in the RPM so the endpoint properly references your server's XACML endpoint. For Argus, use the attached lcmaps.cfg.

If this is a brand new host, you may need to run fetch-crl to update CRLs before starting Xrootd.

Operating xrootd

There are two init services, xrootd and cmsd, which must both be working for the site to participate in the xrootd service:

service xrootd start
service cmsd start

Everything is controlled by a proper init script (available commands are start, stop, restart, status, and condrestart). To enable these on boot, run:

chkconfig --level 345 xrootd on
chkconfig --level 345 cmsd on

Log files are kept in /var/log/xrootd/{cmsd,xrootd}.log, and are auto-rotated.

After startup, the xrootd and cmsd daemons drop privilege to the xrootd user.

If you used the RPM version of fetch-crl, you will need to enable and start the fetch-crl-cron and fetch-crl-boot services. To start:

service fetch-crl-cron
service fetch-crl-boot # This may take awhile to run

To enable on boot:

chkconfig --level 345 fetch-crl-cron on
chkconfig --level 345 fetch-crl-boot on

Port usage:

The following information is probably needed for sites with strict firewalls:
  • The xrootd server listens on TCP port 1094.
  • The cmsd server needs outgoing TCP port 1213 to
  • Usage statistics are sent to on UDP ports 3333 and 3334.

Testing the install.

The newly installed server can be tested directly using:
xrdcp -d 1 -f xroot:// /dev/null
You will need a grid certificate installed in your user account for the above to work

You can then see if your server is participating properly in the xrootd service by checking:

xrdcp root:// /tmp/bar2
where /store/foo/bar is unique to your site

Working Configuration

T2_FR_CCIN2P3 has donated a working set of configuration files, attached to this page. We've replaced their actual hostnames with:

  • - the dCache xrootd door.
  • - the dCache SRM door.
  • dcap - the dCache dCap door.

Please feel free to contact hn-cms-wanaccess for further help.

Topic attachments
I Attachment History Action Size Date Who Comment
XMLxml storage.xml r1 manage 3.1 K 2013-09-17 - 15:44 BrianBockelman  
Texttxt xrootd-clustered.cfg.txt r1 manage 2.1 K 2013-09-17 - 15:44 BrianBockelman  
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2016-10-04 - MarianZvada
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback