Difference: DcacheXrootdOld (1 vs. 8)

Revision 82016-10-04 - MarianZvada

Line: 1 to 1
 
META TOPICPARENT name="CmsXrootdArchitecture"

Joining a dCache-based SE to the Xrootd service.

Line: 18 to 18
  First, install the OSG software repository. For SL6:
Changed:
<
<
rpm -Uhv http://repo.grid.iu.edu/osg-el6-release-latest.rpm
>
>
rpm -Uhv http://repo.grid.iu.edu/osg/3.3/osg-3.3-el6-release-latest.rpm
 
Changed:
<
<
For SL5:
>
>
For SL7:
 
Changed:
<
<
rpm -Uhv http://repo.grid.iu.edu/osg-el5-release-latest.rpm
>
>
rpm -Uhv http://repo.grid.iu.edu/osg/3.3/osg-3.3-el7-release-latest.rpm
 

Then, install Xrootd using yum. Two notes before doing this:

Line: 93 to 93
 Everything is controlled by a proper init script (available commands are start, stop, restart, status, and condrestart). To enable these on boot, run:
Changed:
<
<
chkconfig --level 345 xroot on
>
>
chkconfig --level 345 xrootd on
 chkconfig --level 345 cmsd on

Revision 72013-09-17 - BrianBockelman

Line: 1 to 1
 
META TOPICPARENT name="CmsXrootdArchitecture"

Joining a dCache-based SE to the Xrootd service.

Line: 133 to 133
  where /store/foo/bar is unique to your site
Added:
>
>

Working Configuration

T2_FR_CCIN2P3 has donated a working set of configuration files, attached to this page. We've replaced their actual hostnames with:

  • xrootd.example.com - the dCache xrootd door.
  • srm.example.com - the dCache SRM door.
  • dcap - the dCache dCap door.

Please feel free to contact hn-cms-wanaccess for further help.

 
META FILEATTACHMENT attachment="XrootdDcacheIntegration.png" attr="h" comment="" date="1320191437" name="XrootdDcacheIntegration.png" path="XrootdDcacheIntegration.png" size="45366" stream="XrootdDcacheIntegration.png" tmpFilename="/usr/tmp/CGItemp43673" user="bbockelm" version="1"
Added:
>
>
META FILEATTACHMENT attachment="storage.xml" attr="" comment="" date="1379425460" name="storage.xml" path="storage.xml" size="3146" user="bbockelm" version="1"
META FILEATTACHMENT attachment="xrootd-clustered.cfg.txt" attr="" comment="" date="1379425460" name="xrootd-clustered.cfg.txt" path="xrootd-clustered.cfg.txt" size="2153" user="bbockelm" version="1"

Revision 62013-07-05 - BrianBockelman

Line: 1 to 1
 
META TOPICPARENT name="CmsXrootdArchitecture"

Joining a dCache-based SE to the Xrootd service.

Line: 16 to 16
 

Installation

Changed:
<
<
First, install the OSG Xrootd repository:
>
>
First, install the OSG software repository. For SL6:
 
Changed:
<
<
rpm -Uvh http://repo.grid.iu.edu/osg-release-latest.rpm
>
>
rpm -Uhv http://repo.grid.iu.edu/osg-el6-release-latest.rpm

For SL5:

rpm -Uhv http://repo.grid.iu.edu/osg-el5-release-latest.rpm
 

Then, install Xrootd using yum. Two notes before doing this:

Revision 52012-10-23 - BrianBockelman

Line: 1 to 1
 
META TOPICPARENT name="CmsXrootdArchitecture"

Joining a dCache-based SE to the Xrootd service.

Line: 74 to 74
  For GUMS or SCAS, update the /etc/xrootd/lcmaps.cfg provided in the RPM so the endpoint properly references your server's XACML endpoint. For Argus, use the attached lcmaps.cfg.
Added:
>
>
If this is a brand new host, you may need to run fetch-crl to update CRLs before starting Xrootd.
 

Operating xrootd

There are two init services, xrootd and cmsd, which must both be working for the site to participate in the xrootd service:

Revision 42012-09-14 - BrianBockelman

Line: 1 to 1
 
META TOPICPARENT name="CmsXrootdArchitecture"

Joining a dCache-based SE to the Xrootd service.

Line: 60 to 60
 

Integrating with GUMS or SCAS

Changed:
<
<
In order to integrate xrootd with GUMS (v1.3 or higher) or SCAS, install the following RPM:
>
>
In order to integrate xrootd with GUMS (v1.3 or higher), Argus, or SCAS, install the following RPM:
 
Changed:
<
<
yum install --enablerepo=osg-testing xrootd-lcmaps
>
>
yum install xrootd-lcmaps
 
Changed:
<
<
This will bring in several dependencies, including Globus libraries. This does not appear to conflict with gLite installs of these libraries, but please be careful.
>
>
This will bring in several dependencies, including Globus libraries, from the OSG. These do not appear to conflict with gLite installs of these libraries, but please be careful.
 
Changed:
<
<
Next, copy/paste the following line from /etc/xrootd/lcmaps.cfg into /etc/xrootd/xrootd.cfg:
>
>
Next, copy/paste the following line from /etc/xrootd/lcmaps.cfg into /etc/xrootd/xrootd-clustered.cfg:
 
# sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfun:libXrdLcmaps.so -authzfunparms:--osg,--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0|useglobals -gmapopt:10 -gmapto:0
Added:
>
>
Uncomment the line in xrootd-clustered.cfg, of course.
 
Changed:
<
<
Further, update /etc/xrootd/lcmaps.cfg so the endpoint properly references your GUMS or SCAS server's XACML endpoint.
>
>
For GUMS or SCAS, update the /etc/xrootd/lcmaps.cfg provided in the RPM so the endpoint properly references your server's XACML endpoint. For Argus, use the attached lcmaps.cfg.
 

Operating xrootd

Revision 32011-11-05 - BrianBockelman

Line: 1 to 1
 
META TOPICPARENT name="CmsXrootdArchitecture"

Joining a dCache-based SE to the Xrootd service.

Line: 18 to 18
  First, install the OSG Xrootd repository:
Changed:
<
<
if [ ! -e /etc/yum.repos.d/osg-xrootd.repo ]; then rpm -Uvh http://newman.ultralight.org/repos/xrootd/x86_64/osg-xrootd-1-1.noarch.rpm fi
>
>
rpm -Uvh http://repo.grid.iu.edu/osg-release-latest.rpm
 
Added:
>
>
Then, install Xrootd using yum. Two notes before doing this:
  • This will add the xrootd user if it does not already exist - ROCKS users might want to create this user beforehand.
  • This will install certificates into /etc/grid-security/certificates. If you want to handle certificates on your own, doing the following will satisfy the dependency:
    yum install empty-ce-certs
 Then, install Xrootd using yum. This will add the xrootd user if it does not already exist - ROCKS users might want to create this user beforehand.
Changed:
<
<
yum install xrootd xrootd-cmstfc
>
>
yum install --enablerepo=osg-testing xrootd xrootd-cmstfc
 
Changed:
<
<
The version should be at least 3.0.3-0.pre9. If the node does not already have CA certificates and fetch-crl installed, you can also do this from the OSG Xrootd repo:
>
>
The version should be at least 3.1.0.

Warning: The CMS transition to 3.1.0 from previous versions is not a clean upgrade (as we switched to the CERN-based packaging). We believe this is a one-time-only event. Unfortunately, folks will need to remove all local copies of xrootd, lcmaps, lcas, xrootd-lcmaps, xrootd-cmstfc, and lcas-lcmaps-gt4-interface before installing.

If the node does not already have CA certificates and fetch-crl installed, you can also do this from the OSG Xrootd repo:

 
yum install fetch-crl osg-ca-certs
Line: 55 to 62
  In order to integrate xrootd with GUMS (v1.3 or higher) or SCAS, install the following RPM:
Changed:
<
<
yum install xrootd-lcmaps
>
>
yum install --enablerepo=osg-testing xrootd-lcmaps
 
Changed:
<
<
This will bring in several dependencies, including Globus libraries. This does not appear to conflict with OSG or gLite installs of these libraries.
>
>
This will bring in several dependencies, including Globus libraries. This does not appear to conflict with gLite installs of these libraries, but please be careful.
  Next, copy/paste the following line from /etc/xrootd/lcmaps.cfg into /etc/xrootd/xrootd.cfg:

Revision 22011-11-02 - BrianBockelman

Line: 1 to 1
 
META TOPICPARENT name="CmsXrootdArchitecture"

Joining a dCache-based SE to the Xrootd service.

Line: 61 to 61
  Next, copy/paste the following line from /etc/xrootd/lcmaps.cfg into /etc/xrootd/xrootd.cfg:
Changed:
<
<
# sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfun:libXrdLcmaps.so -authzfunparms:--osg,--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0|useglobals --gmapopt:2 --gmapto:0
>
>
# sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfun:libXrdLcmaps.so -authzfunparms:--osg,--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0|useglobals -gmapopt:10 -gmapto:0
 

Further, update /etc/xrootd/lcmaps.cfg so the endpoint properly references your GUMS or SCAS server's XACML endpoint.

Revision 12011-11-02 - BrianBockelman

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="CmsXrootdArchitecture"

Joining a dCache-based SE to the Xrootd service.

This document covers joining a dCache-based storage element to the CMS Xrootd service based on the redirector xrootd-itb.unl.edu. The architecture setup is diagrammed below:

XrootdDcacheIntegration.png

This architecture uses the built-in dCache Xrootd door and adds a "proxy host". The proxy host gives the following missing functionality:

  1. GSI security
  2. Namespace translation (from site namespace to CMS namespace and vice-versa)
  3. Integration with the global federation.
It additionally allows a site to keep its dCache pools behind a firewall. Note that future versions of dCache provide (1); it may be possible to serve data directly from pools in the future.

After being redirected to the site, a user will contact the xrootd daemon of the Xrootd proxy host. The filesystem plugin for this host is libXrdPss.so; this library is simply a wrapper around the xrootd client. The proxy then acts as a client inside the network boundary; it first contacts the dCache door to locate the file (or other metadata operations), then contacts a data pool directly to receive the file.

Installation

First, install the OSG Xrootd repository:

if [ ! -e /etc/yum.repos.d/osg-xrootd.repo ]; then
  rpm -Uvh http://newman.ultralight.org/repos/xrootd/x86_64/osg-xrootd-1-1.noarch.rpm
fi

Then, install Xrootd using yum. This will add the xrootd user if it does not already exist - ROCKS users might want to create this user beforehand.

yum install xrootd xrootd-cmstfc
The version should be at least 3.0.3-0.pre9. If the node does not already have CA certificates and fetch-crl installed, you can also do this from the OSG Xrootd repo:
yum install fetch-crl osg-ca-certs

Copy the template config file, /etc/xrootd/xrootd.sample.dcache.cfg to /etc/xrootd/xrootd.cfg.

Copy your site's storage.xml to /etc/xrootd/storage.xml. If you are unsure of what this means, please contact your site's CMS representative. Uncomment and update the oss.namelib line in xrootd.cfg to read:

oss.namelib /usr/lib64/libXrdCmsTfc.so file:/etc/xrootd/storage.xml?protocol=direct

When using the protocol direct with the storage.xml, it should map CMS filenames starting with /store to something starting with /pnfs. Adjust the protocol name accordingly for your site.

Finally, create a copy of the host certs to be xrootd service certs:

mkdir -p /etc/grid-security/xrd
cp /etc/grid-security/hostcert.pem /etc/grid-security/xrd/xrdcert.pem
cp /etc/grid-security/hostkey.pem /etc/grid-security/xrd/xrdkey.pem
chown xrootd: -R /etc/grid-security/xrd
chmod 400 /etc/grid-security/xrd/xrdkey.pem # Yes, 400 is required

Integrating with GUMS or SCAS

In order to integrate xrootd with GUMS (v1.3 or higher) or SCAS, install the following RPM:

yum install xrootd-lcmaps
This will bring in several dependencies, including Globus libraries. This does not appear to conflict with OSG or gLite installs of these libraries.

Next, copy/paste the following line from /etc/xrootd/lcmaps.cfg into /etc/xrootd/xrootd.cfg:

# sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfun:libXrdLcmaps.so -authzfunparms:--osg,--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0|useglobals --gmapopt:2 --gmapto:0

Further, update /etc/xrootd/lcmaps.cfg so the endpoint properly references your GUMS or SCAS server's XACML endpoint.

Operating xrootd

There are two init services, xrootd and cmsd, which must both be working for the site to participate in the xrootd service:

service xrootd start
service cmsd start

Everything is controlled by a proper init script (available commands are start, stop, restart, status, and condrestart). To enable these on boot, run:

chkconfig --level 345 xroot on
chkconfig --level 345 cmsd on

Log files are kept in /var/log/xrootd/{cmsd,xrootd}.log, and are auto-rotated.

After startup, the xrootd and cmsd daemons drop privilege to the xrootd user.

If you used the RPM version of fetch-crl, you will need to enable and start the fetch-crl-cron and fetch-crl-boot services. To start:

service fetch-crl-cron
service fetch-crl-boot # This may take awhile to run

To enable on boot:

chkconfig --level 345 fetch-crl-cron on
chkconfig --level 345 fetch-crl-boot on

Port usage:

The following information is probably needed for sites with strict firewalls:
  • The xrootd server listens on TCP port 1094.
  • The cmsd server needs outgoing TCP port 1213 to xrootd.unl.edu.
  • Usage statistics are sent to xrootd.t2.ucsd.edu on UDP ports 3333 and 3334.

Testing the install.

The newly installed server can be tested directly using:
xrdcp -d 1 -f xroot://local_hostname.example.com//store/foo/bar /dev/null
You will need a grid certificate installed in your user account for the above to work

You can then see if your server is participating properly in the xrootd service by checking:

xrdcp root://xrootd.unl.edu//store/foo/bar /tmp/bar2
where /store/foo/bar is unique to your site

META FILEATTACHMENT attachment="XrootdDcacheIntegration.png" attr="h" comment="" date="1320191437" name="XrootdDcacheIntegration.png" path="XrootdDcacheIntegration.png" size="45366" stream="XrootdDcacheIntegration.png" tmpFilename="/usr/tmp/CGItemp43673" user="bbockelm" version="1"
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback