Enable HTTP protocol for XRootD


The xrootd daemon can export both the xrootd protocol and the https protocol; this page covers the configuration changes required for enabling HTTPS.

For clients to successfully read from the regional redirector, HTTPS must be enabled for the data servers and the site-level redirector.

Versions and Platforms

We strongly suggest†utilizing RHEL7 and XRootD 4.8.3.

Prior versions of XRootD have known segfaults in some of the HTTP code. The latest version of xrootd-lcmaps (1.3 or later) has features required by CMS and doesn't support RHEL6.

Xrootd Config Changes

Verify you have the most up-to-date version of the sec.protocol line:

sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfun:libXrdLcmaps.so -authzfunparms:--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0 -gmapopt:10 -gmapto:0

In the xrootd daemon specific configuration, enable HTTPS:

if exec xrootd
  xrd.protocol http:1094 /usr/lib64/libXrdHttp-4.so
  http.cadir /etc/grid-security/certificates
  http.cert /etc/grid-security/xrd/xrdcert.pem
  http.key /etc/grid-security/xrd/xrdkey.pem
  http.secxtractor /usr/lib64/libXrdLcmaps.so
  http.listingdeny yes
  http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt
  http.desthttps yes

Updates to Auth database contents

The /etc/xrootd/Authfile should contain these lines:

g /cms /store lr

In particular, it should not have a line that looks like this:

u * /store lr

Create robots.txt

Add the file robots.txt to /etc/xrootd with these contents:

User-agent: *
Disallow: /

Testing the configuration

Make sure you see file not found from any browser session, even if you have your certificate loaded. This should fail because a VOMS extension isnít present in your browser; we want to require VOMS.

From the terminal, generate a CMS VOMS proxy and attempt to use davix-get to copy from your XRootD host:

davix-get https://cmstest2.rcac.purdue.edu:1094/store/user/goughes/test.root -E /tmp/x509up_u28772 --capath /etc/grid-security/certificates
Do not use curl (as it does not support proxy certificates).

Enable HTTP-based Writes

In 2018, the USCMS sites are required to enable HTTPS as the stageout protocol.

Xrootd changes

The primary changes are to the Authfile; you will need to add several a (all) authorizations to where users need to be able to write. Here's an example:

t writecmsdata /store/backfill/          a  \
               /store/data/              a  \
               /store/generator/         a  \
               /store/group/             a  \
               /store/hidata/            a  \
               /store/mc/                a  \
               /store/PhEDEx_LoadTest07/ a  \
               /store/relval/            a  \
               /store/temp/              a  \
               /store/unmerged/          a

t readcmsdata  /store/                   lr

# cmsprod and PhEDEx have full access to managed CMS data, and read for CMS
u cmsprod   readcmsdata writecmsdata
u cmsphedex readcmsdata writecmsdata

# lcgadmin can write into /store/user/sam.
u lcgadmin readcmsdata /store/user/sam/ a

# CMS users have full access to their own directory, and read for CMS
# While xrootd allows the user to *attempt* any operation - even in other user's
# home directories - the underlying filesystem also has its internal permissions and will further
# limit things.
g /cms /store/user a /store/temp/ a readcmsdata


  • This guide assumes that you have an underlying filesystem (HDFS, Lustre, etc) implementing filesystem permissions for /store/user directories. If not (pure Xrootd, files are written as the same user), you will need one line per CMS username.
  • List authorizations from most specific to least specific.
  • The first two columns must be unique; if multiple authorizations are needed for a user, add them to the same line.
  • t is a template
  • u lines are for users as mapped by LCMAPS (such as cmsprod). These should correspond to Unix usernames at your site.
  • g lines refer to VOMS groups (such as /cms). These do not correspond to Unix group names at your site.
  • If you have u *, recall this allows ALL users, including unauthenticated. This includes random web spiders!

The upstream documentation has further information on the Authfile format.

CMS-specific changes

  • Update storage.xml to have a davs protocol. Add a line that looks like this:
      <lfn-to-pfn protocol="davs"  destination-match=".*" chain="direct" path-match="(.*)" result="davs://xrootd-local.unl.edu:1094/$1"/>
  • Update the site-local-config.xml to point the stageout protocol to davs. Inside the <local-stage-out> stanza, convert the catalog protocol to davs. Example:
       <catalog url="trivialcatalog_file://etc/cvmfs/SITECONF/PhEDEx/storage.xml?protocol=davs"/>

It is best to test this out on a few worker nodes before deploying widely.

Testing the configuration

From the terminal, generate a CMS VOMS proxy and attempt to use davix-get to copy from your XRootD host:

davix-put /tmp/hello_world.txt https://cmstest2.rcac.purdue.edu:1094/store/user/goughes/test.root -E /tmp/x509up_u28772 --capath /etc/grid-security/certificates
Do not use curl as it does not support proxy certificates.

Enable Third-Party-Copy

The OSG has third-party-copy support for XRootD; this section walks through enabling the support.

Install RPMs

You will need the XRootD 4.8.3 release and some new XRootD plug-ins.

For the new plugins:

yum install --enablerepo=osg-contrib xrootd-tpc xrootd-scitokens scitokens

Verify you pull in python2-scitokens > 1.1.0; older versions miss configuration options mentioned here.

If you are a HDFS site, you will also want to do the following:

yum install xrootd-hdfs

Verify you have version 2.0.2 or later; this is only in OSG 3.4.

Configuration Changse

If you have a line like this in your XRootD config, remove it:

xrootd.fslib /usr/lib64/libXrdOfs.so

This is no longer necessary and breaks the configuration for the third party copy daemon.

Add the following lines for the xrootd daemon:

if exec xrootd
# Enable third-party-copy
http.exthandler xrdtpc libXrdHttpTPC.so

# Pass the bearer token to the Xrootd authorization framework.
http.header2cgi Authorization authz

# Enable SciTokens-based mappings; if no token is present, then the GSI certificate will be used.
ofs.authlib libXrdAccSciTokens.so

Create scitokens.cfg

Create /etc/xrootd/scitokens.cfg with the following contents:

[Issuer CMS]

issuer = https://scitokens.org/cms
base_path = /
map_subject = False
default_user = cmsprod

Adjust base_path to be wherever /store lives inside your HDFS. The issuer enabled at https://scitokens.org/cms will be able to issue tokens for anything inside that base path. Additionally, the default_user should be set to the username that should own all CMS files written via HTTP.

At Nebraska, all files are written inside /user/uscms01/pnfs/unl.edu/data4/store, so we have set base_path = /user/uscms01/pnfs/unl.edu/data4/

Testing Setup

To test the SciTokens support:

  • Install x509-scitokens-issuer-client via yum. It's currently in the HCC repo or available in source form at https://github.com/scitokens/x509-scitokens-issuer (it's working its way into the osg-contrib to make this step easier).
  • Run =x509-scitoken-init https://scitokens.org/cms= with a CMS VOMS proxy in your environment (do not use a host certificate). Should print a big nasty token to stdout. Copy that.
  • Try to curl something with the token from the previous step set in the Authorization header. Example:

curl --capath /etc/grid-security/certificates -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjYzMjUxYWFmMjM2OWQ2Njc3ZjYwZmFhZDY0ZDJjYTcwMzFjZjNlYTVkMDYxM2M5YmU3OGZlYWYyZGVkNWM0NzcifQ.eyJzdWIiOiJiYm9ja2VsbSIsImV4cCI6MTUxODIxMjU1MywiaXNzIjoiaHR0cHM6Ly9zY2l0b2tlbnMub3JnL2NtcyIsImlhdCI6MTUxODIwODk1Mywic2NwIjpbInJlYWQ6L3N0b3JlIiwid3JpdGU6L3N0b3JlL3VzZXIvYmJvY2tlbG0iXSwibmJmIjoxNTE4MjA4OTUzfQ.mpwWPA5aWwWNkxhAUMAs0Ytzcfw7VJvu7TQVygRSiUR8xfYPJ9COBre1ZdKmhcOBIB_gxXddd8Wf6I07hOoDivRQrQD0bwOf26M1_ux0siy8N2-9IEuH7lL-R8e24ydco_r75u4gBt7u_bbAuLizZQMR-p6jZuPJ1mGczI4fUUB1XwR3Cb97QF4ejQUi8wHVgzU4bk0Hl9HMnmtdAG99mwD6-stzYk1glBwf6tEDd_7DLgOd3yVYdPymcbDRl41yA62AHYOnfa1KQyX9fJFvUTnGkKxiG94C-0457w9T2Kffo2CAZtrYntj6CIMmktg5NtDaR57xuS_BSv58btlfrQ' https://transfer-3.ultralight.org:1094//store/test/docker.json > /dev/null

For third-party-copy support: you may utilize the FTS server at https://fts3-devel.cern.ch:8446.

fts-transfer-submit --overwrite -s https://fts3-devel.cern.ch:8446 \
    https://xrootd-local.unl.edu:1094//user/uscms01/pnfs/unl.edu/data4/cms/store/mc/RunIISpring18wmLHEGS/TTJets_DiLept_TuneCP5_13TeV-madgraphMLM-pythia8/GEN-SIM/100X_upgrade2018_realistic_v10-v1/100000/FC337F4C-BB22-E811-B941-0242AC130002.root \
    davs://transfer-8.hep.caltech.edu:1094//store/user/bbockelm/test/writes_new/scitokens.8 \
    --file-metadata '{"source-issuer": "https://scitokens.org/cms", "dest-issuer": "https://scitokens.org/cms"}' \
    --blocking --verbose

PhEDEx Changes

Change your storage.xml to provide a mapping for the davs protocol:

  <lfn-to-pfn protocol="davs"  destination-match=".*" chain="direct" path-match=".*/LoadTest07_Nebraska_(.*)_.*_.*" result="davs://xrootd-local.unl.edu:1094/user/uscms01/pnfs/unl\.edu/data4/cms/(.*)/$1"/>

Update as appropriate for your site. Remember to synchronize this with CVMFS and any other place you store your storage.xml.

Next, startup a dedicated transfer agent in the Debug instance:

### AGENT LABEL=download-davs PROGRAM=Toolkit/Transfer/FileDownload
 -db              ${PHEDEX_DBPARAM}
 -nodes           ${PHEDEX_NODE}
 -delete          ${PHEDEX_CONF}/FileDownloadDelete
 -validate        ${PHEDEX_CONF}/FileDownloadVerify,-d
 -accept          'T2_US_Nebraska'
 -backend         FTS3
 -service       https://fts3-devel.cern.ch:8446
 -protocols       davs
 -batch-files     10
 -max-active-files 30
 -jobs            3

Make sure your primary download agent ignores the same link so there aren't two agents attempting transfers on the same link.

Patch PhEDEx with the following: https://patch-diff.githubusercontent.com/raw/dmwm/PHEDEX/pull/1114.patch. We are working to get this into the 4.2.3 release.

Experimental Macaroons Support

Macaroons are a token format that allows delegation of authorization and attenuation. They are a convenient way to enable a user to upload or download a single file without having a grid proxy.

Install the plugin

Install xrootd-macaroons RPM via yum. At the time of writing (12 July 2018), this version is still in the process of getting in the osg-contrib repo and can be found in http://t2.unl.edu/store/repos/hcc/7/hcc-el7/x86_64/ as an alternate source.

The current version of xrootd-macaroons is 0.3.


Add the following lines to your xrootd configuration:

http.exthandler xrdmacaroons libXrdMacaroons.so
macaroons.secretkey /etc/xrootd/macaroon-secret
ofs.authlib libXrdMacaroons.so libXrdAccSciTokens.so

The secret key is a symmetric key necessary to verify macaroons; the same key must be deployed to all XRootD servers in your cluster (so puppetize its distribution).

The secret key must be base64-encoded. The most straightforward way to generate this is the following:

openssl rand -base64 -out /etc/xrootd/macaroon-secret 64

NOTE: The current implementation is sensitive to errant newline characters. Use the openssl command above and try to avoid editing with a text editor.


To generate a macaroon for personal use, you can run:

macaroon-init https://host.example.com//path/to/directory/ --validity 60 --activity DOWNLOAD,UPLOAD

(the macaroon-init CLI can be found as part of the x509-scitokens-issuer-client package). This will generate a macaroon with 60 minutes of validity that has upload and download access to the path specified at /path/to/directory, provided that your X509 identity has that access.

The output will look like the following:

Querying https://host.example.com//path/to/directory/ for new token.
Successfully generated a new token:

The contents of the macaroon key is your new security token. Anyone you share it with will be able to read and write from the same path. You can use this token as a bearer token for HTTPS authorization. For example, it can authorize the following transfer:

curl -v
Edit | Attach | Watch | Print version | History: r23 | r15 < r14 < r13 < r12 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r13 - 2018-07-24 - BrianBockelman
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback