FAX for Posix storage sites

Hardware and OS

You need a machine with a fast network connection (10Gbps or faster recommended). This machine should allow inbound traffic to xrootd port 1094, allow unrestricted outbound traffic, have the site storage mounted as a Posix file system, and run SL5/6 x86_64.

RPM Installation

Set up YUM repositories

Please enable the following YUM repositories

  • The xrootd.org repo
The repo file can be found at
Please save them to /etc/yum.repos.d. FAX only supports Xrootd rpms from xrootd.org repo. Note EPEL and OSG repos also come with their own xrootd rpms, which should not be used. To exclude xrootd rpms from EPEL and/or OSG repos, please add
to the EPEL and/or OSG repo files.

  • Other required repositories:
Some of the packages (i.e. xrootd N2N rpm and xrootd x509 security rpm) come from the WLCG repo (see http://linuxsoft.cern.ch/wlcg/). These packages have automatic dependencies on other packages in the EPEL repo and the EGI repo (or the OSG repo if you are an OSG site).

Install the RPMs

  • Install or upgrade Xrootd rpms
yum install --disablerepo="*" --enablerepo=xrootd-stable xrootd. This will also install user "xrootd" and group "xrootd" if they don't already exist. Make sure you install Xrootd 4.2.0 or above.

If you already have pre-4.1.1 installed, you may need to uninstall all xrootd rpms before you upgrade to 4.2.0 or later (due to RPM layout change). In that case, be sure to save the two important configuration files (/etc/xrootd/xrootd-clustered.cfg and /etc/sysconfig/xrootd) before removing the old rpms.

  • Install FAX name-to-name translation rpm
yum install xrootd-server-atlas-n2n-plugin. If you previous installed a version like xrootd-server-atlas-n2n-plugin-20131014. Please uninstall the rpm and then run the above yum command. The recent rpm (version 2.0-1 or later) requires the json-c rpm and wget rpm.

  • Install VOMS authorization rpm
yum install vomsxrd. This rpm requires voms-2.0.6 or above, which is available from EGI repo and OSG repo.

  • If you just upgraded Xrootd rpms

Please also do yum update xrootd-server-atlas-n2n-plugin vomsxrd. This will make sure that you have the correct version of those plugins that work with the upgraded Xrootd rpms.

From time to time, you may need to check or change the following files for configuration changes or debugging:

  • /etc/xrootd/xrootd-clustered.cfg : Main Xrootd configuration file
  • /etc/sysconfig/xrootd : System level configuration to set up runtime environment for Xrootd
  • /var/log/xrootd/xrootd.log : xrootd log file
  • /var/log/xrootd/cmsd.log : cmsd log file


EGI or OSG environment

Make sure a standard EGI or OSG environment is installed, including
  • host certificate (default /etc/grid-security/{hostcert.pem, hostkey.pem})
  • CA certificates (default /etc/grid-security/certificates)
  • voms certificates (default /etc/grid-security/vomsdir)

System level configuration

/etc/sysconfig/xrootd is a shell script that setup the runtime environment for xrootd and cmsd.
  • Adjust XROOTD_USER and XROOTD_GROUP if needed. This is the owner that will run xrootd and cmsd processes.
  • Add the following lines to the end of this file
export MALLOC_ARENA_MAX=4 # recomend for SL6

X509_CERT_DIR and X509_VOMS_DIR can also be defined in this file if they are in non-default locations.

Also make sure the limits on nproc and nofile in /etc/security/limits.d/* are set to 4096 and 16384, respectively, or higher for the XROOTD_USER. By default SLC6 includes in /etc/security/limits.d/90-nproc.conf a limit of 1024, which is too small.

Xrootd configuration

/etc/xrootd/xrootd-clustered.cfg is the main xrootd configuration file. The following is a template. The first few lines are site specific. They are explained after this template file:
# site parameters: please change for your site
all.manager redirector+:cmsd_port
xrootd.redirect redirector:xrootd_port ? /atlas
all.sitename SITENAME

# A new initiative from ADC asking sites to export their physical path via xroot protocol. For example, /pnfs/abc.xyz.com/atlasstorage
all.export /path_to_atlasdatadisk r/o
all.export /path_to_atlasscratchdisk  r/o
all.export /path_to_atlas_whatever_disk r/o

# Do not change the lines below
all.export /atlas r/o
all.role server
all.adminpath /var/run/xrootd
all.pidpath /var/run/xrootd
xrootd.async off

# Monitoring 
if exec xrootd
    xrd.report uct2-int.mwt2.org:9931 every 60s all -buff -poll sync
xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest redir fstat info user uct2-int.mwt2.org:9930 dest redir fstat info user atlas-fax-eu-collector.cern.ch:9330

# N2N configuration. Please change for your site
oss.namelib /usr/lib64/XrdOucName2NameLFC.so

# X509 configuration, change nothing
xrootd.seclib /usr/lib64/libXrdSec.so
sec.protparm gsi -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:certfmt=raw|vos=atlas|grps=/atlas
sec.protocol /usr/lib64 gsi -ca:1 -crl:3 -gridmap:/dev/null
acc.authdb /etc/xrootd/auth_file
acc.authrefresh 60

# Enable remote debugging
#xrootd.diglib * /etc/xrootd/digauth.cf

Other configurations

  • Copy your /etc/grid-security/{hostcert.pem, hostkey.pem} to /etc/grid-security/xrd/{xrdcert.pem, xrdkey.pem}. Make sure the owner and group of xrdcert.pem and xrdkey.pem are identical to the XROOTD_USER and XROOTD_GROUP environment variables defined in /etc/sysconfig/xrootd. Make sure xrdkey.pem has permission 400 (can not be 600).
  • Run this command to set up the authorization file: echo "g /atlas /atlas rl /your_storage_physical_path rl" > /etc/xrootd/auth_file
  • Run this command (and uncomment the "xrootd.diglib" line above) to enable the FAX support team to collect debugging info (log, etc) from a remote host: echo "all allow host h=atlint04.slac.stanford.edu g=/atlas" > /etc/xrootd/digauth.cf.

Start/Stop Services

  • Execute once:
service xrootd setup

  • Start/stop xrootd and cmsd services:
# Start
service xrootd start
service cmsd start
# Stop
service xrootd stop
service cmsd stop

Major updates:
-- WeiYang - 10 Nov 2013

Responsible: WeiYang
Last reviewed by: Andrew Hanushevsky - 12 Mar 2014

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2015-02-01 - WeiYang
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Atlas All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback