This document describes how to set up a site for CMS data transfers. The intended audience is site systems admins that need to import or export data through PhEDEx.

Necessary Reading

If you are a lazy sysadmin who can't be bothered to read all the installation docs, here's the minimal reading list:

  1. Pre-Install
  2. Software Install
  3. Create proxy certificate
  4. Register your PhEDEx node
  5. Quick Agent Configuration

If you're lucky enough that a minimal install will work for your site (and it should for many T2 and T3 sites), you can stop here. However, PhEDEx is a necessarily complex piece of software, so you'll probably have to troubleshoot problems anyway. This guide will help you monitor data transfers.

If you need to run FTS transfers, additionally read this page


To use PhEDEx to transfer files you need:


  1. Storage element, preferably supporting the SRM interface. The common choices are dCache, Castor, or DPM. If you are a small site and only need to download files, you may be able to operate with only a large amount of disks mounted onto your PhEDEx machine; this will not achieve optimal performance by any metric.
  2. Machine for running the agents.


The PhEDEx documentation will cover installing the software in this section.
  1. PhEDEx agents
  2. Transfer tools
  3. Certificate management
  4. Site-specific scripts

Once you have configured your PhEDEx machine, you will need to register your site in the transfer topology.

Getting the hardware

To act as a transfer node in the PhEDEx network, you will need a storage system. This can be either local or mounted disks, a managed storage system (Castor, dCache, DPM...), and may or may not be backed up by a tape mass storage system. Although you may be able to set up special agreements for transfer protocols used between sites - PhEDEx is protocol independent - by default, the following is required:

  • FTS must be used when one of the endpoints is an LCG site.
  • SRM may be used when both sides are in the OSG.

It is important that you have high-speed network connection to your disk pool. Setting this up is beyond the scope of this document, but it is advisable there to be a gridftp network access path that bypasses firewalls.

You will also need a computer on which you will run the agents. This needs to be an SL6/x86_64 system. You need access to local or mounted disk space on the machine; AFS will not do. You do not need large amounts of disk space, network bandwidth, or CPU capacity on this machine. If you are using a client which uses up a large amount of memory per process (such as srmcp) and would like to achieve several Gbps transfer rates, you might need a significant amount of memory on the machine.

For LCG sites, you may also utilize a VO box for CMS at your site. It will be configured as a LCG UI plus VO box specific services: gsissh and proxy-renewal. Please read this link for more details.

PhEDEx Agent Installation


While a standalone box is not required, a lot of sites have chosen to dedicate one to PhEDEx. PhEDEx is designed to be run as its own user; usually sites create a user called "phedex". When upgrades are announced for major releases, it is usually recommended to completely reinstall rather than attempt an upgrade.

In this manual, we will make the following assumptions:

  • PhEDEx is installed on
  • The phedex user is "phedex".
  • The site name is TX_FOO_Buffer.

Before installing the software, you should create a few empty directories:

  cd /home/phedex
  mkdir -p state logs gridcert
  chmod 700 gridcert

The state directory is where agents keep their working states; logs is where logs are kept for each agent; sw is the software install directory; and gridcert is where you should keep your grid proxies.

You can configure your PhEDEx box to use the software deployed on CVMFS, or install the software locally. Both options are covered in the following subsections.

Using software installed on CVMFS

Starting with the release of PhEDEx 4.2.0, PhEDEx is automatically deployed on CVMFS. You can simply point your sw installation directory to cvfms:

export sw=/cvmfs/
ln -s $sw sw

Set variables for the current architecture and version:

myarch=slc6_amd64_gcc493 # for 4.2.X on SLC6
version=4.2.1  # for 4.2.1 on SLC6

FInally create a symbolic link to the current version:

rm -f PHEDEX; ln -s $sw/$myarch/cms/PHEDEX/$version PHEDEX

In case of upgrade, you just need to repeat this procedure to update the PHEDEX symlink.

Local Software Install

Software installation is done through the cmspkg program, which is installed running the bootstrap script. The entire process should take around 5 minutes. Please check the prerequisites for CMS software installation.

First set a variable for your architecture. You will need a 64-bit, RHEL-6 derivative and you can install PHEDEX_4_2_1 or newer.

myarch=slc6_amd64_gcc493 # for 4.2.X on SLC6
repo=comp # for 4.2.1 on SLC6

Then create a directory for software installation:

mkdir sw
export sw=$PWD/sw

Now configure the CMS software area with the following commands:

wget -O $sw/
sh -x $sw/ setup -path $sw -arch $myarch -repository $repo 2>&1|tee $sw/bootstrap_$myarch.log
$sw/common/cmspkg -a $myarch update
$sw/common/cmspkg -a $myarch search PHEDEX|grep PHEDEX+

Then set a variable for the version of PhEDEx you are going to install. Please use the latest formal release (e.g. not a pre-release) unless you know what you are doing. The list of available versions is found in the cmspkg search output:

version=4.2.1  # for 4.2.1 on SLC6

Finally install PhEDEx:

$sw/common/cmspkg -a $myarch install cms+PHEDEX+$version
rm -f PHEDEX; ln -s $sw/$myarch/cms/PHEDEX/$version PHEDEX

Local Software Update

Updating an existing software installation does not require you repeat all the steps above. Simply set the variables $sw, $repo, $myarch, and $version as described above:

export sw=(insert path to your sw directory here)

2016-08-23 IMPORTANT NOTE: With the release of PhEDEx 4.2.0, PhEDEx RPMs are now installed with cmspkg. If you have an existing sw area bootstrapped with apt, you can use the following procedure to update your existing area After the bootstrap, apply the rest of the standard upgrade instructions below.

# Only if upgrading from PHEDEX_4_1_X or below to PHEDEX_4_2_X or above.
wget -O $sw/
sh -x $sw/ setup -path $sw -arch $myarch -repository $repo 2>&1|tee $sw/bootstrap_$myarch.log

Then simply run:

$sw/common/cmspkg -a $myarch update
$sw/common/cmspkg -a $myarch search PHEDEX|grep PHEDEX+
$sw/common/cmspkg -a $myarch install cms+PHEDEX+$version
rm -f PHEDEX; ln -s $sw/$myarch/cms/PHEDEX/$version PHEDEX
# Then remember to edit your PhEDEx configuration files increasing the value of the PHEDEX_VERSION env variable to PHEDEX_VERSION=$version;

Download Configuration Templates

Once the software has been installed, one must configure it. As PhEDEx is designed to run in many environments (T0 CERN, T1 sites with tape backends, T2 sites with disk only, or even T3 sites with no storage element), there are a wide range of configurations it must support; creating your own from scratch may be quite overwhelming.

For this reason, all of the site configurations are kept in the gitlab repository. You will want to start from the template, adjust it for your site, then commit your own changes to SITECONF. As "Site Admin" for TX_Y_Z in SiteDB, you should have permission to push config files to the SITECONF/TX_Y_Z project in gitlab (the directory must contain at least one file in order for the commit to do someting). See here for more about gitlab:

The following commands check out your SITECONF directory:

  cd /home/phedex
  mkdir SITECONF
  # requires ssh key to be set up properly, see docs
  git clone ssh://<name_of_site>.git 
  # update files using your prefered editor, for instance
  vi <name_of_site>/PhEDEx/a_new_file.txt
  # remember that any new files need to be added explicitly, i.e.
  git add <name_of_site>/PhEDEx/a_new_file.txt 
  git commit -m "<comment describing the update>" 
  git push 

The templates are available in ~/PHEDEX/Custom/Template. At a minimum, you will want to copy and edit these files, placing them in the ~/SITECONF/TX_Y_Z/PhEDEx/ directory:

  • Config
  • ConfigPart.Common
  • ConfigPart.Export
  • ConfigPart.FTSDownload
  • ConfigPart.Verify
  • ConfigPart.Watchdog
  • FileDownloadGFALVerify
  • FileDownloadGFALDelete
  • ProxyRenew

If you do not have a storage.xml file, you will also want to copy that one over.

Edit the files customizing them to your site needs!

Agent configuration is covered here.

Post-Install Sanity Check

The RPMs include part of the PhEDEx environment setup, while the config files written above will provide the rest. To test your RPM install, try the following command:

  source $sw/$myarch/cms/PHEDEX/$version/etc/profile.d/

You should verify the following environment variables are set:

  • PATH
  • X509_USER_CERT
  • X509_USER_KEY

Ensure the following commands exist and work:

  • A transfer client such as gfal-copy (from LCG) or lcg-cp (from LCG) or srmcp (from OSG/LCG)
    • WARNING: srmcp included in dcache-srmclient-2.10.7 has known issues, it is recommended to use an older version of the client until further notice
  • voms-proxy-init (from OSG/LCG)
  • voms-proxy-info (from OSG/LCG)
  • myproxy-init (from OSG/LCG)
  • myproxy-get-delegation (from OSG/LCG)
  • glite-transfer-submit (from LCG)
    • Currently (as of 2016-05-23) the required version of the FTS client rpm is fts2-client-2.2.9-1 (from EMI-3 UI, for EGI sites) or glite-fts-client-3.7.4-9.osg32 (included in OSG3.2, for OSG sites).
  • fts-transfer-submit (from LCG)
    • Currently (as of 2017-04-12) the required version of the FTS3 client rpm is fts-client 3.6.8 (recommended, from EPEL) or fts-client 3.5.7 (known to work, on cvmfs)

You will need to make sure all these commands are either in the user's environment (i.e., place the appropriate setup scripts in ~/.bashrc) or are included in the environment session in ConfigPart.Common (see agent config page).

To set up the environment that each download agent will use, try the following command:

cd $sw/..
eval `PHEDEX/Utilities/Master -config ~/SITECONF/TX_Y_Z/PhEDEx/Config environ`

It is recommended that you hand-test gfal-copy/lcg-cp/srmcp downloads from another site, such as your local T1. Make sure you use the above command so your tests will use the same environment as the download agent will.

Certificate Management

To run the transfer agents, you need to have a grid certficate and be registered to the CMS VO. Note that LCG policy states that all file transfers must be done with personal certificates, not service certificates.

(If you are in a hurry to set things up or to try things out, you may wish to skip this section and come back to the certificate management once you have all the rest running fine. Just create a normal grid proxy certificate and off you go.)

There are three recommended configurations:

  • Long-Term Proxy. Easiest.
  • MyProxy. Recommended, but may be difficult to get working.
  • VOBox. For LCG site only.
  • Service Certificate. Only recommended for sites which do not need to interact with the LCG.

Long-Term Proxy

This is the easiest option. Set up the grid UI environment on a computer that has your grid certificate installed, and create the proxy with this command:

voms-proxy-init -voms cms -valid 192:00

The created proxy will be in the following location:

/tmp/x509up_u`id -u`

Here, `id -u` should evaluate to your user ID. Copy the above proxy certificate into the following file:


The created proxy will be valid for 192 hours, or 8 days. So, every 8 days, you must repeat this process.

Management with MyProxy

We recommend obtaining a service or host certificate, loading a long-lived personal proxy into myproxy, then using the service certificate to renew a short-lived proxy which is then used by the transfers. This is not easy to get right, mainly because myproxy generally gives useless error messages if anything happens to go wrong, but it does reduce operational burden considerably.

If there will be only one person administering the agents at your site, it's simplest to use your personal certificate. We recommend following these conventions (X509_USER_PROXY is /home/phedex/gridcert/proxy.cert):

  1. Once a month, load a proxy certificate to myproxy service
            myproxy-init -c 720
  2. Extract the proxy into the certificate directory
            cp /tmp/x509up_u$(id -u) $X509_USER_PROXY
  3. Make sure the $X509_USER_PROXY is set in the environment when you start transfer agents. Make sure $X509_USER_KEY and $X509_USER_CERT are not set.
  4. Periodically (e.g. every four hours in cron job) update the proxy:
            myproxy-get-delegation -a $X509_USER_PROXY -o $X509_USER_PROXY --voms cms

For using a service certificate the instructions are similar, but slightly more complicated because the certificates must be properly protected. First of all, you need to obtain a service certificate, e.g. for phedex/ Then obtain a unix group with all service administrators as members. Change directory /home/phedex/gridcert to this group and make it group-writeable. Copy the service hostcert.pem and hostkey.pem certificate files into this directory.

Then the administrator starting the service should:

  1. Once a week, load a proxy certificate to myproxy service
            myproxy-init -l foo_phedex -R "phedex/" -c 720
  2. As yourself, extract the proxy into the certificate directory
            mv /tmp/x509up_u$(id -u) /home/phedex/gridcert/
            chmod 660 /home/phedex/gridcert/x509*

The phedex admin account should run something like ProxyRenew once an hour as a cron job.

More details about the management of VOMS proxies with myproxy found here.


An alternative is asking grid admins at your site to set up a VO box. Proxy renewal will come "for free" with properly installed VO box. More details found here.

Service Certificate

Not all sites have restrictions on downloads using service certificates. Get a service certificate for phedex/, and place the certificate and key file in /home/phedex/gridcert. Once placed, this will not require renewal until the certificate itself expires (after one year).

If using srmcp, you will need to alter your configuration to force it to use the service certificate instead of a proxy. You will want to use the following options:

  • -x509_user_cert=/home/phedex/gridcert/servicecert.pem
  • -x509_user_key=/home/phedex/gridcert/servicekey.pem
  • -use_proxy=false

PhEDEx Configuration

Registering your Node

To be able to transfer any files in our out, your site must become part of the CMS transfer topology. Please review the necessary steps outlined in Setting up a brand new site.

Access to the PhEDEx database from outside CERN is firewalled, and your hosts will need to be granted access. Please send a mail to Physics-Database.Support AT and cms-phedex-admins AT asking for your host(s) to be allowed to connect. You should give the name of the domain or subnet that needs access. Ideally this domain/subnet should not contain too many machines, but it should be open enough that you can change hosts without having to repeatedly ask for new holes in the firewall.

This is a one-time thing, unless you keep changing subnets etc.

Agent Configuration

CMS encourages all sites to keep the configurations of their CMS services in the SITECONF area in the gitlab repository. If you do not have a SITECONF directory for your site, please request one from the CMS Site Support Team.

Agent configuration is covered in a separate page. The agent configuration page goes into much more detail than this one. Read the Quick Configuration section if you are impatient.

PhEDEx uses simple text configuration files that describes the agents that should run for your site. The purpose of the configuration files is to make the agent management easy, and to capture a self-contained environment including everything required to run the agents. There is usual one "master" file called Config which sets up some basic environment information, then imports other partial config files. The partial config files (such as ConfigPart.FTSDownload, ConfigPart.Export, ConfigPart.Common, etc) each describe the setup of one or two agents. By adding or removing IMPORT lines in the Config file, you can edit the list of agents which are run by default.

If you follow the template config files, you usually only have to edit Config and one or two lines in the download files.

All nodes should run three agents: FileDownload, FileExport and FileRemove. Sites with a tape storage (PhEDEx MSS node) also need to run tape migration agents such as File{Castor,DCache}Migrate and a stager agent such as File{Castor,DCache}Stager. You can even use !FileDownload with suitable options to copy between local storage nodes, for example if you have separate disk and tape storage systems.

Other activities, including synchronisation with DBS and DLS and routing files, are controlled by the CERN agents, and you will not need to set these up.

Optional Custom Scripts

Some sites will require customized scripts; most sites won't. You would normally have:

  1. For the file FileDownload agent:.
    • FileDownloadVerify to validate downloaded file.
    • FileDownloadDelete to clean up failed downloads. Both of these scripts may be taken from the PHEDEX/Custom/Templates directory, usually without modification.
  2. Archive all your logs to a safe place. See AFSCrontab and LogArchive in SITECONF/T1_CH_CERN/PhEDEx.
  3. Renew your proxy. See VomsProxyRenew in SITECONF/T1_CH_CERN/PhEDEx.


There are several places where you can go for support. PhEDEx has an active community of smart sysadmins who will be able to help you out when you are stuck.

  • This wiki should begin to provide answers to a lot of questions; browse around and see if your topic is covered.
  • The PhEDEx HyperNews forum is very active. The web link is here:
    • PhEDEx Developers HN. For development discussions.
    • PhEDEx Operations HN. For PhEDEx operations questions and problems. We encourage each site to have at least one person subscribed to the HyperNews forum, as important information is often first announced there.
  • PhEDEx Issues Page on github. For feature requests and bug reports.

Responsible: NataliaRatnikova
Edit | Attach | Watch | Print version | History: r76 < r75 < r74 < r73 < r72 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r76 - 2017-11-28 - NataliaRatnikova
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback