NEW (24 Jun 2013) CVMFS 2.1 is ready for production !

The focus of this page is on the CVMFS area.

This page is divided into several main sections, among these:

  • The first deals with the client setup which is of relevance to users and sites.
  • The second section deals with the server and this is for experts - sites and users do not need to know anything about the server setup.

CVMFS Client Setup at ATLAS Sites

This section is what users and sites need to do to use CVMFS.

IDEA! Instructions for installing the CVMFS client can be found on the ATLAS page: ATLAS CVMFS client installation instructions (and in the official CVMFS documentation).

Keep informed

Setup Instructions for LCG Grid Sites

  • Install and configure CVMFS on WNs:
    • Install cvmfs 2.1.X (almost all use cases, please do this)
    • For nfs exported cvmfs, install cvmfs 2.1.X (This is not recommended except as a last resort)
  • Define the following environment variables:
    • VO_ATLAS_SW_DIR=/cvmfs/
    • ATLAS_LOCAL_AREA=<some nfs area seen by WNs> where ATLAS_LOCAL_AREA must be writeable by the atlassgm account which does validation of releases.
  • To define VO_ATLAS_SW_DIR, you may want to edit your gLite site_info.def file and rerun yaim.
  • To define ATLAS_LOCAL_AREA,
    • you may define it in a script inside /etc/profile.d
    • or if you install EMI-WN rpms:
      • Fetch the files from and put them in these directories:
        • /opt/glite/yaim/node-info.d/localenv
        • /opt/glite/yaim/functions/config_localenv
      • define ATLAS_LOCAL_AREA in your site-info.def file
      • configure with the yaim option "-n LOCALENV"
    • or if you install EMI-WN tarballs, and do not run yaim, simply add it to the setup*.sh scripts.
  • Setup your squid server ( T2SquidDeployment#Installation_Instructions ). For this activity, the squid servers should point to the public replicas (CERN, BNL, RAL)
  • Contact to get the releases validated at the site (software validation monitoring)
  • If needed, copy or create a local override file as $ATLAS_LOCAL_AREA/ (Note: CDB PFC will be used from cvmfs so you do not need to define the Pool File Catalog area.)

Setup Instructions for OSG Grid Sites

These instructions assume that the site is currently operational with an NFS distribution of the Atlas releases.

The site manager should synchronize the date of the switch over with Alessandro De Salvo prior to starting these steps.

It is highly recommended that all Panda queues be set offline and all worker nodes drained of any jobs before starting the switch over.

  • Install/configure autofs and fuse on all worker nodes. See for an example.

  • Install the current release of CVMFS on all worker nodes. See for an example.

  • Modify the CVMFS configuration files. See for an example.
    Items you will need to change are your local squid, backup squids, location of the local caches and the sizes

  • Change $OSG_APP/atlas_app/atlas_rel to point at the CVMFS atlas repository
    # mv $OSG_APP/atlas_app/atlas_rel $OSG_APP/atlas_app/atlas_rel.nfs
    # ln -s /cvmfs/ $OSG_APP/atlas_app/atlas_rel

  • Create an area for local setup files that are NFS available to all worker nodes. A good choice is within $OSG_APP
    # mkdir -p $OSG_APP/atlas_app/local
    # chown usatlas2:usatlas $OSG_APP/atlas_app/local

  • Put a copy of your file into this area.
    The wrapper/pilot will execute the setup scripts in the following order
       source /cvmfs/
       source $ATLAS_LOCAL_AREA/
       source $ATLAS_LOCAL_AREA/

  • Add the following to your CE/config.ini file:
    [Local Settings]

  • Execute configure-osg -c to make the changes effective. These variables will be added to osg-local-job-environment.conf. You could also add them by hand to osg-local-job-environment.conf to avoid having to run configure-osg -c on a running site. All jobs submitted to the site via WMS will have these variables defined, so this is a site wide change. All Panda queues will be affected immediately.

  • Remove unneeded schedconfig vars that might cause confusion as to where the releases are located
    appdir = None

  • Validate your site installation. This can take 6 or more hours to complete
    * Remove all BDII defintiions by cleaning out all VO-atlas tags in $OSG_APP/etc/grid3-locations.txt
    * Allow CEMON time to remove all tags in the BDII (about 10 minutes)
    * Notify Alessandro De Salvo that you would like a full site validation
    * A full site validation will repopulate grid3-locations.txt and publish the available releases in the BDII

  • Choose where you would like jobs to find the conditions files
    Currently it is not possible to use condtions files from HOTDISK with CVMFS for most Tier2 sites
    This is due to a problem of short vs long form SURLS in the LFC for a given Tier2
    Please do not attempt the following until LJSFi has been updated to handle mixed SURLs at a site.
    By default, the conditions files will be found on the CVMFS repository   
    To use the conditions files from the local HOTDISK, add the following line to $ATLAS_LOCAL_AREA/
    When Alessandro's PoolCondPFC task executes on the site,  appropriate changes/additions will be made 
    to files in $ATLAS_LOCAL_AREA. It also creates a local PFC using entries from HOTDISK located at 

Setup Instructions for Tier3 Sites

User Interface part for Tier3s

Most Tier3s will need an interactive component and this is handled by manageTier3SW which will automatically recognize CVMFS directories for Athena, DBReleases and Condition Pool Files.

Or, if your site is uniformly 64-bit nodes, you can do the following to use manageTier3SW from CVMFS; simply do:

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/ and follow the instructions for local overrides

Users should see the ATLASLocalRootBase instructions.

Additional Step for Tier3 sites which are like Tier2s (ie. grid enabled CE)

Do either the LCG Tier2 (non-US sites) or the OSG Tier2 (US sites) steps as outlined above.

Setup Instructions for Standalone SL6 Desktops or Laptops

Note: The OS needs to be 64-bit.

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/ and follow the instructions for local overrides

Users should see the ATLASLocalRootBase instructions.

Setup Instructions for CernVM Users

  • Download ( very small image : ~ 13MB image for VMWare Fusion)
  • Start it up. You will then need to do the next 2 steps to get the pin.
  • Go to
    • click on Marketplace link on the left.
    • clone ATLAS Development.
    • under users, add your username.
    • remove the default atlas user.
    • add your ssh key for root.
    • save it.
  • Go to
    • (You should see your cloned Context.)
    • Click on “Pair an instance of CernVM".
    • On the new page, click “Pair with this context” for the context you just cloned in the previous steps.
    • Note the PIN number.
    • Do not close this web browser and go to the next step (do not wait as it can expire) You should see this updated when you complete the next step.
  • With the PIN number of the previous step, enter it to your VM to finish th setup.
    • remember to put # before the PIN number.

Your VM should be ready for use now. Simply login and type setupATLAS

Site Fail-over

If your site squid goes down, jobs that require files that are not already cached on the WN will start failing. If you are running a batch farm with more than 100 WN and are worried about site availability, you are strongly recommended to run two squids. The performance of the machines is not important so it can be your two oldest WNs. Running two identically configured squids should not be that much more difficult than running one.

To configure your worker nodes to use this you should list your squids in the CVMFS_HTTP_PROXY variable in /etc/cvmfs/default.local separated by a "|". An example at RAL is:

This will mean that CVMFS tries one of these squids and random and will try the other one, if the first attempt fails.

It is also possible to try squids in order. You can use this to fail-over to another sites squid if your site fails. However this may cause weird job failures as it may take time to get files from the distance squid. There has been a documented case of jobs failing at CERN when some WN were accidentally mis-configured to use the RAL squids. If you are a very small site while being down while it may not be ideal for jobs to fail if your squid is down it will make it much easier to debug problems. However if two sites are co-located it could be acceptable to fail-over to each others squid. To do this you would use a ";" in the configuration:

This would mean to try your local Site Proxy first and if that failed to randomly choose one of the RAL squids to use (and if that failed to try the other RAL squid).

ATLAS CVMFS Software Server

This section explains some of the details of how to maintain each of the ATLAS CernVMFS content servers. Warning, important This is only for experts and not for sites or users.



unset VO_ATLAS_SW_DIR; --installALRB=/cvmfs/ --noCronJobs

Maintenance (for Shifters)

  1. The following is run daily as a cron jobs. This will do the update of the software as well as the CA/CRL certificates for gLite and then publish.
    Note that there is a lock that will be checked every 5 minutes to ensure that no other process is publishing at the same time.


ATLAS Conditions Database Server

The conditions database server is currently running on voatlas94. This vobox machine is accessed via lxvoadm. One has to be in the proper ATLAS egroup to get access to lxvoadm and you can only log into lxvoadm from a non lxplus machine on the CERN network.

Once you are on voatlas94, become the atcvmadm account. This unprivileged account is used to update the conditions db files and publish the results with cvmfs server code.

Maintenance (for Shifters)

  • become the atcvmadm account and go to home area
sudo su
su atcvmadm
  • The following should be run daily since there are no cron jobs (until we get them to be bullet proof). This will update the conditions db
Warning, important the log file information is in > /var/log/condb/cron-condb.log

ATLAS Nightly Release Server



Obsolete Stuff

Major updates:
-- DougBenjamin - 09-May-2011

Responsible: DougBenjamin
Last reviewed by: Never reviewed

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf TRIUMF.pdf r1 manage 286.8 K 2011-06-07 - 00:31 AsokaDeSilva  
Unix shell scriptsh r1 manage 0.4 K 2011-07-06 - 16:12 DavidLesny IllinoisHEP autofs/fuse installation/setup
Unix shell scriptsh r9 r8 r7 r6 r5 manage 4.0 K 2011-09-15 - 21:16 DavidLesny IllinoisHEP CVMFS configuration
Unix shell scriptsh r1 manage 0.3 K 2011-07-06 - 16:13 DavidLesny IllinoisHEP CVMFS installation
Edit | Attach | Watch | Print version | History: r54 < r53 < r52 < r51 < r50 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r54 - 2014-11-20 - TWikiAdminUser
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Atlas All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback