cvmfs Client Installation

Hints on updating: Please do it only on a small number of nodes and monitor it. Grid sites should monitor for "hot nodes" - for example, in Panda, you can look at http://bigpanda.cern.ch/wns/TRIUMF/ to see failures by nodes (substitute your site name). To revert, simply reinstall the old production version.

This is the client installation documentation for cvmfs in ATLAS; please also refer to the starting point ATLAS cvmfs TWiki and additional relevant site and usage information. Installation needs to be done on each machine that mounts the cvmfs volume (except as noted otherwise in the appropriate section - eg for nfs exports.)

In addition to the installation of cvmfs described here, you will need to also install required software for ATLAS; these can be found here:

or you you can run this script after cvmfs installation /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/utilities/installCheck.sh to show what is missing (it can be run as normal user).

Machine configuration

  • Create a partition /<path>/cache/cvmfs2 TIP If you use /var/lib/cvmfs then you do not need to define CVMFS_CACHE_BASE in /etc/cvmfs/default.local
  • it can be anywhere but avoid the root (/) partition.
  • space of 25-50 GB is recommend for the local caching of cvmfs <some local disk, not nfs, partition's directory>
    • (Note the partition above; in the Configure cvmfs section below, in /etc/cvmfs/default.local, "CVMFS_CACHE_BASE" should point to it.)
    • (Note the size of the partition above in MB; in the Configure cvmfs section below, in /etc/cvmfs/default.local,"CVMFS_QUOTA_LIMIT " should be equal 80% of the available space (in MB) , due to internal db files.

Install cvmfs

If you are installing at a Tier1 or Tier2 site, you are encouraged to follow the OSG or EGI instructions. These OSG/EGI instructions will configure CVMFS fallback proxies. This is important for the WLCG squid operations team and the stratum 1 operators because it keeps failovers (when local squids aren't working) from going directly to the CVMFS stratum 1s.

The OSG/EGI configuration also redirects those who set a DIRECT proxy to use Cloudflare for cvmfs stratum 1s; this is a good reason for everyone else to follow these instructions instead of the default ones.

However note that the OSG/EGI may install cvmfs versions that are older than that of the default ones.

OSG Sites

US OSG sites, please use these instructions instead: Open Science Grid CVMFS install instructions Afterwords, when you configure /etc/cvmfs/default.local, you should set this as shown in the Configure cvmfs section below in this Twiki:
  • CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,atlas-nightlies.cern.ch,sft.cern.ch,unpacked.cern.ch

EGI Sites

Please see these EGI-cvmfs instructions but note that when you configure /etc/cvmfs/default.local, you should set this as shown in the Configure cvmfs section below in this Twiki:

  • CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,atlas-nightlies.cern.ch,sft.cern.ch,unpacked.cern.ch

While you should be diligent and read the EGI-cvmfs instructions, this is a summary of what it involves.

# This is for CentOS7; if you are on SLC6, please upgrade to CentOS7.

#install umd repo
yum install http://repository.egi.eu/sw/production/umd/4/centos7/x86_64/updates/umd-release-4.1.3-1.el7.centos.noarch.rpm

#install egi cvmfs
yum install egi-cvmfs

# enable and start autofs
systemctl enable autofs
systemctl start autofs

#add cvmfs to autofs
test  ! -e /etc/auto.master.d/cvmfs.autofs && echo "/cvmfs /etc/auto.cvmfs" > /etc/auto.master.d/cvmfs.autofs

# follow the "Configure cvmfs" section of this Twiki and restart autofs

Default Installation (Users, non-OSG/EGI sites)

Fetch and install the repo information (only needed first time):

yum install https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest.noarch.rpm

Install or update cvmfs:

(Note: If you are updating, do "yum update" instead of "yum install" below).

# for Tier1 and 2 only: 
yum install cvmfs

# for Tier3s and others:
yum install cvmfs cvmfs-auto-setup

(  or if you prefer to install the versions that were tested on a Tier3 (tested on 24 Feb 2020) do instead:
# for SL6  (SL5 users should migrate to SL6)
# you really should not be using SL6 anymore.  Please migrate to centos7.
yum install cvmfs-2.7.1-1.el6 cvmfs-auto-setup-1.5-2

# for SL7 
yum install cvmfs-2.7.1-1.el7 cvmfs-auto-setup-1.5-2
)

Warning, important You are advised not to have yum update cvmfs automatically. See this post in the atlas-adc-tier3-managers eGroup on how to do this.

Warning, important For Tier1/2s, note that the cvmfs-auto-setup rpm was intended for Tier3s (please install it if you are a Tier3 !) and will configure settings automatically. It does

  • linking the cvmfs map in /etc/auto.master
  • adding user_allow_other to /etc/fuse.conf
  • starting cvmfs
You will have to manually configure these yourself if you do not install cvmfs-auto-setup.

Be informed !

  • Please subscribe to atlas-adc-tier3-managers if you are a Tier3 manager and we will advice you on when to update.
  • You may also want to subscribe to cvmfs-talk.
  • You should subscribe to atlas-adc-cvmfs which is focused on cvmfs in ATLAS.
  • If you are testing nightlies or release candidates, please subscribe to cvmfs-testing for announcements etc.

Configure cvmfs

Note that the default CVMFS_CACHE_BASE location has changed.

Edit (create if it does not exist) the file /etc/cvmfs/default.local, in particular, change

  • CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,atlas-nightlies.cern.ch,sft.cern.ch,unpacked.cern.ch
  • CVMFS_CACHE_BASE=/<path>/cache/cvmfs2 HELP Not needed if you use var/lib/cvmfs
  • CVMFS_QUOTA_LIMIT=<should be equal 80% of the available space (in MB)>

If you have your own squid server (which you should), then include that value; ie do (don't forget the quotation marks and the semicolons separating the fields if there is more than one squid):

  • CVMFS_HTTP_PROXY="< address of the squid server>"
Note the syntax:
  • CVMFS_HTTP_PROXY="localsquid1|localsquid2;remotesquid3" This "load-balancing" of local squids allows for removing them consecutively for maintenance tasks. (recommended) Note that as of version 2.1.20, if you have a DNS round robin, you can define the DNS alias instead of defining each proxy. The alias will then be automatically resolved into a load balancing group by cvmfs.
  • CVMFS_HTTP_PROXY="squid1;squid2" This will use squid2 if squid1 becomes unavailable.

If you don't have a squid server: (only be used in very small numbers at a site. Please setup a local squid otherwise.)

  • CVMFS_HTTP_PROXY="DIRECT"

At CERN, check configuration instructions here: ClientSetupCERN

A list of Stratum-1 servers is available and they will be selected based on GeoIP from version 2.1.20.

Start services

Note that there are no system services for cvmfs in version 2.1 and newer. Automount will mount it when the dir is accessed.

  • Mare sure autofs is started:
    • SLC6: service autofs start
    • CentOS7: systemctl start autofs
  • If you had changed settings in /etc/cvmfs, you should restart autofs:
    • SLC6: service autofs restart
    • CentOS7: systemctl restart autofs

  • Make sure cvmfs is a member of group fuse (fuse should be installed as part of the cvmfs installation if it's not already there. If cvmfs is not part of group fuse, or group fuse doesn't exist in /etc/groups, you will have to add it or edit it)

  • Check the setup scripts: cvmfs_config chksetup

  • Some quick tests:
    • cvmfs_config probe
    • cvmfs_config status
    • cvmfs_config stat -v

Note that init scripts have been removed; you will need to use cvmfs_config to reload etc.

checking cvmfs

* Tools for debugging:

  • cvmfs_config stat : (can be run as normal user) shows cvmfs usage information (very useful !)

  • cvmfs_config chksetup : check the configurations

  • cvmfs_config showconfig : show the configurations

  • cvmfs_config probe : check that the cvmfs mount point is working

  • cvmfs_talk : query the cvmfs process running on a given node. Changed in 2.1: not cvmfs-talk
    • You will first need to do ls /cvmfs/atlas.cern.ch or cvmfs_config probe
    • type cvmfs_talk to see the help; for example, you can type
      • cvmfs_talk cache list to see what caches are on your local disk
      • cvmfs_talk cleanup 1000 will remove the oldest files until the used space is below 1GB.

Site/Local squid server

If you have many computers accessing cvmfs, it will be beneficial to run your own squid server. Instructions to do this are the same as that for the Frontier-squid server (You can use the same squid to serve both cvmfs and Frontier). The instructions for setting a squid proxy cache can be found here: Squid rpm installation instructions. When you have one setup, please let us know so that we can add it to the database.

Machines which have limited space for cache

Although 25-50 GB of space is highly recommended, you can set a limit on how much cache disk is used if you have much less space. In /etc/cvmfs/default.local, set the variable CVMFS_QUOTA_LIMIT=<Soft Limit in MB>. Warning, important you must set the value no matter how much space you have. Remember to set it to only 80% of the available space (the other 20% will be taken up by hidden sql-lite files). Consider using an alien cache if the nodes are diskless or have small disks.

NFS export of cvmfs

This is something that should be done only as a very last resort for a site. It introduces a single point of failure and a performance bottleneck; also, other benefits of cvmfs, such as local file access instead of network file access, are lost.

Install cvmfs on nfs server

For this situation, you will need to
  • have good networking
  • have ample local cache space for cvmfs
  • run RHEL6 variant OS (ie SL6). Note this will be nfs4.
  • you are also recommended to install a local squid on that machine and allow 127.0.0.1 access to it.

Install and Configure

In /etc/cvmfs/default.local, also add

CVMFS_NFS_SOURCE=yes
CVMFS_MEMCACHE_SIZE=256

In /etc/fuse.conf, add

user_allow_other

Note: you may see this warning - it can be safely ignored for now. (The message will be fixed in a future version of cvmfs.)

cvmfs_config chksetup
Warning: CernVM-FS map is not referenced from autofs master map

Hard mount cvmfs and nfs export

On the nfs server where cvmfs was installed:
# make mount points :
mkdir -p /cvmfs/atlas.cern.ch
mkdir -p /cvmfs/atlas-nightlies.cern.ch
mkdir -p /cvmfs/atlas-condb.cern.ch
mkdir -p /cvmfs/sft.cern.ch
mkdir -p /cvmfs/unpacked.cern.ch

# in /etc/fstab, have these entries:
atlas         /cvmfs/atlas.cern.ch   cvmfs   defaults   0 0
atlas-nightlies      /cvmfs/atlas-nightlies.cern.ch   cvmfs   defaults   0 0
atlas-condb      /cvmfs/atlas-condb.cern.ch   cvmfs   defaults   0 0
sft         /cvmfs/sft.cern.ch   cvmfs   defaults   0 0
unpacked        /cvmfs/unpacked.cern.ch   cvmfs   defaults   0 0

# mount 
mount --all
# df -h should show you the mounted cvmfs repositories and you can then redo the quick tests described previous section.

# in /etc/exports  (replace <netmask> with your own)
/cvmfs/atlas.cern.ch <netmask>(ro,sync,no_root_squash,no_subtree_check,fsid=101)
/cvmfs/atlas-condb.cern.ch <netmask>(ro,sync,no_root_squash,no_subtree_check,fsid=102)
/cvmfs/atlas-nightlies.cern.ch <netmask>(ro,sync,no_root_squash,no_subtree_check,fsid=103)
/cvmfs/sft.cern.ch <netmask>(ro,sync,no_root_squash,no_subtree_check,fsid=104)
/cvmfs/unpacked.cern.ch <netmask>(ro,sync,no_root_squash,no_subtree_check,fsid=105)

# modify the file /etc/sysconfig/nfs:
 diff nfs nfs.old
20c20
< LOCKD_TCPPORT=32803
---
> #LOCKD_TCPPORT=32803
22c22
< LOCKD_UDPPORT=32769
---
> #LOCKD_UDPPORT=32769
43c43
< MOUNTD_PORT=892
---
> #MOUNTD_PORT=892

# start nfs
service nfs start
chkconfig nfs on


# fix iptables - change the <netmask> below to your netmask !
iptables -I INPUT -m state --state NEW -p tcp     -m multiport --dport 111,892,2049,32803 -s <netmask> -j ACCEPT
iptables -I INPUT -m state --state NEW -p udp     -m multiport --dport 111,892,2049,32769 -s <netmask> -j ACCEPT
service iptables save

On Worker nodes

Do not install cvmfs.

You should be able to see the nfs exports from your server: showmount -e <your nfs server hostname>

# make mount points:
mkdir -p /cvmfs/atlas.cern.ch
mkdir -p /cvmfs/atlas-nightlies.cern.ch
mkdir -p /cvmfs/atlas-condb.cern.ch
mkdir -p /cvmfs/sft.cern.ch
mkdir -p /cvmfs/unpacked.cern.ch

# add to /etc/fstab; replace <nfs server> with your nfs server hostname:
<nfs server>:/cvmfs/atlas.cern.ch /cvmfs/atlas.cern.ch nfs noatime,ac,actimeo=60 0 0
<nfs server>:/cvmfs/atlas-nightlies.cern.ch /cvmfs/atlas-nightlies.cern.ch nfs noatime,ac,actimeo=60 0 0
<nfs server>:/cvmfs/atlas-condb.cern.ch /cvmfs/atlas-condb.cern.ch nfs noatime,ac,actimeo=60 0 0
<nfs server>:/cvmfs/sft.cern.ch /cvmfs/sft.cern.ch nfs noatime,ac,actimeo=60 0 0
<nfs server>:/cvmfs/unpacked.cern.ch /cvmfs/unpacked.cern.ch nfs noatime,ac,actimeo=60 0 0

mount -all
# ls of the mount points should show identical listings to that when done on the nfs server.

Debugging

Please see the cvmfs debugging page if you encounter problems.

Mac OSX client installation and configuration

MacOS X Versions Catalina and newer (10.15 and newer)

Tested on Mac OS X 10.15

Install software

  • Install "FUSE for OS X"; Download the latest release from http://osxfuse.github.io and install it.
  • Download the latest CVMFS client image from http://cernvm.cern.ch/portal/filesystem and install it.
    • After you download, and try to open the installation .pkg or .dmg file, it may fail to open because it was from an "unidentified developer". If this happens:
      • Open System Preferences from the Apple Icon on the left top corner
      • Select Security & Privacy
      • In General Tab, Click on "Open Anyway" for the cvmfs pkg file
    • Remember to reboot after installing cvmfs.

Configure cvmfs

Open a terminal session as superuser (sudo) and create /etc/cvmfs/default.local as shown below:
#
# example contents of /etc/cvmfs/default.local
#
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,atlas-nightlies.cern.ch,sft.cern.ch,unpacked.cern.ch
CVMFS_QUOTA_LIMIT=10000
#
# at CERN
#CVMFS_HTTP_PROXY="http://ca-proxy.cern.ch:3128;http://ca-proxy1.cern.ch:3128|http://ca-proxy2.cern.ch:3128|http://ca-proxy3.cern.ch:3128|http://ca-proxy4.cern.ch:3128|http://ca-proxy5.cern.ch:3128"
#
# or direct if you do not have a nearby squid
#CVMFS_HTTP_PROXY="DIRECT"

Create the mount points for cvmfs

The installer will mount cvmfs on /User/shared/cvmfs with a firm link to /cvmfs (after reboot). Create the mount points as shown below.
mkdir -p /Users/Shared/cvmfs/atlas.cern.ch
mkdir -p /Users/Shared/cvmfs/atlas-condb.cern.ch
mkdir -p /Users/Shared/cvmfs/atlas-nightlies.cern.ch
mkdir -p /Users/Shared/cvmfs/sft.cern.ch
mkdir -p /Users/Shared/cvmfs/unpacked.cern.ch

Create some script to manually mount to unmount cvmfs

You can create scripts mount.sh and umount.sh as shown below: Put them somewhere you can access as sudo.

#! /bin/bash
# eg of mount.sh script; needs to run as superuser.  (sudo <path>/mount.sh)
mount -t cvmfs atlas.cern.ch /cvmfs/atlas.cern.ch
mount -t cvmfs atlas-condb.cern.ch /cvmfs/atlas-condb.cern.ch
mount -t cvmfs atlas-nightlies.cern.ch /cvmfs/atlas-nightlies.cern.ch
mount -t cvmfs sft.cern.ch /cvmfs/sft.cern.ch
mount -t cvmfs unpacked.cern.ch /cvmfs/unpacked.cern.ch

#! /bin/bash
# eg of unmount.sh script; needs to run as superuser.  (sudo <path>/unmount.sh)
umount  /cvmfs/atlas.cern.ch
umount  /cvmfs/atlas-condb.cern.ch
umount  /cvmfs/atlas-nightlies.cern.ch
umount  /cvmfs/sft.cern.ch
umount  /cvmfs/unpacked.cern.ch

Make sure those scripts are executable.

chmod +x mount.sh
chmod +x unmount.sh

How to use

Make sure cvmfs is mounted (Mount the cvmfs repositories by doing sudo <path to your script>/mount.sh)

You will use a "relocated" cvmfs on your Mac that will be bind mounted inside your container. Please see Twiki for containers. In particular, you will need to install Docker and XQuartz as indicated in the instructions and configurations sections of the link.

Create the following script (or put it inside your login script to define setupATLAS). Source this script once in your terminal session.


# A check so that this works only on MacOS and not inside Linux containers
if [ "`uname`" = "Darwin" ]; then
    export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase

# setupATLAS is defined as a function here
    setupATLAS () 
    { 
   if [ -d $ATLAS_LOCAL_ROOT_BASE ]; then
            source $ATLAS_LOCAL_ROOT_BASE/user/atlasLocalSetup.sh;
            return $?;
   else
            \echo "Error: cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase is unavailable" 1>&2;
            return 64;
   fi
    }

# alternately you can use an alias instead of a function
#    alias setupATLAS='source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh'
    
fi 

Then run a container:

  • setupATLAS -c slc6 for a slc6 container
  • setupATLAS -c centos7 for a centos7 container
Refer to the Containers Documentation for more information or other use cases.

MacOS X versions older than Catalina (older than 10.15)

Tested on MAC OSX 10.9

Install software

First you need "FUSE for OS X" - get the latest release from http://osxfuse.github.io and install it.

Then download the latest CVMFS client image from http://cernvm.cern.ch/portal/filesystem and install it.

Configure cvmfs

Open a terminal session and edit few files as su (sudo) as described here:
  • Configure Cvmfs
  • The main changes are in /etc/cvmfs/default.local; here is an example of the file contents:
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,atlas-nightlies.cern.ch,sft.cern.ch,unpacked.cern.ch
CVMFS_QUOTA_LIMIT=10000
CVMFS_HTTP_PROXY="DIRECT"

Mount cvmfs

(do as su - sudo)

To mount, we recommend to copy cvmfs_check.sh script (and make sure it is executable) in your /Users/<USERNAME>/private

You can now execute the cvmfs_check.sh script and it will mount the needed cvmfs repositories.

If you want to have /cvmfs mounted after each reboot of your machine you can copy cvmfs.filesystems.plist under /Library/LaunchDaemons and change inside <USERNAME> to your <USERNAME> and then make sure the location is correct and file is there in /Users/<USERNAME>/private/cvmfs_check.sh . ALERT! Caution: this may cause login problems on MacOS with FileVault enabled. If you are unable to login, boot into recovery mode and remove cvmfs.filesystems.plist .

CERN VM / cvmfs official website


Major updates:
-- AsokaDeSilva - 12-Mar-2013

Responsible: AsokaDeSilva
Last reviewed by: Never reviewed

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatplist cvmfs.filesystems.plist r1 manage 0.5 K 2014-01-29 - 15:20 EmilObreshkov  
Unix shell scriptsh cvmfs_check.sh r3 r2 r1 manage 0.5 K 2014-05-09 - 16:48 EmilObreshkov  
Edit | Attach | Watch | Print version | History: r67 < r66 < r65 < r64 < r63 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r67 - 2020-03-27 - AsokaDeSilva
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Atlas All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback