Castor

CASTOR is the acronym for CERN Advanced STORage manager, a data storage system used at CERN, more on cern.ch/castor

Documentation

• CMS documentation for working with CASTOR
• TOTEM software howto
• CASTOR quick tutorial for beginners
• CASTOR documentation on project webpage
• CASTOR2 getting started

Useful commands

• rfcp
• xrdcp
• nsls [-l]
• nsmkdir
• nsrm
• nsgetacl , nssetacl
• stager_qry
• stager_get

Recursive copy

• /afs/cern.ch/exp/totem/soft/bin/rfcp_r

Sample usage:

> /afs/cern.ch/exp/totem/soft/bin/rfcp_r
Usage: ./rfcp_r [from_dir] <to_dir>
> /afs/cern.ch/exp/totem/soft/bin/rfcp_r /castor/cern.ch/user/l/lgrzanka /tmp

This script is stored in SVN repository: trunk/offline/cmssw/tools/config_splitter/workspace/input/templates/rfcp_r

CASTOR1 vs CASTOR2

By default all commands are executed by CASTOR1 (old and deprecated). In order to use CASTOR2 one need to set some variables. In bash:

export RFIO_USE_CASTOR_V2=YES
export STAGE_HOST=castorpublic
export STAGE_SVCCLASS=default

Copying single files

Remember to set proper variables for CASTOR2. You can use now rfcp command to copy file:

rfcp /castor/cern.ch/totem/.../somefile.root /home/user/workspace

Or new XROOT protocol (might be faster, no need to setup variables for CASTOR2):

xrdcp root://castorpublic//castor/cern.ch/totem/.../somedir/somefile.root /home/user/workspace/ -OSsvcClass=default 
xrdcp /home/user/workspace/somefile.root root://castorpublic//castor/cern.ch/totem/.../somedir/ -ODsvcClass=default

If some of these commands hangs - read next section of this page.

Long waiting time with rfcp

To check status of file, use stager_qry command:

stager_qry -M /castor/cern.ch/totem/.../somefile.root

In most cases, as an result you can get one of following answers:

• STAGED
• STAGEIN
• Error 2/No such file

---

STAGED means that means "file on disk matrix (fast access)", so you should obtain it quickly with rfcp command.

Received 1 responses
/castor/cern.ch/totem/.../somefile.root
312327210@castorns STAGED

---

STAGEIN means that "file is on a tape" and need to be transferred to the disk matrix, which takes some time. You need to wait for the result of rfcp command.

Received 1 responses
/castor/cern.ch/totem/.../somefile.root
386766073@castorns STAGEIN

---

"Error 2/No such file" also means that "file is on a tape" and need to be transferred to the disk matrix, which takes some time. You need to wait for the result of rfcp command.

Error 2/No such file or directory (File /castor/cern.ch/totem/.../somefile.root not on this service class)

This error is very similar to the one that you obtain when asking about not existing file. In such case (file really missing) you will see:

Error 2/No such file or directory (Filename /castor/cern.ch/totem/.../somefile.root)

You can check is file is existing with nsls command.

---

In second and third case you can ask stager directly to move files to the "disk matrix" by issuing following command:

stager_get -S default -M /castor/cern.ch/totem/.../somefile.root

More on this topic - read this

Reporting errors

Switch on debugging for rfcp (bash/zsh):

export RFIO_TRACE=3
export STAGER_TRACE=3

or for tcsh/csh

setenv RFIO_TRACE 3
setenv STAGER_TRACE 3

To switch debugging for xrdcp, use -d switch (i.e. -d 1, -d 2, -d 3)

Resources

Global for TOTEM

• /castor/cern.ch/totem/offline/ for simulation data and reconstructed data
• /castor/cern.ch/totem/rawdata/ for real data (testbeams,...)
• /castor/cern.ch/totem/LHCRawData/ for real data (calibration, physics - LHC)

See more here

Personal

• similar to AFS home directory, for example /castor/cern.ch/user/l/lgrzanka

Permissions

To set permission we use nssetacl, to get them nsgetacl.

Checking permissions

[lxplus222] ~ > nsgetacl /castor/cern.ch/totem/offline/tmp
# file: /castor/cern.ch/totem/offline/tmp
# owner: totemcdr
# group: zj
user::rwx
user:lgrzanka:rwx               #effective:rwx
user:hniewiad:rwx               #effective:rwx
group::r-x              #effective:r-x
mask::rwx
other::r-x

Removing permissions

Remove ACL entry:

nssetacl -d u:jkaspar:rwx /castor/cern.ch/totem/offline

Changing permissions

Modify default ACLs to 775 (will be propagated).

nssetacl -m u::7,g::7,o::5 /castor/cern.ch/totem/offline/test
nssetacl -m d:u::7,d:g::7,d:o::5 /castor/cern.ch/totem/offline/test

For every directory (or file) created under /castor/cern.ch/totem/offline/test owner will be set up to the person who created it, but permissions of all directories will be set to 775. This way everybody in zj group will be able to write to any of subdirectories (whoever will create it).

Let us do the same as above, but not for group zj, but for list of users.

nssetacl -m d:u::7,d:g::5,d:o::5 /castor/cern.ch/totem/offline/test
nssetacl -m u::7,g::5,o::5 /castor/cern.ch/totem/offline/test
nssetacl -m default:group:zj:rx,default:other:-,default:mask:rwx /castor/cern.ch/totem/offline/test
nssetacl -m default:user:lgrzanka:rwx,default:mask:rwx /castor/cern.ch/totem/offline/test
nssetacl -m user:lgrzanka:rwx,mask:rwx /castor/cern.ch/totem/offline/test
nssetacl -m d:u:hniewiad:7,d:m:7 /castor/cern.ch/totem/offline/test
nssetacl -m u:hniewiad:7,m:7 /castor/cern.ch/totem/offline/test

For every directory (or file) created under /castor/cern.ch/totem/offline/test owner will be set up to the person who created it, but permissions of all directories will be set to 755. This way everybody in zj group will be able just to read from any of subdirectories (whoever will create it). Only users from the list can write to subdirectories.

The result is:

bash-3.00$ nsgetacl /castor/cern.ch/totem/offline
# file: /castor/cern.ch/totem/offline
# owner: totemcdr
# group: zj
user::rwx
user:jkaspar:rwx                #effective:rwx
user:glatino:rwx                #effective:rwx
user:bruecken:rwx               #effective:rwx
user:lgrzanka:rwx               #effective:rwx
user:brucken:rwx                #effective:rwx
user:fabferro:rwx               #effective:rwx
user:fgarciaf:rwx               #effective:rwx
user:hniewiad:rwx               #effective:rwx
group::r-x              #effective:r-x
mask::rwx
other::---
default:user::rwx
default:user:jkaspar:rwx
default:user:glatino:rwx
default:user:bruecken:rwx
default:user:lgrzanka:rwx
default:user:brucken:rwx
default:user:fabferro:rwx
default:user:fgarciaf:rwx
default:user:hniewiad:rwx
default:group::r-x
default:group:zj:r-x
default:mask::rwx
default:other::---

If after some time, new user needs to be added to the list, then we shall add him permissions to write to existing files. Following command will work only with bash.

nsfind /castor/cern.ch/totem/offline/test -type f > filelist.txt
cat filelist.txt | xargs nssetacl -m u:jkaspar:rw,m:rw
nsfind /castor/cern.ch/totem/offline/test -type d > dirlist.txt
cat dirlist.txt | xargs nssetacl -m d:u:jkaspar:rwx,d:m:rwx
cat dirlist.txt | xargs nssetacl -m u:jkaspar:rwx,m:rwx

Changing ownership

If recursive method for applying access rights was applied as it is described above, it might happen that newly created directories might have different owner as their parent directories. To change it one may use recursive version of nschown command, i.e:.

nschown -R totemcdr /castor/cern.ch/totem/offline/test

IMPORTANT: such command might disable effective access rights (bug in CASTOR?). To bring them back, read from output of nsgetacl commands who has for example read access and grant again such access for one user. This will make effective access rights correct, i.e:.

nsfind /castor/cern.ch/totem/offline/test -type d > dirlist.txt
cat dirlist.txt | xargs nssetacl -m d:u:jkaspar:rx,d:m:rx
cat dirlist.txt | xargs nssetacl -m u:jkaspar:rx,m:rx