CERN Storage Tools

The tools presented on this page are useful for interacting with the storage systems available at CERN. In fact they should be generic enough to work elsewhere but they haven't been tested anywhere else. They reside in CVS but also require the IMProv and StageOut modules from PRODCOMMON/PRODAGENT.

cmsStage

This command is used to copy files. The syntax is;

cmsStage [options] <source> <destination>

The files specified can be LFNs or local file names (or a directory for destinations). An LFN is assumed to start /store (there is a temporary addition for /user to be an LFN pointing to the CERN CASTOR User area).

cmsLs

This command will provide ls like output. You can specify multiple LFNs and it accepts a few options, not all of which are supported by all storage elements types.

cmsRm

This can be used to remove files. You can specify mutliple LFNs/files. It can also be used to recursively remove directories (-r) but only if supported by the storage element (there is a special addition to support this on EOS at CERN and it will warn you that it is attempting to use it).

cmsRmdir

This can be used to remove empty directories. You can specify multiple directories. It takes the -p option to try to remove parent directories that are empty also (also not supported by all storage elements).

cmsPfn

This command can be used to determine the PFN of an LFN. This can be useful to pass to ROOT, or other utilities which do not use the CMS Trivial File Catalog.

eos quota

This one isn't generic. It will display any quota you have available to you on EOS. This includes ones in areas that you can't actually write to, like the size of /store/relval is controlled with a group quota on zh. While you are in zh you do not have permission to write there. You should therefore usually only be interested in quotas in your name. A way to get this sumarised is using a command like;

[lxplus429] ~ $ eos quota | grep ^user -A1 -B2
# ==> Quota Node: /eos/cms/store/caf/user/
# _______________________________________________________________________________________________
user       used bytes logi bytes used files aval bytes aval logib aval files filled[%]  vol-status ino-status
gowdy      0.00 B     0.00 B     0.00 -     2.00 TB    1.00 TB    10.00 k-   0.00       ok         ok        
--
# ==> Quota Node: /eos/cms/store/cmst3/user/
# _______________________________________________________________________________________________
user       used bytes logi bytes used files aval bytes aval logib aval files filled[%]  vol-status ino-status
gowdy      43.37 MB   21.68 MB   2.00 -     2.00 TB    1.00 TB    10.00 k-   0.00       ok         ok        
--
# ==> Quota Node: /eos/cms/store/user/
# _______________________________________________________________________________________________
user       used bytes logi bytes used files aval bytes aval logib aval files filled[%]  vol-status ino-status
gowdy      441.58 MB  220.79 MB  14.00 -    2.00 TB    1.00 TB    10.00 k-   0.02       ok         ok        

There are two quotas normally applied to any area. A space usage is what you would normally expect, you are allowed to write so much data to each area, for example by default each user gets 1TB of space in the /store/user/ area. There is also a quota on the number of files you can write. This helps ensure that the name space stays responsive, EOS isn't designed for tiny files, those should go in AFS. If you see 'warning' then it means you are within 10% of that quota. If you see 'exceeded' then you have gone over (or are so close to it that it believes you have) that quota. If you see 'ignored' it means that quota isn't applied, you may see this for areas where you no longer have write permission. The size quotas are on the physical disk used, which is normally twice the size of the files you store as each file is usually replicated for redundancy. You see both numbers in the above output.

-- StephenJGowdy - 14-Jul-2012


This topic: CMSPublic > FacilitiesOps > CERNStorageTools
Topic revision: r2 - 2012-09-28 - StephenJGowdy
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback