CERNStorageTools < CMSPublic

CMSPublic Web>FacilitiesOps>CERNStorageTools (2015-10-20, AndreasPfeiffer) (raw view)
---+!! CERN Storage Tools


---++!! Please note that after a change in the the CERN site local config that was deployed on Mon, Oct 12, 2015, the old storage tools scripts (below) are no longer supported and are not guaranteed to work properly any more.

The scripts can be replaced with the following equivalent EOS commands (see `eos help` for more):

| cmsLs        | eos ls |
| cmsMkdir     | eos mkdir|
| cmsRm        | eos rm|
| cmsRmdir     | eos rmdir|
| cmsPfn       | [not needed any more as PFN==LFN] |
| cmsStage     | eos cp|
| cmsStageIn   | eos cp|
| cmsStageOut  | eos cp|

Please update your scripts using these commands accordingly. 

We will soon update the scripts to simply "redirect" the commands to their EOS counterpart and will print a warning message. If this warning causes you serious problems, let us know and we can discuss a temporary fix.

The scripts listed above will be removed in Jan 2016. 

As `eos` is not a command, its just a shell alias, you will need to call eos via the shell in python (default is direct and not via shell) and as the shell within python is not a login shell, the usual environment isn't setup and the alias doesn't exist.

As such, something like (pseudo code)

<code>subprocess.call("source /afs/cern.ch/project/eos/installation/cms/etc/setup.sh ; eos ...", shell = True)</code>

should work. Alternatively you can also call the actual eos command directly:

<code>subprocess.call("/afs/cern.ch/project/eos/installation/cms/bin/eos.select")</code>


---+ eos quota

This one isn't generic. It will display any quota you have available to you on EOS. This includes ones in areas that you can't actually write to, like the size of /store/relval is controlled with a group quota on zh. While you are in zh you do not have permission to write there. You should therefore usually only be interested in quotas in your name. A way to get this sumarised is using a command like;

<verbatim>
[lxplus429] ~ $ eos quota | grep ^user -A1 -B2
# ==> Quota Node: /eos/cms/store/caf/user/
# _______________________________________________________________________________________________
user       used bytes logi bytes used files aval bytes aval logib aval files filled[%]  vol-status ino-status
gowdy      0.00 B     0.00 B     0.00 -     2.00 TB    1.00 TB    10.00 k-   0.00       ok         ok        
--
# ==> Quota Node: /eos/cms/store/cmst3/user/
# _______________________________________________________________________________________________
user       used bytes logi bytes used files aval bytes aval logib aval files filled[%]  vol-status ino-status
gowdy      43.37 MB   21.68 MB   2.00 -     2.00 TB    1.00 TB    10.00 k-   0.00       ok         ok        
--
# ==> Quota Node: /eos/cms/store/user/
# _______________________________________________________________________________________________
user       used bytes logi bytes used files aval bytes aval logib aval files filled[%]  vol-status ino-status
gowdy      441.58 MB  220.79 MB  14.00 -    2.00 TB    1.00 TB    10.00 k-   0.02       ok         ok        
</verbatim>

There are two quotas normally applied to any area. A space usage is what you would normally expect, you are allowed to write so much data to each area, for example by default each user gets 1TB of space in the /store/user/ area. There is also a quota on the number of files you can write. This helps ensure that the name space stays responsive, EOS isn't designed for tiny files, those should go in AFS. If you see 'warning' then it means you are within 10% of that quota. If you see 'exceeded' then you have gone over (or are so close to it that it believes you have) that quota. If you see 'ignored' it means that quota isn't applied, you may see this for areas where you no longer have write permission. The size quotas are on the physical disk used, which is normally twice the size of the files you store as each file is usually replicated for redundancy. You see both numbers in the above output.


---++ Old storage tools info

The tools presented on this page are useful for interacting with the storage systems available at CERN. In fact they should be generic enough to work elsewhere but they haven't been tested anywhere else. They reside in [[http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/UserCode/gowdy/src/python/cmsStage/?pathrev=HEAD][CVS]] but also require the [[http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/PRODCOMMON/src/python/IMProv/?pathrev=HEAD][IMProv]] and [[http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/PRODAGENT/src/python/StageOut/?pathrev=HEAD][StageOut]] modules from PRODCOMMON/PRODAGENT.

%TOC%

---+ cmsStage

This command is used to copy files. The syntax is;

<verbatim>
cmsStage [options] <source> <destination>
</verbatim>

The files specified can be LFNs or local file names (or a directory for destinations). An LFN is assumed to start /store (there is a temporary addition for /user to be an LFN pointing to the [[CASTORUserAreaMigration][CERN CASTOR User area]]).

---+ cmsLs

This command will provide ls like output. You can specify multiple LFNs and it accepts a few options, not all of which are supported by all storage elements types.

---+ cmsRm

This can be used to remove files. You can specify mutliple LFNs/files. It can also be used to recursively remove directories (-r) but only if supported by the storage element (there is a special addition to support this on EOS at CERN and it will warn you that it is attempting to use it).

---+ cmsRmdir

This can be used to remove empty directories. You can specify multiple directories. It takes the -p option to try to remove parent directories that are empty also (also not supported by all storage elements).

---+ cmsPfn

This command can be used to determine the PFN of an LFN. This can be useful to pass to ROOT, or other utilities which do not use the CMS Trivial File Catalog.


-- Main.AndreasPfeiffer - 2015-10-20
Topic revision: r3 - 2015-10-20 - AndreasPfeiffer
CMSPublic
Create a LeftBar
Public webs
- Cern Search
- TWiki Search
- Google Search
CMSPublic All webs
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback