TWiki
>
CMSPublic Web
>
FacilitiesOps
>
CERNStorageTools
(2015-10-20,
AndreasPfeiffer
)
(raw view)
E
dit
A
ttach
P
DF
---+!! CERN Storage Tools ---++!! Please note that after a change in the the CERN site local config that was deployed on Mon, Oct 12, 2015, the old storage tools scripts (below) are no longer supported and are not guaranteed to work properly any more. The scripts can be replaced with the following equivalent EOS commands (see `eos help` for more): | cmsLs | eos ls | | cmsMkdir | eos mkdir| | cmsRm | eos rm| | cmsRmdir | eos rmdir| | cmsPfn | [not needed any more as PFN==LFN] | | cmsStage | eos cp| | cmsStageIn | eos cp| | cmsStageOut | eos cp| Please update your scripts using these commands accordingly. We will soon update the scripts to simply "redirect" the commands to their EOS counterpart and will print a warning message. If this warning causes you serious problems, let us know and we can discuss a temporary fix. The scripts listed above will be removed in Jan 2016. As `eos` is not a command, its just a shell alias, you will need to call eos via the shell in python (default is direct and not via shell) and as the shell within python is not a login shell, the usual environment isn't setup and the alias doesn't exist. As such, something like (pseudo code) <code>subprocess.call("source /afs/cern.ch/project/eos/installation/cms/etc/setup.sh ; eos ...", shell = True)</code> should work. Alternatively you can also call the actual eos command directly: <code>subprocess.call("/afs/cern.ch/project/eos/installation/cms/bin/eos.select")</code> ---+ eos quota This one isn't generic. It will display any quota you have available to you on EOS. This includes ones in areas that you can't actually write to, like the size of /store/relval is controlled with a group quota on zh. While you are in zh you do not have permission to write there. You should therefore usually only be interested in quotas in your name. A way to get this sumarised is using a command like; <verbatim> [lxplus429] ~ $ eos quota | grep ^user -A1 -B2 # ==> Quota Node: /eos/cms/store/caf/user/ # _______________________________________________________________________________________________ user used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status gowdy 0.00 B 0.00 B 0.00 - 2.00 TB 1.00 TB 10.00 k- 0.00 ok ok -- # ==> Quota Node: /eos/cms/store/cmst3/user/ # _______________________________________________________________________________________________ user used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status gowdy 43.37 MB 21.68 MB 2.00 - 2.00 TB 1.00 TB 10.00 k- 0.00 ok ok -- # ==> Quota Node: /eos/cms/store/user/ # _______________________________________________________________________________________________ user used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status gowdy 441.58 MB 220.79 MB 14.00 - 2.00 TB 1.00 TB 10.00 k- 0.02 ok ok </verbatim> There are two quotas normally applied to any area. A space usage is what you would normally expect, you are allowed to write so much data to each area, for example by default each user gets 1TB of space in the /store/user/ area. There is also a quota on the number of files you can write. This helps ensure that the name space stays responsive, EOS isn't designed for tiny files, those should go in AFS. If you see 'warning' then it means you are within 10% of that quota. If you see 'exceeded' then you have gone over (or are so close to it that it believes you have) that quota. If you see 'ignored' it means that quota isn't applied, you may see this for areas where you no longer have write permission. The size quotas are on the physical disk used, which is normally twice the size of the files you store as each file is usually replicated for redundancy. You see both numbers in the above output. ---++ Old storage tools info The tools presented on this page are useful for interacting with the storage systems available at CERN. In fact they should be generic enough to work elsewhere but they haven't been tested anywhere else. They reside in [[http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/UserCode/gowdy/src/python/cmsStage/?pathrev=HEAD][CVS]] but also require the [[http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/PRODCOMMON/src/python/IMProv/?pathrev=HEAD][IMProv]] and [[http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/PRODAGENT/src/python/StageOut/?pathrev=HEAD][StageOut]] modules from PRODCOMMON/PRODAGENT. %TOC% ---+ cmsStage This command is used to copy files. The syntax is; <verbatim> cmsStage [options] <source> <destination> </verbatim> The files specified can be LFNs or local file names (or a directory for destinations). An LFN is assumed to start /store (there is a temporary addition for /user to be an LFN pointing to the [[CASTORUserAreaMigration][CERN CASTOR User area]]). ---+ cmsLs This command will provide ls like output. You can specify multiple LFNs and it accepts a few options, not all of which are supported by all storage elements types. ---+ cmsRm This can be used to remove files. You can specify mutliple LFNs/files. It can also be used to recursively remove directories (-r) but only if supported by the storage element (there is a special addition to support this on EOS at CERN and it will warn you that it is attempting to use it). ---+ cmsRmdir This can be used to remove empty directories. You can specify multiple directories. It takes the -p option to try to remove parent directories that are empty also (also not supported by all storage elements). ---+ cmsPfn This command can be used to determine the PFN of an LFN. This can be useful to pass to ROOT, or other utilities which do not use the CMS Trivial File Catalog. -- Main.AndreasPfeiffer - 2015-10-20
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r3
<
r2
<
r1
|
B
acklinks
|
V
iew topic
|
WYSIWYG
|
M
ore topic actions
Topic revision: r3 - 2015-10-20
-
AndreasPfeiffer
Log In
CMSPublic
CMSPublic Web
CMSPrivate Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Create
a LeftBar
Public webs
Public webs
ABATBEA
ACPP
ADCgroup
AEGIS
AfricaMap
AgileInfrastructure
ALICE
AliceEbyE
AliceSPD
AliceSSD
AliceTOF
AliFemto
ALPHA
ArdaGrid
ASACUSA
AthenaFCalTBAna
Atlas
AtlasLBNL
AXIALPET
CAE
CALICE
CDS
CENF
CERNSearch
CLIC
Cloud
CloudServices
CMS
Controls
CTA
CvmFS
DB
DefaultWeb
DESgroup
DPHEP
DM-LHC
DSSGroup
EGEE
EgeePtf
ELFms
EMI
ETICS
FIOgroup
FlukaTeam
Frontier
Gaudi
GeneratorServices
GuidesInfo
HardwareLabs
HCC
HEPIX
ILCBDSColl
ILCTPC
IMWG
Inspire
IPv6
IT
ItCommTeam
ITCoord
ITdeptTechForum
ITDRP
ITGT
ITSDC
LAr
LCG
LCGAAWorkbook
Leade
LHCAccess
LHCAtHome
LHCb
LHCgas
LHCONE
LHCOPN
LinuxSupport
Main
Medipix
Messaging
MPGD
NA49
NA61
NA62
NTOF
Openlab
PDBService
Persistency
PESgroup
Plugins
PSAccess
PSBUpgrade
R2Eproject
RCTF
RD42
RFCond12
RFLowLevel
ROXIE
Sandbox
SocialActivities
SPI
SRMDev
SSM
Student
SuperComputing
Support
SwfCatalogue
TMVA
TOTEM
TWiki
UNOSAT
Virtualization
VOBox
WITCH
XTCA
Cern Search
TWiki Search
Google Search
CMSPublic
All webs
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use
Discourse
or
Send feedback