Instructions for Site Admins

This page describes steps for CMS Space Monitoring deployment at the sites.

In order to meet the timelines, system deployment was split in two phases:

Phase I Accumulate and store storage usage info locally at the sites
Phase II Aggregate and publish data into central database

The Phase I is now complete. Instructions have been updated on Feb 7th, 2017.

Space Monitoring deployment at the site

Complete: 5

Step 1: Locate/produce working tool to create storage-dumps

Storage-dump tools are storage technology specific. We maintain a common repository for the CMS supported storage technologies.
There you will find instructions and scripts developed by CMS/ATLAS site admins and/or references for the tools provided with the storage solutions, as well as sample storage-dumps.

If you use your own storage-dump tool, please follow storage-dump formats as described at https://twiki.cern.ch/twiki/bin/view/LCG/ConsistencyChecksSEsDumps#Format_of_SE_dumps.

NOTE:

  • Dump should contain metadata for all files on CMS storage, including data outside the official CMS datasets
  • For each file at least two values are required :
    • PFN (Physical File Name) consistent with the path returned by direct protocol in CMS Trivial File Catalog
    • file size is bytes
  • Dump files in txt format must have timestamp encoded in the file name, dumps in xml format must contain tag dump with attribute recorded, see examples in https://github.com/dmwm/DMWMMON/tree/master/SiteInfoProviders

Please do not hesitate to contribute your tools and bug-fixes to the common repository.
You can fork the repository and make a pull request to merge your branch, or you can ask Eric for write-access.

Step 2: Install and configure the client tool.

Using spacemon-client installed on CVMFS

The spacemon-client releases are now automatically deployed on CVMFS.
Assuming the cvmfs client is installed on your machine, you can start using spacemon-client directly from cvmfs cache, usually mounted as /cvmfs .
No setup is necessary. For convenience you may link the Utilities directory from preferred release to your working directory:

cd ~/mywork
ln -s /cvmfs/cms.cern.ch/spacemon-client/slc6_amd64_gcc493/cms/spacemon-client/1.0.2/DMWMMON/SpaceMon/Utilities/   ./
./Utilities/spacemon -h

To install CVMFS client, you can use instructions on how to set up the CVMFS client on CMS worker nodes as a reference, skipping the jobs configuration related steps.

Local installation as CMS rpm

  1. Create a directory for software installation:
    mkdir sw
    export sw=$PWD/sw
    
  2. Bootstrap externals, this is only needed once per architecture:
    myarch=slc6_amd64_gcc493 
    repo=comp 
    
  3. Now configure the CMS software area and search for available spcamon-client releases:
    wget -O $sw/bootstrap.sh http://cmsrep.cern.ch/cmssw/repos/bootstrap.sh
    sh -x $sw/bootstrap.sh setup -path $sw -arch $myarch -repository $repo 2>&1|tee $sw/bootstrap_$myarch.log
    $sw/common/cmspkg -a $myarch update
    $sw/common/cmspkg -a $myarch search spacemon-client
    
  4. Finally, install the desired version:
    version=1.0.2
    $sw/common/cmspkg -a $myarch install cms+spacemon-client+$version
    
  5. To test start a new session with a clean environment
    myarch=slc6_amd64_gcc493
    sw=`pwd`/sw
    source $sw/$myarch/cms/spacemon-client/1.0.2/etc/profile.d/init.sh
    grid-proxy-init
    spacemon -h
    

Installation from the github repository

This method is preferred for testing and development.
git clone https://github.com/dmwm/DMWMMON.git
cd DMWMMON/SpaceMon
git checkout spacemon-client_1_0_2
Utilities/spacemon -h

Configure local aggregation parameters (optional)

Spacemon new configuration feature allows to specify the level of depth at which directories in [[https://twiki.cern.ch/twiki/bin/view/CMS/DMWMPG_Namespace][CMS DMWMOG Namespace] are monitored.

To view a set of globally defined configuration rules, try

 spacemon --defaults 
User can override or add more rules in the local configuration file, defining %USERCFG perl hash with rules in terms of PFNs, as shown in the example.
%USERCFG = (
   '/' => 3,
   '/localtests/' => -1,
   '/dcache/uscmsdisk/store/user/' => 3,
   '/dcache/uscmsdisk/store/' => 4,
);

Namespace rules values define how many directory levels under the specified path are monitored.

Depth value Resulting behavior
0 - total size of the directory is monitored, the contents are concealed
1 - the directory and immediate sub-directories are monitored
2 (or 3, 4, ..) - two or more levels of sub-directories are monitored
-1 (negative int) - exclude all contents of the directory from the monitoring record

Spacemon will look for user's configuration in ~/.spacemonrc, this location can be overwritten with --config option.

Step 3: Manually upload storage records to the central database

Enable authentication

Upload to the central monitoring service requires certificate based authentication:
  • Make sure you have site admin role for your site defined in the SiteDB .
  • Make sure perl-Crypt-SSLeay rpm package is installed on the node where you do the upload. This package provides support for the https protocol used for the upload.
  • An RFC 3280-compliant proxy with at least 1024-bit key strength is required.

To verify your authentication use spacemon --check-auth= command. See spacemon -h for authentication related options.

Upload your record

By default spacemon prints the generated monitoring record on the standard output. To force the upload, add the --upload= option. For example:

spacemon --dump mystoragedump.1486494092.txt --node T2_MY_NODE --upload   

Storage dump files compressed with gzip or bzip2 are automatically detected and uncompressed on the fly.
Beware of issue with xml.bz2 files affecting 1.0.2 and earlier

Update your entry in this table

Once the upload step is complete, please add an entry for your site in the table below.

Site Name Date Storage Technology
T2_AT_Vienna 2015-06-23 dpm
T2_BE_IIHE    
T2_BE_UCL 2015-04-16 POSIX
T2_BR_SPRACE    
T2_BR_UERJ 2015-06-29 HDFS
T2_CH_CERN 2015-06-19 EOS
T2_CH_CSCS 2014-06-05 dCache
T2_CN_Beijing 2015-06-01 dCache
T2_DE_DESY 2015-04-20 dCache
T2_DE_RWTH 2017-03-18 dCache
T2_EE_Estonia    
T2_ES_CIEMAT 2014-05-12 dCache
T2_ES_IFCA 2015-05-28 POSIX (GPFS)
T2_FI_HIP 2015-04-09 dCache
T2_FR_CCIN2P3 2014-10-23 dCache
T2_FR_GRIF_IRFU 2014-08-01 dpm
T2_FR_GRIF_LLR 2014-05-28 dpm
T2_FR_IPHC 2014-10-16 dpm
T2_GR_Ioannina    
T2_HU_Budapest 2014-05-14 dpm
T2_IN_TIFR 2017-03-29 dpm
T2_IT_Bari    
T2_IT_Legnaro 2015-06-01 dCache
T2_IT_Pisa 2017-02-24 posix (GPFS)
T2_IT_Rome 2014-05-20 dCache
T2_KR_KNU 2014-05-20 dCache
T2_MY_UPM_BIRUNI    
T2_PK_NCP 2015-08-13 dpm
T2_PL_Warsaw    
T2_PL_Swierk 2015-05-14 dpm
T2_PT_NCG_Lisbon 2015-09-19 posix
T2_RU_IHEP    
T2_RU_INR 2015-07-07 dpm
T2_RU_ITEP    
T2_RU_JINR    
T2_RU_PNPI 2016-03-26 dpm
T2_RU_RRC_KI    
T2_RU_SINP    
T2_TH_CUNSTDA    
T2_TR_METU    
T2_TW_Taiwan    
T2_UA_KIPT 2016-04-11 DPM
T2_UK_London_Brunel 2015-06-06 DPM
T2_UK_London_IC    
T2_UK_SGrid_Bristol 2015-04-24 posix (GPFS/HDFS)
T2_UK_SGrid_RALPP    
T2_US_Caltech 2014-06-03 HDFS
T2_US_Florida 2014-05-02 Lustre
T2_US_MIT 2015-05-05 HDFS
T2_US_Nebraska 2015-04-20 HDFS
T2_US_Purdue 2015-04-03 HDFS
T2_US_UCSD 2014-11-03 HDFS
T2_US_Vanderbilt 2015-04-27 LStore
T2_US_Wisconsin 2014-11-25 HDFS
T2_KR_KISTI 2017-09-13 dCache

Step 4: Produce storage-dumps and upload records routinely

Sites are asked to upload storage usage records once per week.
Usually this involves setting up one cronjob to produce storage dumps, and another cronjob to run the spacemon command.
Second cron job should have access to the storage dump file and to a valid proxy file.
We recommend to use the voms proxy certificate maintained for the PhEDEx data transfers, please see certificate management details.

FAQ

Frequently asked questions by the site admins :

Q: Which sites are required to deploy CMS space monitoring

Answer: All Tier-1 and Tier 2 sites should report space usage information for each PhEDEx endpoint node except MSS and Buffer types.

Q: How often do the sites need to report their CMS storage space usage

Answer: reports are to be produced and uploaded weekly .

In case of problems with upload, e.g. is authentication expires, the sites can keep a local copy of the storage dumps, and upload it later.
The dump file name (or xml Recorded tag) must contain the timestamp when the storage dump was collected.

Q: What are the prerequisites for the authorized upload

Answer: The upload command requires a valid certificate with a DN registered in the CMS SiteDB to a person that has a ~site admin~ role for the site.

Q: How to check if the upload was successful

Answer: The dates of the most recent sites reports are periodically synchronized with the Space Check metric in CMS dashboard .
To initiate the real-time update, click on the 'date' in the 'Space Check' metric column next to the respective site name.
Or use DMWMMON data service APIs to get back your records.

Q: How to report problems and get help:

Answer: Problems and questions related to space monitoring deployment can be sent to hn-cms-comp-ops@cernNOSPAMPLEASE.ch.
In case of problems, please open a DMWMMON github issue .
We may ask you to provide your storage-dump for us to validate and tune the tools with, so do not delete it yet.

Q: How long until CMS asks for a new way to monitor site usage?

-- NataliaRatnikova - 14 Feb 2014

Edit | Attach | Watch | Print version | History: r82 < r81 < r80 < r79 < r78 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r82 - 2017-09-13 - GeonmoRyu
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback