CMSPublic Web>CMSPublicData>CompProjOffice>CompProjSpaceMon>SpaceMonSiteAdmin (2017-09-13, GeonmoRyu)

Instructions for Site Admins

This page describes steps for CMS Space Monitoring deployment at the sites.

In order to meet the timelines, system deployment was split in two phases:

Phase I	Accumulate and store storage usage info locally at the sites
Phase II	Aggregate and publish data into central database

The Phase I is now complete. Instructions have been updated on Feb 7th, 2017.

Instructions for Site Admins
- Space Monitoring deployment at the site
- FAQ

Space Monitoring deployment at the site

Complete:

Step 1: Locate/produce working tool to create storage-dumps

Storage-dump tools are storage technology specific. We maintain a common repository for the CMS supported storage technologies.
There you will find instructions and scripts developed by CMS/ATLAS site admins and/or references for the tools provided with the storage solutions, as well as sample storage-dumps.

If you use your own storage-dump tool, please follow storage-dump formats as described at https://twiki.cern.ch/twiki/bin/view/LCG/ConsistencyChecksSEsDumps#Format_of_SE_dumps.

NOTE:

Dump should contain metadata for all files on CMS storage, including data outside the official CMS datasets
For each file at least two values are required :
- PFN (Physical File Name) consistent with the path returned by direct protocol in CMS Trivial File Catalog
- file size is bytes
Dump files in txt format must have timestamp encoded in the file name, dumps in xml format must contain tag dump with attribute recorded, see examples in https://github.com/dmwm/DMWMMON/tree/master/SiteInfoProviders

Please do not hesitate to contribute your tools and bug-fixes to the common repository.
You can fork the repository and make a pull request to merge your branch, or you can ask Eric for write-access.

Step 2: Install and configure the client tool.

Using spacemon-client installed on CVMFS

The spacemon-client releases are now automatically deployed on CVMFS.
Assuming the cvmfs client is installed on your machine, you can start using spacemon-client directly from cvmfs cache, usually mounted as /cvmfs .
No setup is necessary. For convenience you may link the Utilities directory from preferred release to your working directory:

cd ~/mywork
ln -s /cvmfs/cms.cern.ch/spacemon-client/slc6_amd64_gcc493/cms/spacemon-client/1.0.2/DMWMMON/SpaceMon/Utilities/   ./
./Utilities/spacemon -h

To install CVMFS client, you can use instructions on how to set up the CVMFS client on CMS worker nodes as a reference, skipping the jobs configuration related steps.

Local installation as CMS rpm

Show instructions

Hide instructions

Create a directory for software installation:
```
mkdir sw
export sw=$PWD/sw
```
Bootstrap externals, this is only needed once per architecture:
```
myarch=slc6_amd64_gcc493 
repo=comp 
```

Now configure the CMS software area and search for available spcamon-client releases:

wget -O $sw/bootstrap.sh http://cmsrep.cern.ch/cmssw/repos/bootstrap.sh
sh -x $sw/bootstrap.sh setup -path $sw -arch $myarch -repository $repo 2>&1|tee $sw/bootstrap_$myarch.log
$sw/common/cmspkg -a $myarch update
$sw/common/cmspkg -a $myarch search spacemon-client

Finally, install the desired version:

version=1.0.2
$sw/common/cmspkg -a $myarch install cms+spacemon-client+$version

To test start a new session with a clean environment

myarch=slc6_amd64_gcc493
sw=`pwd`/sw
source $sw/$myarch/cms/spacemon-client/1.0.2/etc/profile.d/init.sh
grid-proxy-init
spacemon -h

Installation from the github repository

This method is preferred for testing and development.

git clone https://github.com/dmwm/DMWMMON.git
cd DMWMMON/SpaceMon
git checkout spacemon-client_1_0_2
Utilities/spacemon -h

Configure local aggregation parameters (optional)

Spacemon new configuration feature allows to specify the level of depth at which directories in [[https://twiki.cern.ch/twiki/bin/view/CMS/DMWMPG_Namespace][CMS DMWMOG Namespace] are monitored.

To view a set of globally defined configuration rules, try

 spacemon --defaults

User can override or add more rules in the local configuration file, defining %USERCFG perl hash with rules in terms of PFNs, as shown in the example.

Show example:

Hide example

%USERCFG = (
   '/' => 3,
   '/localtests/' => -1,
   '/dcache/uscmsdisk/store/user/' => 3,
   '/dcache/uscmsdisk/store/' => 4,
);

Namespace rules values define how many directory levels under the specified path are monitored.

Depth value	Resulting behavior
0	- total size of the directory is monitored, the contents are concealed
1	- the directory and immediate sub-directories are monitored
2 (or 3, 4, ..)	- two or more levels of sub-directories are monitored
-1 (negative int)	- exclude all contents of the directory from the monitoring record

Spacemon will look for user's configuration in ~/.spacemonrc, this location can be overwritten with --config option.

Step 3: Manually upload storage records to the central database

Enable authentication

Upload to the central monitoring service requires certificate based authentication:

Make sure you have site admin role for your site defined in the SiteDB .
Make sure perl-Crypt-SSLeay rpm package is installed on the node where you do the upload. This package provides support for the https protocol used for the upload.
An RFC 3280-compliant proxy with at least 1024-bit key strength is required.

To verify your authentication use spacemon --check-auth= command. See spacemon -h for authentication related options.

Upload your record

By default spacemon prints the generated monitoring record on the standard output. To force the upload, add the --upload= option. For example:

spacemon --dump mystoragedump.1486494092.txt --node T2_MY_NODE --upload

Storage dump files compressed with gzip or bzip2 are automatically detected and uncompressed on the fly.
Beware of issue with xml.bz2 files affecting 1.0.2 and earlier

Update your entry in this table

Once the upload step is complete, please add an entry for your site in the table below.

Site Name	Date	Storage Technology
T2_AT_Vienna	2015-06-23	dpm
T2_BE_IIHE
T2_BE_UCL	2015-04-16	POSIX
T2_BR_SPRACE
T2_BR_UERJ	2015-06-29	HDFS
T2_CH_CERN	2015-06-19	EOS
T2_CH_CSCS	2014-06-05	dCache
T2_CN_Beijing	2015-06-01	dCache
T2_DE_DESY	2015-04-20	dCache
T2_DE_RWTH	2017-03-18	dCache
T2_EE_Estonia
T2_ES_CIEMAT	2014-05-12	dCache
T2_ES_IFCA	2015-05-28	POSIX (GPFS)
T2_FI_HIP	2015-04-09	dCache
T2_FR_CCIN2P3	2014-10-23	dCache
T2_FR_GRIF_IRFU	2014-08-01	dpm
T2_FR_GRIF_LLR	2014-05-28	dpm
T2_FR_IPHC	2014-10-16	dpm
T2_GR_Ioannina
T2_HU_Budapest	2014-05-14	dpm
T2_IN_TIFR	2017-03-29	dpm
T2_IT_Bari
T2_IT_Legnaro	2015-06-01	dCache
T2_IT_Pisa	2017-02-24	posix (GPFS)
T2_IT_Rome	2014-05-20	dCache
T2_KR_KNU	2014-05-20	dCache
T2_MY_UPM_BIRUNI
T2_PK_NCP	2015-08-13	dpm
T2_PL_Warsaw
T2_PL_Swierk	2015-05-14	dpm
T2_PT_NCG_Lisbon	2015-09-19	posix
T2_RU_IHEP
T2_RU_INR	2015-07-07	dpm
T2_RU_ITEP
T2_RU_JINR
T2_RU_PNPI	2016-03-26	dpm
T2_RU_RRC_KI
T2_RU_SINP
T2_TH_CUNSTDA
T2_TR_METU
T2_TW_Taiwan
T2_UA_KIPT	2016-04-11	DPM
T2_UK_London_Brunel	2015-06-06	DPM
T2_UK_London_IC
T2_UK_SGrid_Bristol	2015-04-24	posix (GPFS/HDFS)
T2_UK_SGrid_RALPP
T2_US_Caltech	2014-06-03	HDFS
T2_US_Florida	2014-05-02	Lustre
T2_US_MIT	2015-05-05	HDFS
T2_US_Nebraska	2015-04-20	HDFS
T2_US_Purdue	2015-04-03	HDFS
T2_US_UCSD	2014-11-03	HDFS
T2_US_Vanderbilt	2015-04-27	LStore
T2_US_Wisconsin	2014-11-25	HDFS
T2_KR_KISTI	2017-09-13	dCache

Step 4: Produce storage-dumps and upload records routinely

Sites are asked to upload storage usage records once per week.
Usually this involves setting up one cronjob to produce storage dumps, and another cronjob to run the spacemon command.
Second cron job should have access to the storage dump file and to a valid proxy file.
We recommend to use the voms proxy certificate maintained for the PhEDEx data transfers, please see certificate management details.

FAQ

Frequently asked questions by the site admins :

Q: Which sites are required to deploy CMS space monitoring

Answer: All Tier-1 and Tier 2 sites should report space usage information for each PhEDEx endpoint node except MSS and Buffer types.

Q: How often do the sites need to report their CMS storage space usage

Answer: reports are to be produced and uploaded weekly .

In case of problems with upload, e.g. is authentication expires, the sites can keep a local copy of the storage dumps, and upload it later.
The dump file name (or xml Recorded tag) must contain the timestamp when the storage dump was collected.

Q: What are the prerequisites for the authorized upload

Answer: The upload command requires a valid certificate with a DN registered in the CMS SiteDB

to a person that has a ~site admin~ role for the site.

Q: How to check if the upload was successful

Answer: The dates of the most recent sites reports are periodically synchronized with the Space Check metric in CMS dashboard

.
To initiate the real-time update, click on the 'date' in the 'Space Check' metric column next to the respective site name.
Or use DMWMMON data service APIs

to get back your records.

Q: How to report problems and get help:

Answer: Problems and questions related to space monitoring deployment can be sent to hn-cms-comp-ops@cernNOSPAMPLEASE.ch.
In case of problems, please open a DMWMMON github issue

.
We may ask you to provide your storage-dump for us to validate and tune the tools with, so do not delete it yet.