Instructions for Site Admins

In order to meet the timelines, system deployment was split in two phases:

The Phase I is now complete. Instructions have been updated on Feb 7th, 2017.

Space Monitoring deployment at the site

Step 1: Locate/produce working tool to create storage-dumps

Storage-dump tools are storage technology specific. We maintain a common repository for the CMS supported storage technologies.
There you will find instructions and scripts developed by CMS/ATLAS site admins and/or references for the tools provided with the storage solutions, as well as sample storage-dumps.

If you use your own storage-dump tool, please follow storage-dump formats as described at https://twiki.cern.ch/twiki/bin/view/LCG/ConsistencyChecksSEsDumps#Format_of_SE_dumps.

NOTE:

• Dump should contain metadata for all files on CMS storage, including data outside the official CMS datasets
• For each file at least two values are required :
  • PFN (Physical File Name) consistent with the path returned by direct protocol in CMS Trivial File Catalog
  • file size is bytes
• Dump files in txt format must have timestamp encoded in the file name, dumps in xml format must contain tag dump with attribute recorded, see examples in https://github.com/dmwm/DMWMMON/tree/master/SiteInfoProviders

Please do not hesitate to contribute your tools and bug-fixes to the common repository.
You can fork the repository and make a pull request to merge your branch, or you can ask Eric for write-access.

Step 2: Install and configure the client tool.

Using spacemon-client installed on CVMFS

The spacemon-client releases are now automatically deployed on CVMFS.
Assuming the cvmfs client is installed on your machine, you can start using spacemon-client directly from cvmfs cache, usually mounted as /cvmfs .
No setup is necessary. For convenience you may link the Utilities directory from preferred release to your working directory:

cd ~/mywork
ln -s /cvmfs/cms.cern.ch/spacemon-client/slc6_amd64_gcc493/cms/spacemon-client/1.0.2/DMWMMON/SpaceMon/Utilities/   ./
./Utilities/spacemon -h

To install CVMFS client, you can use instructions on how to set up the CVMFS client on CMS worker nodes as a reference, skipping the jobs configuration related steps.

Local installation as CMS rpm

Show instructions Hide instructions

Installation from the github repository

git clone https://github.com/dmwm/DMWMMON.git
cd DMWMMON/SpaceMon
git checkout spacemon-client_1_0_2
Utilities/spacemon -h

Configure local aggregation parameters (optional)

Spacemon new configuration feature allows to specify the level of depth at which directories in [[https://twiki.cern.ch/twiki/bin/view/CMS/DMWMPG_Namespace][CMS DMWMOG Namespace] are monitored.

To view a set of globally defined configuration rules, try

 spacemon --defaults 

Show example: Hide example

%USERCFG = (
   '/' => 3,
   '/localtests/' => -1,
   '/dcache/uscmsdisk/store/user/' => 3,
   '/dcache/uscmsdisk/store/' => 4,
);

Namespace rules values define how many directory levels under the specified path are monitored.

Spacemon will look for user's configuration in ~/.spacemonrc, this location can be overwritten with --config option.

Step 3: Manually upload storage records to the central database

Enable authentication

• Make sure you have site admin role for your site defined in the SiteDB .
• Make sure perl-Crypt-SSLeay rpm package is installed on the node where you do the upload. This package provides support for the https protocol used for the upload.
• An RFC 3280-compliant proxy with at least 1024-bit key strength is required.

To verify your authentication use spacemon --check-auth= command. See spacemon -h for authentication related options.

Upload your record

By default spacemon prints the generated monitoring record on the standard output. To force the upload, add the --upload= option. For example:

spacemon --dump mystoragedump.1486494092.txt --node T2_MY_NODE --upload   

Storage dump files compressed with gzip or bzip2 are automatically detected and uncompressed on the fly.
Beware of issue with xml.bz2 files affecting 1.0.2 and earlier

Update your entry in this table

Once the upload step is complete, please add an entry for your site in the table below.

Step 4: Produce storage-dumps and upload records routinely

FAQ

Q: Which sites are required to deploy CMS space monitoring

Q: How often do the sites need to report their CMS storage space usage

In case of problems with upload, e.g. is authentication expires, the sites can keep a local copy of the storage dumps, and upload it later.
The dump file name (or xml Recorded tag) must contain the timestamp when the storage dump was collected.

Q: What are the prerequisites for the authorized upload

CMS SiteDB

Q: How to check if the upload was successful

CMS dashboard

DMWMMON data service APIs

Q: How to report problems and get help:

DMWMMON github issue

Q: How long until CMS asks for a new way to monitor site usage?

-- NataliaRatnikova - 14 Feb 2014