Instructions for Site Admins
This page describes steps for CMS Space Monitoring deployment at the sites.
In order to meet the
timelines, system deployment was split in two phases:
Phase I |
Accumulate and store storage usage info locally at the sites |
Phase II |
Aggregate and publish data into central database |
The Phase I is now complete. Instructions have been updated on Feb 7th, 2017.
Space Monitoring deployment at the site
Complete:
Step 1: Locate/produce working tool to create storage-dumps
Storage-dump tools are storage technology specific. We maintain a common
repository
for the CMS supported storage technologies.
There you will find instructions and scripts developed by CMS/ATLAS site admins and/or references for the tools provided with the storage solutions, as well as sample storage-dumps.
If you use your own storage-dump tool, please follow storage-dump formats as described at
https://twiki.cern.ch/twiki/bin/view/LCG/ConsistencyChecksSEsDumps#Format_of_SE_dumps.
NOTE:
- Dump should contain metadata for all files on CMS storage, including data outside the official CMS datasets
- For each file at least two values are required :
- PFN (Physical File Name) consistent with the path returned by direct protocol in CMS Trivial File Catalog
- file size is bytes
- Dump files in
txt
format must have timestamp encoded in the file name, dumps in xml
format must contain tag dump
with attribute recorded
, see examples in https://github.com/dmwm/DMWMMON/tree/master/SiteInfoProviders
Please do not hesitate to contribute your tools and bug-fixes to the common repository.
You can fork the repository and make a pull request to merge your branch, or you can ask Eric for write-access.
Step 2: Install and configure the client tool.
Using spacemon-client installed on CVMFS
The spacemon-client releases are now automatically deployed on CVMFS.
Assuming the cvmfs client is installed on your machine, you can start using spacemon-client directly from cvmfs cache, usually mounted as
/cvmfs
.
No setup is necessary. For convenience you may link the Utilities directory from preferred release to your working directory:
cd ~/mywork
ln -s /cvmfs/cms.cern.ch/spacemon-client/slc6_amd64_gcc493/cms/spacemon-client/1.0.2/DMWMMON/SpaceMon/Utilities/ ./
./Utilities/spacemon -h
To install CVMFS client, you can use
instructions on how to set up the CVMFS client on CMS worker nodes as a reference, skipping the jobs configuration related steps.
Local installation as CMS rpm
- Create a directory for software installation:
mkdir sw
export sw=$PWD/sw
- Bootstrap externals, this is only needed once per architecture:
myarch=slc6_amd64_gcc493
repo=comp
- Now configure the CMS software area and search for available spcamon-client releases:
wget -O $sw/bootstrap.sh http://cmsrep.cern.ch/cmssw/repos/bootstrap.sh
sh -x $sw/bootstrap.sh setup -path $sw -arch $myarch -repository $repo 2>&1|tee $sw/bootstrap_$myarch.log
$sw/common/cmspkg -a $myarch update
$sw/common/cmspkg -a $myarch search spacemon-client
- Finally, install the desired version:
version=1.0.2
$sw/common/cmspkg -a $myarch install cms+spacemon-client+$version
- To test start a new session with a clean environment
myarch=slc6_amd64_gcc493
sw=`pwd`/sw
source $sw/$myarch/cms/spacemon-client/1.0.2/etc/profile.d/init.sh
grid-proxy-init
spacemon -h
Installation from the github repository
This method is preferred for testing and development.
git clone https://github.com/dmwm/DMWMMON.git
cd DMWMMON/SpaceMon
git checkout spacemon-client_1_0_2
Utilities/spacemon -h
Configure local aggregation parameters (optional)
Spacemon new configuration feature allows to specify the level of depth at which directories in [[https://twiki.cern.ch/twiki/bin/view/CMS/DMWMPG_Namespace][CMS DMWMOG Namespace] are monitored.
To view a set of globally defined configuration rules, try
spacemon --defaults
User can override or add more rules in the local configuration file, defining %USERCFG perl hash with rules in terms of PFNs, as shown in the example.
%USERCFG = (
'/' => 3,
'/localtests/' => -1,
'/dcache/uscmsdisk/store/user/' => 3,
'/dcache/uscmsdisk/store/' => 4,
);
Namespace rules values define how many directory levels under the specified path are monitored.
Depth value |
Resulting behavior |
0 |
- total size of the directory is monitored, the contents are concealed |
1 |
- the directory and immediate sub-directories are monitored |
2 (or 3, 4, ..) |
- two or more levels of sub-directories are monitored |
-1 (negative int) |
- exclude all contents of the directory from the monitoring record |
Spacemon will look for user's configuration in ~/.spacemonrc, this location can be overwritten with --config option.
Step 3: Manually upload storage records to the central database
Enable authentication
Upload to the central monitoring service requires certificate based authentication:
- Make sure you have site admin role for your site defined in the SiteDB
.
- Make sure perl-Crypt-SSLeay rpm package is installed on the node where you do the upload. This package provides support for the https protocol used for the upload.
- An RFC 3280-compliant proxy with at least 1024-bit key strength is required.
To verify your authentication use
spacemon --check-auth=
command. See
spacemon -h
for authentication related options.
Upload your record
By default spacemon prints the generated monitoring record on the standard output. To force the upload, add the
--upload=
option. For example:
spacemon --dump mystoragedump.1486494092.txt --node T2_MY_NODE --upload
Storage dump files compressed with
gzip
or
bzip2
are automatically detected and uncompressed on the fly.
Beware of issue with xml.bz2 files
affecting 1.0.2 and earlier
Update your entry in this table
Once the upload step is complete, please add an entry for your site in the table below.
Step 4: Produce storage-dumps and upload records routinely
Sites are asked to upload storage usage records once per week.
Usually this involves setting up one cronjob to produce storage dumps, and another cronjob to run the spacemon command.
Second cron job should have access to the storage dump file and to a valid proxy file.
We recommend to use the voms proxy certificate maintained for the PhEDEx data transfers, please see
certificate management details.
FAQ
Frequently asked questions by the site admins :
Q: Which sites are required to deploy CMS space monitoring
Answer: All Tier-1 and Tier 2 sites should report space usage information for each
PhEDEx endpoint node except MSS and Buffer types.
Q: How often do the sites need to report their CMS storage space usage
Answer: reports are to be produced and uploaded weekly .
In case of problems with upload, e.g. is authentication expires, the sites can keep a local copy of the storage dumps, and upload it later.
The dump file name (or xml Recorded tag) must contain the timestamp when the storage dump was collected.
Q: What are the prerequisites for the authorized upload
Answer: The upload command requires a valid certificate with a DN registered in the
CMS SiteDB
to a person that has a ~site admin~ role
for the site.
Q: How to check if the upload was successful
Answer: The dates of the most recent sites reports are periodically synchronized with the
Space Check metric in
CMS dashboard
.
To initiate the real-time update, click on the 'date' in the 'Space Check' metric column next to the respective site name.
Or use
DMWMMON data service APIs
to get back your records.
Q: How to report problems and get help:
Answer: Problems and questions related to space monitoring deployment can be sent to
hn-cms-comp-ops@cernNOSPAMPLEASE.ch.
In case of problems, please open a
DMWMMON github issue
.
We may ask you to provide your storage-dump for us to validate and tune the tools with, so do not delete it yet.
Q: How long until CMS asks for a new way to monitor site usage?
--
NataliaRatnikova - 14 Feb 2014