CMS Space Monitoring development
Complete:
This page is for CMS
SpaceMon developers.
Code management
Second prototype of
SpaceMon has been implemented within
PhEDEx repository
git@github.com:dmwm/PHEDEX.git
Code lives in the following locations:
The plan is to migrate
SpaceMon into DMWMMON repository on github:
git@github.com:dmwm/DMWMMON.git
Ask Tony, Nicolo, Erik, Giulio for access details.
Git configuration
Here is Natalia's git config for
SpaceMon development with added remote upstream branch for merging in changes from the master branch in
PhEDEx github repo.
$ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = https://github.com/nataliaratnikova/PHEDEX.git
[branch "master"]
remote = origin
merge = refs/heads/master
[remote "upstream"]
url = https://github.com/dmwm/PHEDEX.git
fetch = +refs/heads/*:refs/remotes/upstream/*
The order of commands is something like this:
- make sure work area is clean -> everything is committed
-
git branch -vv
make sure you are on local master branch and tracking origin master branch
-
git push
push changes to remote private github repo
-
git pull --rebase upstream master
get in changes from the upstream repo : you may see branches diverge,
-
git merge origin/master
this should cause no conflicts, as long as SpaceMon is developed in private github repo, you may still be ahead of origin master, but no longer diverged.
-
git fetch upstream master
(is it really needed?)
-
git push
everything should be consistent now.
Access to Information Store
Access to Information Store (DMWMMON Database) is realized via
PhEDEx Data Service interface, and uses certificate based authentication with additional rules
for write access based on
SiteDb roles information. Particularly
site admin role for the node in question is required, see
current implementation
.
You can verify your authentication state and abilities by Data Service
auth request url:
https://cmsweb.cern.ch/dmwmmon/datasvc/perl/auth
as documented here:
https://cmsweb.cern.ch/dmwmmon/datasvc/doc/auth
From the command line you can check results using wget or phedex provided utility with your proxy-certificate:
# grid-proxy-init -rfc -bits 1024
# myproxy=`grid-proxy-info -path`
# wget --certificate=$myproxy --private-key=$myproxy --ca-certificate=$myproxy --ca-directory=/etc/grid-security/certificates -O - https://cmsweb.cern.ch/dmwmmon/datasvc/perl/auth
$VAR1 = {
'PHEDEX' => {
'REQUEST_DATE' => '2014-11-05 23:26:16 UTC',
'REQUEST_CALL' => 'auth',
'REQUEST_URL' => 'http://cmsweb.cern.ch:8280/dmwmmon/datasvc/perl/auth',
'CALL_TIME' => '9e-05',
'REQUEST_VERSION' => '1.0.2-comp',
'REQUEST_TIMESTAMP' => '1415229976.52901',
'INSTANCE' => 'read',
'AUTH' => [
{
'USERNAME' => 'Natalia Ratnikova',
'STATE' => 'cert',
'DN' => '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ratnik/CN=577888/CN=Natalia Ratnikova',
'ROLE' => [
{
'NAME' => 'developer',
'GROUP' => 'phedex'
},
{
'NAME' => 'site admin',
'GROUP' => 't1-us-fnal'
},
{
'NAME' => 'site admin',
'GROUP' => 't1-us-fnal-disk'
},
{
'NAME' => 'site admin',
'GROUP' => 't3-us-fnallpc'
},
{
'NAME' => 'phedex contact',
'GROUP' => 't1-us-fnal'
},
{
'NAME' => 'phedex contact',
'GROUP' => 't1-us-fnal-disk'
},
{
'NAME' => 'phedex contact',
'GROUP' => 't3-us-fnallpc'
},
{
'NAME' => 'data manager',
'GROUP' => 't1-us-fnal'
},
{
'NAME' => 'data manager',
'GROUP' => 't1-us-fnal-disk'
},
{
'NAME' => 'data manager',
'GROUP' => 't3-us-fnallpc'
}
],
'ABILITY' => undef,
'EMAIL' => 'Natalia.Ratnikova@cern.ch'
}
]
}
};
# phedex --cert_file=$myproxy --key_file=$myproxy --instance=prod --format=perl --ca_file=$myproxy --ca_dir=/etc/grid-security/certificates auth
$VAR1 = {
'PHEDEX' => {
'REQUEST_DATE' => '2014-11-05 23:25:58 UTC',
'REQUEST_CALL' => 'auth',
'REQUEST_URL' => 'http://cmsweb.cern.ch:7001/phedex/datasvc/perl/prod/Auth',
'CALL_TIME' => '0.04579',
'REQUEST_VERSION' => '2.3.19-comp',
'REQUEST_TIMESTAMP' => '1415229958.1619',
'INSTANCE' => 'prod',
'AUTH' => [
{
'USERNAME' => 'Natalia Ratnikova',
'STATE' => 'cert',
'DN' => '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ratnik/CN=577888/CN=Natalia Ratnikova',
'ROLE' => [
{
'NAME' => 'developer',
'GROUP' => 'phedex'
},
{
'NAME' => 'site admin',
'GROUP' => 't1_us_fnal'
},
{
'NAME' => 'site admin',
'GROUP' => 't1_us_fnal_disk'
},
{
'NAME' => 'site admin',
'GROUP' => 't3_us_fnallpc'
},
{
'NAME' => 'phedex contact',
'GROUP' => 't1_us_fnal'
},
{
'NAME' => 'phedex contact',
'GROUP' => 't1_us_fnal_disk'
},
{
'NAME' => 'phedex contact',
'GROUP' => 't3_us_fnallpc'
},
{
'NAME' => 'data manager',
'GROUP' => 't1_us_fnal'
},
{
'NAME' => 'data manager',
'GROUP' => 't1_us_fnal_disk'
},
{
'NAME' => 'data manager',
'GROUP' => 't3_us_fnallpc'
}
],
'ABILITY' => undef,
'EMAIL' => 'Natalia.Ratnikova@cern.ch',
'NODE' => [
{
'ID' => '10',
'NAME' => 'T1_US_FNAL_MSS'
},
{
'ID' => '881',
'NAME' => 'T3_US_FNALLPC'
},
{
'ID' => '9',
'NAME' => 'T1_US_FNAL_Buffer'
},
{
'ID' => '1781',
'NAME' => 'T1_US_FNAL_Disk'
}
]
}
]
}
};
Or use direct openssl connection for debugging non-client related issues:
# openssl s_client -connect cmsweb.cern.ch:443 -showcerts -prexit -cert $myproxy -key $myproxy -CAfile $myproxy -state
And type in the HTTP command:
---
GET /dmwmmon/datasvc/perl/auth
See
man s_client
for more details.
Additional hints
- On CERN machines wget option
--ca-directory=/etc/grid-security/certificates
is not necessary.
-
vomp-proxy-init
and old versions of grid-proxy-init
may create wrong proxy type
and strength, use -rfc -bits 1024
options to force required settings.
- Useful examples can be found in CERN PhEDEx operations twiki.
Development environment setup
- Login to the node where you already have PhEDEx release installed, or install it following the standard procedure. No need to configure anything, we just need to get the externals.
- Clone latest development code from PhEDEx git repo:
git clone git@github.com:dmwm/PHEDEX.git
- Set up the environment. This is an example of development environment installed in cern afs public area:
unset PERL5LIB
source /afs/cern.ch/user/r/ratnik/public/SpaceMon/slc6_amd64_gcc461/cms/PHEDEX/4.1.3-comp3/etc/profile.d/init.sh
export PHEDEX_ROOT=/afs/cern.ch/user/r/ratnik/public/SpaceMon/PHEDEX
export PERL5LIB=$PHEDEX_ROOT/perl_lib:$PERL5LIB
export PATH=$PHEDEX_ROOT/Utilities:$PATH
Replace locations with a path to your PhEDEx release installation and PHEDEX code respectively.
- Now you should be able to run
spaceInsert -h
spacecount -h
and use this environment for developing utilities and agents.
- For Data Service development environment see:
Implementing parser for a new dump format.
Guidelines for developing plugins for storage dump formats:
See also
SpaceMon README file
.
- choose a proper name for a new format, here we call it MYFORMAT
- create a module Format::MYFORMAT and implement the following functions:
new, formattingHelp, and lookupFileSize (see XML, TXT formats)
- include examples of recognized syntaxes in formattingHelp
- lookupFileSize returns a tuple: ($pfn, $size)
- if fail to parse the line, call formattingHelp and die with formatting error
- the verification of the $pfn and $size values doesn't need to be implemented for each format; it happens at the higher level in the Aggregate class
Example (KIT format).
As an example, we take a format produced by
chimera-list tool
, developed at KIT, see
example on github
.
There are three modules involved in this task:
We name our new format
KIT
and create a new plugin module
KIT.pm
.
Meetings minutes
PhEDEx developers meeting June 11, 2014
- Authentication problem at sites
- Four sites reported 401 error during upload. Test results differ: could be a combination of problems.
- Looks like CA is a culprit . Tony: one site passed their proxy as CA info and that helped (T2_CH_CSCS ?)
- spacecount/spaceInsert currently do not have an option to pass the CA file
- Natalia was able to reproduce the problem with a strange certificate, she will test the solution and see if option can be added
- Certificate should be the one registered in the SiteDB. If not, server might give the same type of failure.
- If unknown CA is confirmed to be the main source of problem, and a quick solution exists, there is no need for a test suite
- Support for aggregation by the sites
- several site admins expressed the idea of aggregating storage usage by their own tools
- we want to have this possibility open in the future
- question is where the data type matching will happen, and we need an API for that.
- we agreed that it will be handled by a configuration component, which comes between aggregation and before the insert API
- the spaceInsert API will have additional validation step to ensure no garbage is uploaded
- Volatile data types
- AGIS, Gratia, Visualization, etc
- AGIS is based on Django, which can provide tabular views for relational DB
- Tony will look into this after collaboration week in Bolonya
- Django also works with various plotting plugins
- Ken asked for a simple utility as an example of using SpaceMon APIs. It could be a simple python script to show free space across the sites (diff between used space and site pledges ). This work could be done by someone from the dataops.
Phase II development/testing
Development servers
SpaceMon data service development instance is currently running on CERN VM:
http://spacemon-dev.cern.ch/dmwmmon/datasvc
.
It is behind the firewall, so you need to access from lxplus directly or via a tunnel.
NOTE: The spacemon-dev VM running SLC5 OS, expires on 30-Nov-2015.
A new spacemon-dev-sl6 server has been deployed on a CERN openstack VM.
Additional VM based server is envisioned for testing a new architecture.
To test access to the server (e.g. spacemon-dev):
wget --no-check-certificate -O - http://spacemon-dev.cern.ch/dmwmmon/datasvc/perl/bounce
Computing proposal on name space
at Weekly Joint PPD/Off/Trig/Comp Ops Meeting, Thursday Dec 11th . 2014.
See also
DMWM development documentation
.
Basic testing of HTTP-group deployed releases
Production DMWMMON data service release is deployed on cmsweb.cern.ch. The cmsweb-testbed and cmsweb-dev servers are used for testing development and integration.
To check basic functionality, try these steps:
- First release of spacemon-client 1.0.1 has been frozen on Aug 29th and deployed on CVMFS on Sept 27th, 2016 after testing and validating at CERN CMS Tier 2 .
- At of end of October, there is no formal release note for this release. I may write one and create a twiki page dedicated for the release notes, changelog and such. Alternatively we could use github "releases" feature and link it to CMS SpaceMon documentation in CERN twiki.
--
NataliaRatnikova - 14 Feb 2014