CMSPublic Web>CMSPublicData>CompProjOffice>CompProjSpaceMon>SpaceMonDevelopment (2016-10-26, NataliaRatnikova)

EditAttachPDF

CMS Space Monitoring development

Complete:

This page is for CMS SpaceMon developers.

CMS Space Monitoring development
Phase II development/testing

Code management

Second prototype of SpaceMon has been implemented within PhEDEx repository

 git@github.com:dmwm/PHEDEX.git

Code lives in the following locations:

Original location	Description of the code component	New location
PHEDEX/Contrib/StorageDumps	tools for producing storage dumps collected during T2 consistency tools validation campaign	DMWMMON/SiteInfoProviders
PHEDEX/Utilities/spacecount	standalone utility for testing SpaceMon for different technologies ( analog of setool for space accounting )
PHEDEX/Utilities/testSpace	examples for SpaceMon prototype developed by Xiaomei
PHEDEX/perl_lib/DMWMMON/StorageAccounting/Core.pm	interface to dmwmmon/datasvc (uploadRecord)	DMWMMON/SpaceMon
PHEDEX/perl_lib/PHEDEX/Namespace/SpaceCountCommon.pm	library of common functions for processing storage dumps	DMWMMON/SpaceMon
PHEDEX/perl_lib/PHEDEX/Namespace/*/spacecount.pm	technology specific plugins for space counting
PHEDEX/perl_lib/PHEDEX/Web/API/StorageInsert.pm	API for uploading storage usage info into DMWMMON database
PHEDEX/perl_lib/PHEDEX/Web/API/StorageUsage.pm	API for querying DMWMMON database for storage usage info
PHEDEX/perl_lib/PHEDEX/Web/SQLSpace.pm	SQL queries for dmwmmon/datasvc

The plan is to migrate SpaceMon into DMWMMON repository on github:

 git@github.com:dmwm/DMWMMON.git

Ask Tony, Nicolo, Erik, Giulio for access details.

Git configuration

Here is Natalia's git config for SpaceMon development with added remote upstream branch for merging in changes from the master branch in PhEDEx github repo.

$ cat .git/config 
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = https://github.com/nataliaratnikova/PHEDEX.git
[branch "master"]
	remote = origin
	merge = refs/heads/master
[remote "upstream"]
	url = https://github.com/dmwm/PHEDEX.git
	fetch = +refs/heads/*:refs/remotes/upstream/*

The order of commands is something like this:

make sure work area is clean -> everything is committed
git branch -vv make sure you are on local master branch and tracking origin master branch
git push push changes to remote private github repo
git pull --rebase upstream master get in changes from the upstream repo : you may see branches diverge,
git merge origin/master this should cause no conflicts, as long as SpaceMon is developed in private github repo, you may still be ahead of origin master, but no longer diverged.
git fetch upstream master (is it really needed?)
git push everything should be consistent now.

Access to Information Store

Access to Information Store (DMWMMON Database) is realized via PhEDEx Data Service interface, and uses certificate based authentication with additional rules for write access based on SiteDb roles information. Particularly site admin role for the node in question is required, see current implementation

You can verify your authentication state and abilities by Data Service auth request url: https://cmsweb.cern.ch/dmwmmon/datasvc/perl/auth as documented here: https://cmsweb.cern.ch/dmwmmon/datasvc/doc/auth

From the command line you can check results using wget or phedex provided utility with your proxy-certificate:

  
# grid-proxy-init -rfc -bits 1024
# myproxy=`grid-proxy-info -path`
# wget --certificate=$myproxy --private-key=$myproxy --ca-certificate=$myproxy  --ca-directory=/etc/grid-security/certificates -O - https://cmsweb.cern.ch/dmwmmon/datasvc/perl/auth

Example of Output

Hide

$VAR1 = {
          'PHEDEX' => {
                        'REQUEST_DATE' => '2014-11-05 23:26:16 UTC',
                        'REQUEST_CALL' => 'auth',
                        'REQUEST_URL' => 'http://cmsweb.cern.ch:8280/dmwmmon/datasvc/perl/auth',
                        'CALL_TIME' => '9e-05',
                        'REQUEST_VERSION' => '1.0.2-comp',
                        'REQUEST_TIMESTAMP' => '1415229976.52901',
                        'INSTANCE' => 'read',
                        'AUTH' => [
                                    {
                                      'USERNAME' => 'Natalia Ratnikova',
                                      'STATE' => 'cert',
                                      'DN' => '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ratnik/CN=577888/CN=Natalia Ratnikova',
                                      'ROLE' => [
                                                  {
                                                    'NAME' => 'developer',
                                                    'GROUP' => 'phedex'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't1-us-fnal'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't1-us-fnal-disk'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't3-us-fnallpc'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't1-us-fnal'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't1-us-fnal-disk'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't3-us-fnallpc'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't1-us-fnal'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't1-us-fnal-disk'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't3-us-fnallpc'
                                                  }
                                                ],
                                      'ABILITY' => undef,
                                      'EMAIL' => 'Natalia.Ratnikova@cern.ch'
                                    }
                                  ]
                      }
        };

# phedex --cert_file=$myproxy --key_file=$myproxy --instance=prod --format=perl --ca_file=$myproxy  --ca_dir=/etc/grid-security/certificates  auth

Example of Output

Hide

$VAR1 = {
          'PHEDEX' => {
                        'REQUEST_DATE' => '2014-11-05 23:25:58 UTC',
                        'REQUEST_CALL' => 'auth',
                        'REQUEST_URL' => 'http://cmsweb.cern.ch:7001/phedex/datasvc/perl/prod/Auth',
                        'CALL_TIME' => '0.04579',
                        'REQUEST_VERSION' => '2.3.19-comp',
                        'REQUEST_TIMESTAMP' => '1415229958.1619',
                        'INSTANCE' => 'prod',
                        'AUTH' => [
                                    {
                                      'USERNAME' => 'Natalia Ratnikova',
                                      'STATE' => 'cert',
                                      'DN' => '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ratnik/CN=577888/CN=Natalia Ratnikova',
                                      'ROLE' => [
                                                  {
                                                    'NAME' => 'developer',
                                                    'GROUP' => 'phedex'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't1_us_fnal'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't1_us_fnal_disk'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't3_us_fnallpc'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't1_us_fnal'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't1_us_fnal_disk'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't3_us_fnallpc'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't1_us_fnal'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't1_us_fnal_disk'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't3_us_fnallpc'
                                                  }
                                                ],
                                      'ABILITY' => undef,
                                      'EMAIL' => 'Natalia.Ratnikova@cern.ch',
                                      'NODE' => [
                                                  {
                                                    'ID' => '10',
                                                    'NAME' => 'T1_US_FNAL_MSS'
                                                  },
                                                  {
                                                    'ID' => '881',
                                                    'NAME' => 'T3_US_FNALLPC'
                                                  },
                                                  {
                                                    'ID' => '9',
                                                    'NAME' => 'T1_US_FNAL_Buffer'
                                                  },
                                                  {
                                                    'ID' => '1781',
                                                    'NAME' => 'T1_US_FNAL_Disk'
                                                  }
                                                ]
                                    }
                                  ]
                      }
        };

Or use direct openssl connection for debugging non-client related issues:

 # openssl s_client -connect cmsweb.cern.ch:443 -showcerts -prexit -cert $myproxy -key $myproxy -CAfile $myproxy -state

And type in the HTTP command:

---
GET /dmwmmon/datasvc/perl/auth

See man s_client for more details.

Additional hints

On CERN machines wget option --ca-directory=/etc/grid-security/certificates is not necessary.
vomp-proxy-init and old versions of grid-proxy-init may create wrong proxy type and strength, use -rfc -bits 1024 options to force required settings.
Useful examples can be found in CERN PhEDEx operations twiki.

Development environment setup

Login to the node where you already have PhEDEx release installed, or install it following the standard procedure. No need to configure anything, we just need to get the externals.
Clone latest development code from PhEDEx git repo:
```
git clone git@github.com:dmwm/PHEDEX.git
```

Set up the environment. This is an example of development environment installed in cern afs public area:

unset PERL5LIB
source /afs/cern.ch/user/r/ratnik/public/SpaceMon/slc6_amd64_gcc461/cms/PHEDEX/4.1.3-comp3/etc/profile.d/init.sh
export PHEDEX_ROOT=/afs/cern.ch/user/r/ratnik/public/SpaceMon/PHEDEX
export PERL5LIB=$PHEDEX_ROOT/perl_lib:$PERL5LIB
export PATH=$PHEDEX_ROOT/Utilities:$PATH

Replace locations with a path to your PhEDEx release installation and PHEDEX code respectively.

Now you should be able to run
```
spaceInsert -h
spacecount -h
```
and use this environment for developing utilities and agents.
For Data Service development environment see:
- PhEDEx Data Service development
- PhEDEx deployment in virtual machine

Implementing parser for a new dump format.

Guidelines for developing plugins for storage dump formats:

Example (KIT format).

As an example, we take a format produced by chimera-list tool, developed at KIT, see example on github.

There are three modules involved in this task:

StorageDump base class,
FormatFactory.pm module instantiating the parser,
The proper format specific plugin, derived from the StorageDump class, which implements the actual parsing rules.
Examples: Format/TXT.pm, Format/XML.pm.

We name our new format KIT and create a new plugin module KIT.pm .

Meetings minutes

PhEDEx developers meeting June 11, 2014

Authentication problem at sites
- Four sites reported 401 error during upload. Test results differ: could be a combination of problems.
- Looks like CA is a culprit . Tony: one site passed their proxy as CA info and that helped (T2_CH_CSCS ?)
- spacecount/spaceInsert currently do not have an option to pass the CA file
- Natalia was able to reproduce the problem with a strange certificate, she will test the solution and see if option can be added
- Certificate should be the one registered in the SiteDB. If not, server might give the same type of failure.
- If unknown CA is confirmed to be the main source of problem, and a quick solution exists, there is no need for a test suite
Support for aggregation by the sites
- several site admins expressed the idea of aggregating storage usage by their own tools
- we want to have this possibility open in the future
- question is where the data type matching will happen, and we need an API for that.
- we agreed that it will be handled by a configuration component, which comes between aggregation and before the insert API
- the spaceInsert API will have additional validation step to ensure no garbage is uploaded
Volatile data types
- we discussed how flexible data types (DT)should be.
- two cases when DT may need to be changed:
```
1. New DT is introduced, and there is matching data earlier assigned to "Unknown" type.
2. CMS Namespace conventions change. 
```
- conclusion: yes, we want this flexibility , and will need an API to rename/reassign DT.
AGIS, Gratia, Visualization, etc
- AGIS is based on Django, which can provide tabular views for relational DB
- Tony will look into this after collaboration week in Bolonya
- Django also works with various plotting plugins
- Ken asked for a simple utility as an example of using SpaceMon APIs. It could be a simple python script to show free space across the sites (diff between used space and site pledges ). This work could be done by someone from the dataops.

Phase II development/testing

Development servers

SpaceMon data service development instance is currently running on CERN VM: http://spacemon-dev.cern.ch/dmwmmon/datasvc . It is behind the firewall, so you need to access from lxplus directly or via a tunnel.
NOTE: The spacemon-dev VM running SLC5 OS, expires on 30-Nov-2015.
A new spacemon-dev-sl6 server has been deployed on a CERN openstack VM.
Additional VM based server is envisioned for testing a new architecture.

To test access to the server (e.g. spacemon-dev):

  
 wget --no-check-certificate -O -  http://spacemon-dev.cern.ch/dmwmmon/datasvc/perl/bounce

Computing proposal on name space

at Weekly Joint PPD/Off/Trig/Comp Ops Meeting, Thursday Dec 11th . 2014.

Basic testing of HTTP-group deployed releases

Production DMWMMON data service release is deployed on cmsweb.cern.ch. The cmsweb-testbed and cmsweb-dev servers are used for testing development and integration. To check basic functionality, try these steps:

Login to lxplus.cern.ch
Set up spacemon environment
Run spacemon-test suite, it has no options or arguments. This will create test data and run a few basic tests against production server.
Look at the output of the previous step and rerun spacemon upload commands found there.
Rerun spacemon with
```
--url https://cmsweb-testbed.cern.ch/dmwmmon/datasvc
```
and compare to the output from the previous step as a reference.

SpaceMon client releases

First release of spacemon-client 1.0.1 has been frozen on Aug 29th and deployed on CVMFS on Sept 27th, 2016 after testing and validating at CERN CMS Tier 2 .
At of end of October, there is no formal release note for this release. I may write one and create a twiki page dedicated for the release notes, changelog and such. Alternatively we could use github "releases" feature and link it to CMS SpaceMon documentation in CERN twiki.