CMS Space Monitoring development

Complete: 3

This page is for CMS SpaceMon developers.

Code management

Second prototype of SpaceMon has been implemented within PhEDEx repository

 git@github.com:dmwm/PHEDEX.git 

Code lives in the following locations:

Original location Description of the code component New location
PHEDEX/Contrib/StorageDumps tools for producing storage dumps collected during T2 consistency tools validation campaign DMWMMON/SiteInfoProviders
PHEDEX/Utilities/spacecount standalone utility for testing SpaceMon for different technologies ( analog of setool for space accounting )  
PHEDEX/Utilities/testSpace examples for SpaceMon prototype developed by Xiaomei  
PHEDEX/perl_lib/DMWMMON/StorageAccounting/Core.pm interface to dmwmmon/datasvc (uploadRecord) DMWMMON/SpaceMon
PHEDEX/perl_lib/PHEDEX/Namespace/SpaceCountCommon.pm library of common functions for processing storage dumps DMWMMON/SpaceMon
PHEDEX/perl_lib/PHEDEX/Namespace/*/spacecount.pm technology specific plugins for space counting  
PHEDEX/perl_lib/PHEDEX/Web/API/StorageInsert.pm API for uploading storage usage info into DMWMMON database  
PHEDEX/perl_lib/PHEDEX/Web/API/StorageUsage.pm API for querying DMWMMON database for storage usage info  
PHEDEX/perl_lib/PHEDEX/Web/SQLSpace.pm SQL queries for dmwmmon/datasvc  

The plan is to migrate SpaceMon into DMWMMON repository on github:

 git@github.com:dmwm/DMWMMON.git

Ask Tony, Nicolo, Erik, Giulio for access details.

Git configuration

Here is Natalia's git config for SpaceMon development with added remote upstream branch for merging in changes from the master branch in PhEDEx github repo.

$ cat .git/config 
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = https://github.com/nataliaratnikova/PHEDEX.git
[branch "master"]
	remote = origin
	merge = refs/heads/master
[remote "upstream"]
	url = https://github.com/dmwm/PHEDEX.git
	fetch = +refs/heads/*:refs/remotes/upstream/*

The order of commands is something like this:

  1. make sure work area is clean -> everything is committed
  2. git branch -vv make sure you are on local master branch and tracking origin master branch
  3. git push push changes to remote private github repo
  4. git pull --rebase upstream master get in changes from the upstream repo : you may see branches diverge,
  5. git merge origin/master this should cause no conflicts, as long as SpaceMon is developed in private github repo, you may still be ahead of origin master, but no longer diverged.
  6. git fetch  upstream master (is it really needed?)
  7. git push everything should be consistent now.

Access to Information Store

Access to Information Store (DMWMMON Database) is realized via PhEDEx Data Service interface, and uses certificate based authentication with additional rules for write access based on SiteDb roles information. Particularly site admin role for the node in question is required, see current implementation.

You can verify your authentication state and abilities by Data Service auth request url: https://cmsweb.cern.ch/dmwmmon/datasvc/perl/auth as documented here: https://cmsweb.cern.ch/dmwmmon/datasvc/doc/auth

From the command line you can check results using wget or phedex provided utility with your proxy-certificate:

  
# grid-proxy-init -rfc -bits 1024
# myproxy=`grid-proxy-info -path`
# wget --certificate=$myproxy --private-key=$myproxy --ca-certificate=$myproxy  --ca-directory=/etc/grid-security/certificates -O - https://cmsweb.cern.ch/dmwmmon/datasvc/perl/auth
$VAR1 = {
          'PHEDEX' => {
                        'REQUEST_DATE' => '2014-11-05 23:26:16 UTC',
                        'REQUEST_CALL' => 'auth',
                        'REQUEST_URL' => 'http://cmsweb.cern.ch:8280/dmwmmon/datasvc/perl/auth',
                        'CALL_TIME' => '9e-05',
                        'REQUEST_VERSION' => '1.0.2-comp',
                        'REQUEST_TIMESTAMP' => '1415229976.52901',
                        'INSTANCE' => 'read',
                        'AUTH' => [
                                    {
                                      'USERNAME' => 'Natalia Ratnikova',
                                      'STATE' => 'cert',
                                      'DN' => '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ratnik/CN=577888/CN=Natalia Ratnikova',
                                      'ROLE' => [
                                                  {
                                                    'NAME' => 'developer',
                                                    'GROUP' => 'phedex'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't1-us-fnal'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't1-us-fnal-disk'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't3-us-fnallpc'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't1-us-fnal'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't1-us-fnal-disk'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't3-us-fnallpc'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't1-us-fnal'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't1-us-fnal-disk'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't3-us-fnallpc'
                                                  }
                                                ],
                                      'ABILITY' => undef,
                                      'EMAIL' => 'Natalia.Ratnikova@cern.ch'
                                    }
                                  ]
                      }
        };
# phedex --cert_file=$myproxy --key_file=$myproxy --instance=prod --format=perl --ca_file=$myproxy  --ca_dir=/etc/grid-security/certificates  auth
$VAR1 = {
          'PHEDEX' => {
                        'REQUEST_DATE' => '2014-11-05 23:25:58 UTC',
                        'REQUEST_CALL' => 'auth',
                        'REQUEST_URL' => 'http://cmsweb.cern.ch:7001/phedex/datasvc/perl/prod/Auth',
                        'CALL_TIME' => '0.04579',
                        'REQUEST_VERSION' => '2.3.19-comp',
                        'REQUEST_TIMESTAMP' => '1415229958.1619',
                        'INSTANCE' => 'prod',
                        'AUTH' => [
                                    {
                                      'USERNAME' => 'Natalia Ratnikova',
                                      'STATE' => 'cert',
                                      'DN' => '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ratnik/CN=577888/CN=Natalia Ratnikova',
                                      'ROLE' => [
                                                  {
                                                    'NAME' => 'developer',
                                                    'GROUP' => 'phedex'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't1_us_fnal'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't1_us_fnal_disk'
                                                  },
                                                  {
                                                    'NAME' => 'site admin',
                                                    'GROUP' => 't3_us_fnallpc'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't1_us_fnal'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't1_us_fnal_disk'
                                                  },
                                                  {
                                                    'NAME' => 'phedex contact',
                                                    'GROUP' => 't3_us_fnallpc'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't1_us_fnal'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't1_us_fnal_disk'
                                                  },
                                                  {
                                                    'NAME' => 'data manager',
                                                    'GROUP' => 't3_us_fnallpc'
                                                  }
                                                ],
                                      'ABILITY' => undef,
                                      'EMAIL' => 'Natalia.Ratnikova@cern.ch',
                                      'NODE' => [
                                                  {
                                                    'ID' => '10',
                                                    'NAME' => 'T1_US_FNAL_MSS'
                                                  },
                                                  {
                                                    'ID' => '881',
                                                    'NAME' => 'T3_US_FNALLPC'
                                                  },
                                                  {
                                                    'ID' => '9',
                                                    'NAME' => 'T1_US_FNAL_Buffer'
                                                  },
                                                  {
                                                    'ID' => '1781',
                                                    'NAME' => 'T1_US_FNAL_Disk'
                                                  }
                                                ]
                                    }
                                  ]
                      }
        };

Or use direct openssl connection for debugging non-client related issues:

 # openssl s_client -connect cmsweb.cern.ch:443 -showcerts -prexit -cert $myproxy -key $myproxy -CAfile $myproxy -state 
And type in the HTTP command:
---
GET /dmwmmon/datasvc/perl/auth
See man s_client for more details.

Additional hints

  • On CERN machines wget option --ca-directory=/etc/grid-security/certificates is not necessary.
  • vomp-proxy-init and old versions of grid-proxy-init may create wrong proxy type and strength, use -rfc -bits 1024 options to force required settings.
  • Useful examples can be found in CERN PhEDEx operations twiki.

Development environment setup

  1. Login to the node where you already have PhEDEx release installed, or install it following the standard procedure. No need to configure anything, we just need to get the externals.
  2. Clone latest development code from PhEDEx git repo:
    git clone git@github.com:dmwm/PHEDEX.git
  3. Set up the environment. This is an example of development environment installed in cern afs public area:
    unset PERL5LIB
    source /afs/cern.ch/user/r/ratnik/public/SpaceMon/slc6_amd64_gcc461/cms/PHEDEX/4.1.3-comp3/etc/profile.d/init.sh
    export PHEDEX_ROOT=/afs/cern.ch/user/r/ratnik/public/SpaceMon/PHEDEX
    export PERL5LIB=$PHEDEX_ROOT/perl_lib:$PERL5LIB
    export PATH=$PHEDEX_ROOT/Utilities:$PATH
    
    Replace locations with a path to your PhEDEx release installation and PHEDEX code respectively.
  4. Now you should be able to run
    spaceInsert -h
    spacecount -h
    and use this environment for developing utilities and agents.

  5. For Data Service development environment see:

Implementing parser for a new dump format.

Guidelines for developing plugins for storage dump formats:

See also SpaceMon README file.
  • choose a proper name for a new format, here we call it MYFORMAT
  • create a module Format::MYFORMAT and implement the following functions:
    new, formattingHelp, and lookupFileSize (see XML, TXT formats)
  • include examples of recognized syntaxes in formattingHelp
  • lookupFileSize returns a tuple: ($pfn, $size)
  • if fail to parse the line, call formattingHelp and die with formatting error
  • the verification of the $pfn and $size values doesn't need to be implemented for each format; it happens at the higher level in the Aggregate class

Example (KIT format).

As an example, we take a format produced by chimera-list tool, developed at KIT, see example on github.

There are three modules involved in this task:

We name our new format KIT and create a new plugin module KIT.pm .

Meetings minutes

PhEDEx developers meeting June 11, 2014

  • Authentication problem at sites
    • Four sites reported 401 error during upload. Test results differ: could be a combination of problems.
    • Looks like CA is a culprit . Tony: one site passed their proxy as CA info and that helped (T2_CH_CSCS ?)
    • spacecount/spaceInsert currently do not have an option to pass the CA file
    • Natalia was able to reproduce the problem with a strange certificate, she will test the solution and see if option can be added
    • Certificate should be the one registered in the SiteDB. If not, server might give the same type of failure.
    • If unknown CA is confirmed to be the main source of problem, and a quick solution exists, there is no need for a test suite
  • Support for aggregation by the sites
    • several site admins expressed the idea of aggregating storage usage by their own tools
    • we want to have this possibility open in the future
    • question is where the data type matching will happen, and we need an API for that.
    • we agreed that it will be handled by a configuration component, which comes between aggregation and before the insert API
    • the spaceInsert API will have additional validation step to ensure no garbage is uploaded
  • Volatile data types
    • we discussed how flexible data types (DT)should be.
    • two cases when DT may need to be changed:
      1. New DT is introduced, and there is matching data earlier assigned to "Unknown" type.
      2. CMS Namespace conventions change. 
    • conclusion: yes, we want this flexibility , and will need an API to rename/reassign DT.
  • AGIS, Gratia, Visualization, etc
    • AGIS is based on Django, which can provide tabular views for relational DB
    • Tony will look into this after collaboration week in Bolonya
    • Django also works with various plotting plugins
    • Ken asked for a simple utility as an example of using SpaceMon APIs. It could be a simple python script to show free space across the sites (diff between used space and site pledges ). This work could be done by someone from the dataops.

Phase II development/testing

Development servers

SpaceMon data service development instance is currently running on CERN VM: http://spacemon-dev.cern.ch/dmwmmon/datasvc . It is behind the firewall, so you need to access from lxplus directly or via a tunnel.
NOTE: The spacemon-dev VM running SLC5 OS, expires on 30-Nov-2015.
A new spacemon-dev-sl6 server has been deployed on a CERN openstack VM.
Additional VM based server is envisioned for testing a new architecture.

To test access to the server (e.g. spacemon-dev):

  
 wget --no-check-certificate -O -  http://spacemon-dev.cern.ch/dmwmmon/datasvc/perl/bounce
Computing proposal on name space at Weekly Joint PPD/Off/Trig/Comp Ops Meeting, Thursday Dec 11th . 2014.

See also DMWM development documentation.

Basic testing of HTTP-group deployed releases

Production DMWMMON data service release is deployed on cmsweb.cern.ch. The cmsweb-testbed and cmsweb-dev servers are used for testing development and integration. To check basic functionality, try these steps:

  • Login to lxplus.cern.ch
  • Set up spacemon environment
  • Run spacemon-test suite, it has no options or arguments. This will create test data and run a few basic tests against production server.
  • Look at the output of the previous step and rerun spacemon upload commands found there.
  • Rerun spacemon with
    --url https://cmsweb-testbed.cern.ch/dmwmmon/datasvc
    and compare to the output from the previous step as a reference.

SpaceMon client releases

  • First release of spacemon-client 1.0.1 has been frozen on Aug 29th and deployed on CVMFS on Sept 27th, 2016 after testing and validating at CERN CMS Tier 2 .
  • At of end of October, there is no formal release note for this release. I may write one and create a twiki page dedicated for the release notes, changelog and such. Alternatively we could use github "releases" feature and link it to CMS SpaceMon documentation in CERN twiki.

See also internal development work logs.

-- NataliaRatnikova - 14 Feb 2014

Edit | Attach | Watch | Print version | History: r18 < r17 < r16 < r15 < r14 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r18 - 2016-10-26 - NataliaRatnikova
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback