VOSpecificServicesMon

Introduction (and before start)

This page contains information remaining the project NAME, that consist in create a dashboard package that contains several scripts for monitoring different services.

BEFORE START take a look to a brief introduction of how to use and develop your own collectors (services) using dashboard could be found in BriefIntroducctionDashboardCollector.

In the rest of the page we'll present the current implementation of the NAME agent and how to manage this service.

List of services implemented

This is a list with the scripts that are running, with services they're monitoring and the general configuration

show hide

VOBOX

Check the DQ2 results:

Files_State_NEW_Number
Files_State_QUEUED_Number
Files_State_STAGING_Number
Files_State_STAGE_FAILED_Number
Files_State_STAGED_Number
Files_State_COPYING_Number
Files_State_COPY_FAILED_Number
Files_State_COPIED_Number
Files_State_REGISTER_FAILED_Number
Files_State_DONE_LastHour
ARDACallbacks_Number
Files_Number

and the lemon metrics 4031,4032, 4033, 4034

Central Catalog monitor

Check the webalizer parameters (repository, content, location, subscript and tracer) and lemon metrics 4035 and 4036

Panda_Mon Monitor

PandaServer_Monitor

VOALICE vobox monitor

for developers: Add a new monitoring service to the package

Add a new script for the package is quite easy, you need to add the .py files to the lib directory and modify the VoboxAgent.py script for call your script from there and get the parameters.

• Checkout the code
• Add your python script to PACKAGE/lib/dashboard/collector/vobox/ directory
• Edit the PACKAGE/lib/dashboard/collector/vobox/VoboxAgent.py to create a class that inherit from Service
• Create an service-config.xml example
• Create the service directory logs.
• Remember to ADD all the new files to the CVS

Add your python script

Just copy your script under the lib directory in the correct location (normally PACKAGE/lib/dashboard/collector/vobox/)

Calling the script from the service

You need to modify VoboxAgent.py )_PACKAGE/lib/dashboard/collector/vobox/_ and add a new class that inherit from the Service class. If remember from the BriefIntroducctionDashboardCollector, this file read the parameters from the service-config.xml and execute the python script.

Follow the example of other class from that file. Follow this instruccion is you need more help.

Creating config files

The convention is that for every new script should be defined a new service group name. This means have a service group section for every monitoring script and inside this group add all the machines needed to be monitoring, example:

  <service-group name="VOBoxSiteServices.monitor">
        <services>
            <service name="atlddm11.vobox.monitor"....
        </services>
  </service-group>
  <service-group name="cc_monitor.service">
        <services>
            <service name="atlddm24.cc_monitor.service"....
        </services>
  </service-group>

Save that file into PACKAGE/config/service-config/ directory.

One is defined a new service-group name is necessary to create a new configuration directory that would be under /opt/dashboard/etc/dashboard-service-config/<service-group-name>/ where the logging information is saved. Place that directory into PACKAGE/config/service-config/. Inside the service group directory should be placed an etc/ directory, and inside the logging files: logging.cfg and dashboard.cfg. Take a look at the existing directory as an example.

Edit the dashboard.cfg to specify the logging.cfg path (usually /opt/dashboard/etc/dashboard-service-config/<service-group-name>/etc/logging.cfg

Edit the loggin.cfg to specify where the logs should be safe (section handler_root_file and handler_dashboard_file).

Don't forget to modify the setup.py for add this directory to the package.

Create an service-config.xml example

Place your xml example under config/service/config with a name associated to the service package (like <service-group-name>.xml). Once the package is installed the example would be copied to /opt/dashboard/etc/dashboard-service-config/

Make a release

The release system of dashboard follows the normal convention for software release. There are major, minor, path and candidate versions:

faraox@faraox-laptop:~/cern/arda.dashboard.service-monitor-sls$ python setup.py release --help
....
Options for 'release' command:
  --major (-M)          major version release
  --minor (-m)          minor version release
  --patch (-p)          patch release
  --candidate (-c)      candidate release
  --release-notes (-r)  location of the RELEASE-NOTES file

You could start a major or minor version, and then keep making candidate versions. Once do that and you're happy with your code, make a release (without the a release option).

In this example we want to make a new candidate release for the version 0.1.0:

faraox@faraox-laptop:~/cern/arda.dashboard.service-monitor-sls$ python setup.py release -c
running release
Current module version is 0.1.0_rc2. Released version will be 0.1.0_rc3.
Continue with release? [Y/N]: Y
Tag will be vobox-agent-siteservices-0-1-0_rc3. Continue [Y/N]: Y
Released version 0.1.0_rc3
faraox@faraox-laptop:~/cern/arda.dashboard.service-monitor-sls$

We keep making release candidates until we're happy, in this case we execute:

faraox@faraox-laptop:~/cern/arda.dashboard.service-monitor-sls$ python setup.py release
.....

and we'll have our 0.1.0 version.

Now we could add the new tag to the dash-build machine and our new rpm.

Generating the package

Generation python 2.3 packages

Access the dashb-build machine (You need root access to the dashb-build machine, ask dashboard team):

[lxplus249] /afs/cern.ch/user/c/cmestre > ssh root@dashb-build
[root@lxarda26 ~]# cd dashb-builds/cmestre/
[root@lxarda26 cmestre]# ls
dashboard-nightly.cron  dashboard-stable.cron  dashboard-unstable.cron
dashboard-nightly.xml   dashboard-stable.xml   dashboard-unstable.xml
[root@lxarda26 cmestre]# 

• Edit dashboard-unstable.xml and add the package name with the tag version. Then build the rpm executing the cron:

[root@lxarda26 cmestre]# cat dashboard-unstable.xml 
[...]
 <modules>
        <module name="arda.dashboard.service-monitor-sls" tag="vobox-agent-siteservices-0-1-0_rc2"/>
    </modules>
</dashboardBuild>
[root@lxarda26 cmestre]# sh dashboard-unstable.cron

Once do that, the package should be create in the corresponding directory in the dashb-build machine. In this case would be in http://dashb-build.cern.ch/apt/RPMS.unstable/ (unstable branch). The administrator of the directory would receive an e-mail is something goes wrong.

TODO: Explain add the tag to the arda.dashboard package.

Generating python 2.5 packages

NOTE The packages in the atladcops machines are python 2.5 versions (needed by the dq2 package). For this reason the packages should be build and get from the http://ddm-build.cern.ch/ddm/apt/RPMS.arda-py25/ .

For generate the new vobox-agent-siteservices ask Fernando Barreiro (fernando.harald.barreiro.megino@cernNOSPAMPLEASE.ch) or Carlos Mestre (carlos.mestre.gonzalez@cernNOSPAMPLEASE.ch), or ask DQ2 Team for root access.

The procedure should be:

• Logging into root@lxmrrk2701 in /root/ddm
• Edit the arda-py25.xml and add the dashboard module with the tag version, e.g:

<module name="arda.dashboard.service-monitor-sls" tag="vobox-agent-siteservices-0-1-0_rc5"/>

• Execute:

 ./build-py25.cron  arda

Get the packages from the arda-py25 repository: http://ddm-build.cern.ch/ddm/apt/RPMS.arda-py25/.

Enviromental variables for python 2.5 installations

If the machine is running with python 2.5 FIRST OF ALL export the correct PYTHONPATH and PATH variables:

# Setting the correct path for dashboard py2.5
export PYTHONPATH=/usr/lib64/python2.5/site-packages:/usr/lib/python2.5/site-packages/:$PYTHONPATH
export PATH=$PATH:/opt/dashboard/bin/

or just use the setup.sh script

source /opt/dashboard/doc/config/vobox-agent-setup/setup-vobox-agent.sh

Package vobox-agent-jinamar

The package with the source code is located under the Dashboard repository, under the directory arda.dashboard.service-monitor-sls/. Ask for write access to the Arda Dashboard Team.

For get the code (in my case cmestre):

$ export CVSROOT=:ext:cmestre@dashboard.cvs.cern.ch:/cvs/dashboard
$ export CVS_RSH=ssh

Installing and configuring

Create a rpm (if is not in the repository)

The standard (automatic) way should be edit the code, upload to the cvs repository, and make a new release of the code. Then you could generate the code using different build machines (python 2.3 or python 2.5).

For a manual rpm package creation get the package from the dashboard CVS (with anonymous or your user):

$ > export CVSROOT=:pserver:anonymous@dashboard.cvs.cern.ch:/cvs/dashboard
$ > cvs co arda.dashboard.service-monitor-sls

and follow the instructions How to Create a RPM.

Installing vobox-agent-jinamar and dependencies

List of dependencies needed :

dashboard-api
dashboard-common
dashboard-api-data
dashboard-util-url
dashboard-cli
dashboard-service-config
dashboard-api-service
pyXML
vobox-agent-jinamar

IMPORTANT: Most of the machines currently in use need python 2.5. The python 2.5 version of the dashboard are generated in DDM repository: http://ddm-build.cern.ch/ddm/apt/RPMS.arda-py25/. If you want to download the RPMs you can use this script (change the .py.txt to .py, run it doing python script.py [py25|py23]) dashb-pck-downloader.py.txt

pyXML could be found http://swrep.cern.ch/swrep/x86_64_slc4/PyXML-0.8.3-6.x86_64.rpm

Remember check with packages are installed before proceed with the installation:

rpm -qa | grep dashboard

and don't installed again. Command for install the pacakged needed. A more general guide: BriefIntroducctionDashboardCollector#Installation_dashboard

Configuring the package

If the dashboard (or the service-config dashboard package) has not been configured before, follow this instructions

Once installed and configured, modify /opt/dashboard/etc/dashboard-service-config/service-config.xml using as example the according xml file located in /opt/dashboard/etc/dashboard-service-config/example-file. (the proper xml config file)

NOTE: If the logs file are used by non-root user, /opt/dashboard/var/log/ should be owned by this user (or let write access to that user):

chown -R ddmusr01:root /opt/dashboard/var/log/

Using it.

Common commands: dashb-agent-list, dashb-agent-start and dashb-agent-stop. Export the variables The scripts are in /usr/lib/pythonY.X/site-packages/dashboard/collector/ (pyYX being the py version you are using) Remember check log file for know if everything went OK.

Fixing a bug and commit the changes into the cvs

The easiest way to fix a bug in the code is fixing that code directly in the installed directory, create a patch file, and then apply this to the cvs repository where you have write access.

Steps:

• Fix the bug under the installed repository /usr/lib/python/site-packages/dashboard/collector/vobox/
• Get the cvs code as anonymous in the tmp/ directory for example:

$ root@atlddm11]> export CVSROOT=:pserver:anonymous@dashboard.cvs.cern.ch:/cvs/dashboard
$ root@atlddm11]> cvs co arda.dashboard.service-monitor-sls

• Copy the changes from the python library to the cvs repository:

$ root@atlddm11] /bin/cp -R /usr/lib/python2.3/site-packages/dashboard/collector/vobox/ /tmp/arda.dashboard.service-monitor-sls/lib/dashboard/collector/

• Use the cvs diff to generate a diff that would be apply to your repository:

$ [root@atlddm11] arda.dashboard.service-monitor-sls> cvs diff -u8p > patchfile

• Send the patch (using scp for example) to a machine where you have write access to the repository. Apply the patch:

$ user@develop-machine arda.dashboard.service-monitor-sls > patch -p0 < patchfile
patching file lib/dashboard/collector/vobox/VoboxXML.py
patching file lib/dashboard/collector/vobox/generatexml.py

• Commit the change to the repository:

$ user@develop-machine arda.dashboard.service-monitor-sls > cvs ci

... and then if needed make a release

Troubleshooting

Import error trying to execute dashboard commands

Example:

Traceback (most recent call last):
  File "/opt/dashboard/bin/dashb-agent-list", line 9, in ?
    from dashboard.cli.config.AgentList import AgentList
ImportError: No module named dashboard.cli.config.AgentList

Normally this is because your PYTHONPATH is not correctly settled. Check in which directory the dashboard libraries are intalled and add that directory to the PYTHONPATH. Important In most of the machines the dashboard package is intalled for python2.5, so you have to add this line to the PATH:

export PYTHONPATH=/usr/lib64/python2.5/site-packages:/usr/lib/python2.5/site-packages/:$PYTHONPATH

Permission Denied

IOError: [Errno 13] Permission denied: '/opt/dashboard/var/log/dashboard'

Check if there is a different user (with other ID) running dashboard. For example, normally the user running the srevice is ddmusr01.

ServiceGroup:156 - ERROR - XXXXX stopped. Restarting

The problem working with dashboard is the common errors (exceptions) doesn't appear in the logs. Normally if you are not catching some exception you'll get an "ERROR <service-name> stoppped. Restarting". For detect the error you should run the script alone, like python /usr/lib/python2.5/site-packages/dashboard/collector/vobox/VoboxAgent.py. Fix every error. After that use the function logger for localize the error.

• dashb-pck-downloader.py.txt: dashb-pck-downloader.py.txt

• dashb-pck-downloader.py.txt: Script download dashboard packages