VOSpecificServicesMon
Introduction (and before start)
This page contains information remaining the project
NAME, that consist in create a dashboard package that contains several scripts for monitoring different services.
BEFORE START take a look to a brief introduction of how to use and develop your own collectors (services) using dashboard could be found in
BriefIntroducctionDashboardCollector.
In the rest of the page we'll present the current implementation of the
NAME agent and how to manage this service.
List of services implemented
This is a list with the scripts that are running, with services they're monitoring and the general configuration
show
hide
VOBOX
Check the DQ2 results:
Files_State_NEW_Number
Files_State_QUEUED_Number
Files_State_STAGING_Number
Files_State_STAGE_FAILED_Number
Files_State_STAGED_Number
Files_State_COPYING_Number
Files_State_COPY_FAILED_Number
Files_State_COPIED_Number
Files_State_REGISTER_FAILED_Number
Files_State_DONE_LastHour
ARDACallbacks_Number
Files_Number
and the lemon metrics 4031,4032, 4033, 4034
Central Catalog monitor
Check the webalizer parameters (repository, content, location, subscript and tracer) and lemon metrics 4035 and 4036
Panda_Mon Monitor
VOALICE vobox monitor
for developers: Add a new monitoring service to the package
Add a new script for the package is quite easy, you need to add the .py files to the lib directory and modify the
VoboxAgent.py script for call your script from there and get the parameters.
- Checkout the code
- Add your python script to PACKAGE/lib/dashboard/collector/vobox/ directory
- Edit the PACKAGE/lib/dashboard/collector/vobox/VoboxAgent.py to create a class that inherit from Service
- Create an service-config.xml example
- Create the service directory logs.
- Remember to ADD all the new files to the CVS
Add your python script
Just copy your script under the lib directory in the correct location (normally PACKAGE/lib/dashboard/collector/vobox/)
Calling the script from the service
You need to modify
VoboxAgent.py )_PACKAGE/lib/dashboard/collector/vobox/_ and add a new class that inherit from the Service class. If remember from the
BriefIntroducctionDashboardCollector, this file read the parameters from the service-config.xml and execute the python script.
Follow the example of other class from that file. Follow
this instruccion is you need more help.
Creating config files
The convention is that for every
new script should be defined a new service group name. This means have a service group section for every monitoring script and inside this group add all the machines needed to be monitoring, example:
<service-group name="VOBoxSiteServices.monitor">
<services>
<service name="atlddm11.vobox.monitor"....
</services>
</service-group>
<service-group name="cc_monitor.service">
<services>
<service name="atlddm24.cc_monitor.service"....
</services>
</service-group>
Save that file into
PACKAGE/config/service-config/ directory.
One is defined a new service-group name is
necessary to create a new configuration directory that would be under /opt/dashboard/etc/dashboard-service-config/<service-group-name>/ where the logging information is saved. Place that directory into
PACKAGE/config/service-config/. Inside the service group directory should be placed an etc/ directory, and inside the logging files: logging.cfg and dashboard.cfg. Take a look at the existing directory as an example.
Edit the dashboard.cfg to specify the logging.cfg path (usually
/opt/dashboard/etc/dashboard-service-config/<service-group-name>/etc/logging.cfg
Edit the loggin.cfg to specify where the logs should be safe (section
handler_root_file and
handler_dashboard_file).
Don't forget to modify the setup.py for add this directory to the package.
Create an service-config.xml example
Place your xml example under config/service/config with a name associated to the service package (like <service-group-name>.xml). Once the package is installed the example would be copied to /opt/dashboard/etc/dashboard-service-config/
Make a release
The release system of dashboard follows the normal convention for software release. There are major, minor, path and candidate versions:
faraox@faraox-laptop:~/cern/arda.dashboard.service-monitor-sls$ python setup.py release --help
....
Options for 'release' command:
--major (-M) major version release
--minor (-m) minor version release
--patch (-p) patch release
--candidate (-c) candidate release
--release-notes (-r) location of the RELEASE-NOTES file
You could start a major or minor version, and then keep making candidate versions. Once do that and you're happy with your code, make a release (without the a release option).
In this example we want to make a new candidate release for the version 0.1.0:
faraox@faraox-laptop:~/cern/arda.dashboard.service-monitor-sls$ python setup.py release -c
running release
Current module version is 0.1.0_rc2. Released version will be 0.1.0_rc3.
Continue with release? [Y/N]: Y
Tag will be vobox-agent-siteservices-0-1-0_rc3. Continue [Y/N]: Y
Released version 0.1.0_rc3
faraox@faraox-laptop:~/cern/arda.dashboard.service-monitor-sls$
We keep making release candidates until we're happy, in this case we execute:
faraox@faraox-laptop:~/cern/arda.dashboard.service-monitor-sls$ python setup.py release
.....
and we'll have our 0.1.0 version.
Now we could add the new tag to the dash-build machine and
our new rpm.
Generating the package
Generation python 2.3 packages
Access the dashb-build machine (You need
root access to the dashb-build machine, ask dashboard team):
[lxplus249] /afs/cern.ch/user/c/cmestre > ssh root@dashb-build
[root@lxarda26 ~]# cd dashb-builds/cmestre/
[root@lxarda26 cmestre]# ls
dashboard-nightly.cron dashboard-stable.cron dashboard-unstable.cron
dashboard-nightly.xml dashboard-stable.xml dashboard-unstable.xml
[root@lxarda26 cmestre]#
- Edit dashboard-unstable.xml and add the package name with the tag version. Then build the rpm executing the cron:
[root@lxarda26 cmestre]# cat dashboard-unstable.xml
[...]
<modules>
<module name="arda.dashboard.service-monitor-sls" tag="vobox-agent-siteservices-0-1-0_rc2"/>
</modules>
</dashboardBuild>
[root@lxarda26 cmestre]# sh dashboard-unstable.cron
Once do that, the package should be create in the corresponding directory in the dashb-build machine. In this case would be in
http://dashb-build.cern.ch/apt/RPMS.unstable/
(unstable branch). The administrator of the directory would receive an e-mail is something goes wrong.
TODO: Explain add the tag to the arda.dashboard package.
Generating python 2.5 packages
NOTE The packages in the atladcops machines are python 2.5 versions (needed by the dq2 package). For this reason the packages should be build and get from the
http://ddm-build.cern.ch/ddm/apt/RPMS.arda-py25/
.
For generate the new vobox-agent-siteservices ask Fernando Barreiro (
fernando.harald.barreiro.megino@cernNOSPAMPLEASE.ch) or Carlos Mestre (
carlos.mestre.gonzalez@cernNOSPAMPLEASE.ch), or ask DQ2 Team for root access.
The procedure should be:
- Logging into root@lxmrrk2701 in /root/ddm
- Edit the arda-py25.xml and add the dashboard module with the tag version, e.g:
<module name="arda.dashboard.service-monitor-sls" tag="vobox-agent-siteservices-0-1-0_rc5"/>
./build-py25.cron arda
Get the packages from the arda-py25 repository:
http://ddm-build.cern.ch/ddm/apt/RPMS.arda-py25/
.
Enviromental variables for python 2.5 installations
If the machine is running with python 2.5
FIRST OF ALL export the correct PYTHONPATH and PATH variables:
# Setting the correct path for dashboard py2.5
export PYTHONPATH=/usr/lib64/python2.5/site-packages:/usr/lib/python2.5/site-packages/:$PYTHONPATH
export PATH=$PATH:/opt/dashboard/bin/
or just use the setup.sh script
source /opt/dashboard/doc/config/vobox-agent-setup/setup-vobox-agent.sh
Package vobox-agent-jinamar
The package with the source code is located under the
Dashboard repository
, under the directory
arda.dashboard.service-monitor-sls/. Ask for write access to the Arda Dashboard Team.
For get the code (in my case cmestre):
$ export CVSROOT=:ext:cmestre@dashboard.cvs.cern.ch:/cvs/dashboard
$ export CVS_RSH=ssh
Installing and configuring
Create a rpm (if is not in the repository)
The
standard (automatic) way should be edit the code, upload to the cvs repository, and
make a new release of the code. Then you could generate the code using different build machines (python 2.3 or python 2.5).
For a
manual rpm package creation get the package from the dashboard CVS (with anonymous or your user):
$ > export CVSROOT=:pserver:anonymous@dashboard.cvs.cern.ch:/cvs/dashboard
$ > cvs co arda.dashboard.service-monitor-sls
and follow the instructions
How to Create a RPM.
Installing vobox-agent-jinamar and dependencies
List of dependencies needed :
dashboard-api
dashboard-common
dashboard-api-data
dashboard-util-url
dashboard-cli
dashboard-service-config
dashboard-api-service
pyXML
vobox-agent-jinamar
IMPORTANT: Most of the machines currently in use need
python 2.5. The python 2.5 version of the dashboard are generated in DDM repository:
http://ddm-build.cern.ch/ddm/apt/RPMS.arda-py25/
.
If you want to download the RPMs you can use this script (change the .py.txt to .py, run it doing python script.py [py25|py23])
dashb-pck-downloader.py.txt
pyXML could be found
http://swrep.cern.ch/swrep/x86_64_slc4/PyXML-0.8.3-6.x86_64.rpm
Remember check with packages are installed before proceed with the installation:
rpm -qa | grep dashboard
and don't installed again. Command for install the pacakged needed.
A more general guide:
BriefIntroducctionDashboardCollector#Installation_dashboard
Configuring the package
If the dashboard (or the service-config dashboard package) has not been configured before, follow
this instructions
Once installed and configured, modify
/opt/dashboard/etc/dashboard-service-config/service-config.xml using as example the according xml file located in
/opt/dashboard/etc/dashboard-service-config/example-file. (the proper xml config file)
NOTE: If the logs file are used by non-root user,
/opt/dashboard/var/log/ should be owned by this user (or let write access to that user):
chown -R ddmusr01:root /opt/dashboard/var/log/
Using it.
Common commands:
dashb-agent-list,
dashb-agent-start and
dashb-agent-stop.
Export the variables
The scripts are in /usr/lib/pythonY.X/site-packages/dashboard/collector/ (pyYX being the py version you are using)
Remember check log file for know if everything went OK.
Fixing a bug and commit the changes into the cvs
The easiest way to fix a bug in the code is fixing that code directly in the installed directory, create a patch file, and then apply this to the cvs repository where you have write access.
Steps:
- Fix the bug under the installed repository /usr/lib/python/site-packages/dashboard/collector/vobox/
- Get the cvs code as anonymous in the tmp/ directory for example:
$ root@atlddm11]> export CVSROOT=:pserver:anonymous@dashboard.cvs.cern.ch:/cvs/dashboard
$ root@atlddm11]> cvs co arda.dashboard.service-monitor-sls
- Copy the changes from the python library to the cvs repository:
$ root@atlddm11] /bin/cp -R /usr/lib/python2.3/site-packages/dashboard/collector/vobox/ /tmp/arda.dashboard.service-monitor-sls/lib/dashboard/collector/
- Use the cvs diff to generate a diff that would be apply to your repository:
$ [root@atlddm11] arda.dashboard.service-monitor-sls> cvs diff -u8p > patchfile
- Send the patch (using scp for example) to a machine where you have write access to the repository. Apply the patch:
$ user@develop-machine arda.dashboard.service-monitor-sls > patch -p0 < patchfile
patching file lib/dashboard/collector/vobox/VoboxXML.py
patching file lib/dashboard/collector/vobox/generatexml.py
- Commit the change to the repository:
$ user@develop-machine arda.dashboard.service-monitor-sls > cvs ci
... and then if needed
make a release
Troubleshooting
Import error trying to execute dashboard commands
Example:
Traceback (most recent call last):
File "/opt/dashboard/bin/dashb-agent-list", line 9, in ?
from dashboard.cli.config.AgentList import AgentList
ImportError: No module named dashboard.cli.config.AgentList
Normally this is because your PYTHONPATH is not correctly settled. Check in which directory the dashboard libraries are intalled and add that directory to the PYTHONPATH.
Important In most of the machines the dashboard package is intalled for python2.5, so you have to add this line to the PATH:
export PYTHONPATH=/usr/lib64/python2.5/site-packages:/usr/lib/python2.5/site-packages/:$PYTHONPATH
Permission Denied
IOError: [Errno 13] Permission denied: '/opt/dashboard/var/log/dashboard'
Check if there is a different user (with other ID) running dashboard. For example, normally the user running the srevice is ddmusr01.
ServiceGroup:156 - ERROR - XXXXX stopped. Restarting
The problem working with dashboard is the common errors (exceptions) doesn't appear in the logs. Normally if you are
not catching some exception you'll get an "ERROR <service-name> stoppped. Restarting".
For detect the error you should run the script alone, like python /usr/lib/python2.5/site-packages/dashboard/collector/vobox/VoboxAgent.py. Fix every error. After that use the function logger for localize the error.