Difference: MRTGSquidMonitoring (1 vs. 3)

Revision 32016-02-02 - AlastairDewhurst

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
Line: 20 to 20
 */5 * * * * [ "`cl_status rscstatus`" == "all" ] || exit; /home/squidmon/scripts/all/mrtg_plots.py >> /home/squidmon/logs/all/mrtg_plots.log 2>&1
Changed:
<
<

Retrieving information from GOCDB

>
>

Retrieving information from GOCDB / OIM

  A cron job run every 3 hours:
40 */3 * * * [ "`cl_status rscstatus`" == "all" ] || exit; /home/squidmon/scripts/grid-services/get_squids.py >> /home/squidmon/logs/get_grid_squids.log 2>&1
Changed:
<
<
This creates a file called /home/squidmon/www/grid-squids.json which contains the information stored in the GOCDB and OIM about the squids:
>
>
This creates a file called /home/squidmon/www/grid-squids.json which contains a list of the squid services, the site name and the monitoring port:
 
% head /home/squidmon/www/grid-squids.json
{
Line: 41 to 41
  "name": "UKI-LT2-RHUL", "port": 3401
Added:
>
>
The information is taken from the GOCDB and OIM. The script then checks and exception list and modifies the list if necessary before saving the file. The field source will either be egi if the squid came from the GOCDB, osg if the squid came from OIM and exception if it came from the exception list.

Adding exceptions

Exceptions can be added to /home/squidmon/conf/all
% cat exceptions.txt 
#######################################################################
#
# This file contains a list of exceptions for the squid monitoring
#
# ADD <site> <alias> <port>
# Will add (or overwrite if alias already exists) squid
#
#
# REMOVE <alias>
# Will remove the listed alias from being monitored
# Will print a warning if alias doesn't exist
#
#######################################################################
ADD CERN-PROD ca-proxy.cern.ch 3401
REMOVE lcg0679.gridpp.rl.ac.uk
 

Creating config files

Revision 22015-12-09 - AlastairDewhurst

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
Changed:
<
<
%ToC{}%
>
>
 

Introduction

Changed:
<
<
>
>
This page describes how the MRTG monitoring for the WLCG Squids is configured and how to make changes.
 

Setup

Line: 31 to 31
 
% head /home/squidmon/www/grid-squids.json
{
Changed:
<
<
"squid.hep.anl.gov": { "source": "osg", "group": "ANLASC", "name": "ANLASC-squid", "types": [ "frontier", "cvmfs" ]
>
>
"cvmfs-px4.cr.cnaf.infn.it": { "source": "egi", "name": "INFN-T1", "port": 3401
  },
Added:
>
>
"squid1.ppgrid1.rhul.ac.uk": { "source": "egi", "name": "UKI-LT2-RHUL", "port": 3401
 

Creating config files

Revision 12015-12-09 - AlastairDewhurst

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WebHome"
%ToC{}%

Introduction

Setup

Cron jobs:

As squidmon user:

% crontab -l
# Updating of GOCDB and OIM squid records
40 */3 * * * [ "`cl_status rscstatus`" == "all" ] || exit; /home/squidmon/scripts/grid-services/get_squids.py >> /home/squidmon/logs/get_grid_squids.log 2>&1
44 */3 * * * [ "`cl_status rscstatus`" == "all" ] || exit; /home/squidmon/scripts/all/makeConfig.py >> /home/squidmon/logs/all/makeConfig.log 2>&1

# Updating MRTG monitoring that uses GOCDB and OIM records
*/5 * * * * [ "`cl_status rscstatus`" == "all" ] || exit; /home/squidmon/scripts/all/mrtg_plots.py >> /home/squidmon/logs/all/mrtg_plots.log 2>&1

Retrieving information from GOCDB

A cron job run every 3 hours:

40 */3 * * * [ "`cl_status rscstatus`" == "all" ] || exit; /home/squidmon/scripts/grid-services/get_squids.py >> /home/squidmon/logs/get_grid_squids.log 2>&1

This creates a file called /home/squidmon/www/grid-squids.json which contains the information stored in the GOCDB and OIM about the squids:

% head /home/squidmon/www/grid-squids.json
{
  "squid.hep.anl.gov": {
    "source": "osg", 
    "group": "ANLASC", 
    "name": "ANLASC-squid", 
    "types": [
      "frontier", 
      "cvmfs"
    ]
  }, 

Creating config files

A cron job run every 3 hours and 4 minutes after the information has been updated from the GOCDB/OIM:

44 */3 * * * [ "`cl_status rscstatus`" == "all" ] || exit; /home/squidmon/scripts/all/makeConfig.py >> /home/squidmon/logs/all/makeConfig.log 2>&1

This creates the .cfg files and puts them in:

% ls /home/squidmon/etc/all/
ANLASC-squid.cfg       Australia-ATLAS_1.cfg  CA-VICTORIA-WESTGRID-T2.cfg  DukeT3_SQUID.cfg  GRIF.cfg        INDIACMS-TIFR.cfg  JINR-T1.cfg            NGI_IT.cfg      RAL-LCG2_1.cfg  RWTH-Aachen.cfg    SARA-MATRIX_1.cfg  SBU_T3_Squid.cfg
ANLASC-squid.ok        Australia-ATLAS_1.ok   CA-VICTORIA-WESTGRID-T2.ok   DukeT3_SQUID.ok   GRIF.ok         INDIACMS-TIFR.ok   JINR-T1.ok             NGI_IT.ok       RAL-LCG2_1.ok   RWTH-Aachen.ok     SARA-MATRIX_1.ok   SBU_T3_Squid.ok
Australia-ATLAS_0.cfg  Australia-ATLAS.cfg    DESY-ZN.cfg                  FMPhI-UNIBA.cfg   IN2P3-LAPP.cfg  JINR-LCG2.cfg      Nevis_cvmfs_squid.cfg  RAL-LCG2_0.cfg  RAL-LCG2.cfg    SARA-MATRIX_0.cfg  SARA-MATRIX.cfg
Australia-ATLAS_0.ok   Australia-ATLAS.ok     DESY-ZN.ok                   FMPhI-UNIBA.ok    IN2P3-LAPP.ok   JINR-LCG2.ok       Nevis_cvmfs_squid.ok   RAL-LCG2_0.ok   RAL-LCG2.ok     SARA-MATRIX_0.ok   SARA-MATRIX.ok

MRTG plots

A cron job run every 5 minutes:

# Updating MRTG monitoring that uses GOCDB and OIM records
*/5 * * * * [ "`cl_status rscstatus`" == "all" ] || exit; /home/squidmon/scripts/all/mrtg_plots.py >> /home/squidmon/logs/all/mrtg_plots.log 2>&1

The script is relatively straight forward, it

Webpage

The index.html file is created by hand.

-- AlastairDewhurst - 2015-12-09

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback