• • • Introduction
• Introduce deployment scenarios, increasing complexity.
  • One NAGIOS Host - remote and native probes.
  • MyProxy and Proxy Retrieval.
  • One NAGIOS Host + One NRPE Host - remote, native and local probes.
• Meta Packages, YAIM variables and targets.
• Live install and configuration of YAIM, NCG and NAGIOS.
• Troubleshooting probe failures.
• Disabling probes.
• Tuning probes.
• Enabling notifications.
• Message bridge.

• • • SA1 package repository is hosted in Manchester.
• To enabled add the egee-SA1.repo to /etc/yum.repos.d/.

[egee-SA1]
name=EGEE SA1 software
baseurl=http://www.sysadmin.hep.ac.uk/rpms/egee-SA1/sl4/$basearch/
enabled=1

[egee-SA1 SRPMS]
name=EGEE SA1 SRPMS
baseurl=http://www.sysadmin.hep.ac.uk/rpms/egee-SA1/sl4/SRPMS/
enabled=1

• Source RPMs are available for all packages.
  • yum's createrepo is enabled for the SRPMS packages.
  • Repoview pretty HTML pages are enabled in the repodata directory.
• Change logs for packages are being added to the RPM %changelogs.

Remote Probes

Collect results from tests executed elsewhere , e.g from SAM (gather_sam) or from ENOC (gather_npm).

Native Probes

Use of standard nagios probes. e.g. check_ldap , check_tcp, ...

Local Probes

Makes use of WLCG format written probes, e.g the LFC-probe is executed with the check_wlcg nagios probe.

Native vs Local Probes

Generally "local" probes require a proxy whilst "native" probes are network level checks.
This is NOT a true statement though.

• Suitable for a site starting from scratch, very simple.
• yum install httpd An annoying bug, httpd should be an RPM PreReq of nagios rpm.
• yum install egee-NAGIOS lcg-CA
• /opt/glite/yaim/bin/yaim -c /root/site-info.def -n glite-NAGIOS
• NCG executed by YAIM queries SiteBDII and remote testers (e.g SAM, ENOC).
  • NCG populates NAGIOS configuration with hosts and probes to run against those hosts.

• /opt/glite/yaim/bin/yaim -c -s /root/site-info.def -n glite-NAGIOS
• NCG collects its hosts and services from the BDII endpoint
  ldap://site-bdii.example.org:2170/mds-vo-name=BIGMAN-LCG2,o=grid.
• The resulting NAGIOS Web interface is located at https://nagioshost.example.org/nagios
  • Visible as read only to set a VO members, maintained by voms2htpasswd.
  • Write operations, e.g rerun test, are restricted to the list of DNs you specify above.
  • Your list a local admins.
• The configuration of the main apache files is switched of by default. e.g ssl.conf.
  • We will extend the same concept to /etc/nagios/nagios.conf as well shortly.

• • • The site-info.def used is available as site-info-remote-native-nagios.def in nagios-tutorial.tgz.

• • • Native Probe: The org.Nagios.BDII-Check uses NAGIOS's standard check_ldap probe.
• Remote Probe: The gather_npm and gather_sam probes query ENOC or SAM and submit the passive test results for all the SAM tests.

• Ask access to sites SAM tests if you haven't done it with original request
• An extra YAIM variable, NCG_LDAP_FILTER, should be added to your site-info.def file on your NAGIOS host.
  • This is an LDAP filter that can be used to extract the GlueSite objects of interest to you from the information system.
• Examples:

• This would be entered in the site-info.def file as. NCG_LDAP_FILTER="GlueSiteOtherInfo=EGEE_ROC=ITALY"
• For ideas on how to match a GlueSite object see: How to publish my GlueSite.

• • • Most of the "local" probes require a valid user proxy to run. e.g globus-url-copy a file to a CE.
• Model chosen is to allow the NAGIOS host to be a trusted_retriever of your proxy.
  1. User uploads to MyProxy service with myproxy-init specifying exactly which DNs can retrieve the proxy.
  2. The NAGIOS service runs nagios-proxy-refresh to keep a valid proxy from MyProxy.
  3. The NAGIOS host certificate is used to identify to the MyProxy service.
• The NAGIOS service never has access to the host certificate which is owned by root.
• Assume PATCH:2184 has been released.
  • Configurations of MyProxy beyond what is needed by the WMS become possible.
  • Easy to work around if PATCH:2184 is not released.
• Choice of MyProxy is yours.
  • Use one you trust in your region.
  • Set one up your self. It's easy and can coexist with NAGIOS if you like.

• • • yum install glite-PX lcg-CA
• YAIM configuration

         SITE_NAME=YourSite
         PX_HOST=myproxy.example.ch
         GRID_TRUSTED_RETRIEVERS="'/DC=ch/DC=cern/OU=computers/CN=nrpe-ui.example.ch'"       
         GRID_AUTHORIZED_RETRIEVERS="'/DC=ch/DC=cern/OU=computers/CN=nrpe-ui.example.ch'"
         SITE_EMAIL="mybigsite@example.ch"
      

• /opt/glite/yaim/bin/yaim -c -s /root/site-info.def -n glite-PX
• This configuration allows one to specify at myproxy-init time that the user MAY specify that nrpe-ui.example.ch can retrieve the proxy.
  • It is NOT the default that all uploaded proxies can be retrieved by nrpe-ui.example.ch
• On a UI run the following to upload the proxy.
  • myproxy-init -c 336 -k NagiosRetrieve-nrpe-ui.example.org- -s myproxy.example.org -l nagios -x -Z "/DC=ch/OU=computers/CN=nrpe-ui.example.ch"

• A glite-UI has been added that is called via NRPE from the NAGIOS service to run "remote" probes.
• NRPE requires configuration which it gathers from the NAGIOS host. .. See later.

• • • A standard glite-UI is needed.
• To it we add the egee-NRPE meta package and configure both with YAIM.
• yum install glite-UI lcg-CA egee-NRPE
• Additional YAIM variables.

• Plus standard UserInterface variables.

• /opt/glite/yaim/bin/yaim -c -s /root/site-info.def -n glite-UI -n glite-NRPE

• YAIM sets up NRPE /etc/nagios/nrpe.cfg
• Sets up proxy retrieval /etc/nagios-proxy-refresh.conf
• Sets up NRPE configuration retrieval /etc/mirror-nrpe-conf.conf

• • • The command nagios-proxy-refresh attempts to retrieve a proxy for running tests with.
• They are retrieved from the MyProxy service of course.
• A cron runs every 4 hours calling nagios-proxy-refresh.
  • Log file is located at =/var/log/nagios-proxy-refresh.conf which is rotated.
  • /sbin/service nagios-proxy-refresh stop|start enables and disables the cron.
  • A start also runs a proxy-refesh there and then.
• There are also nagios checks to:
  1. Check the validity of the retrieved proxy.
  2. Check the time left to expiry on the credential stored in MyProxy.

• • • Use same YAIM configuration as NRPE Node.
  • Different target of course egee-NAGIOS
• /opt/glite/yaim/bin/yaim -c -s /root/site-info.def -n glite-NAGIOS
• NCG creates NAGIOS configuration as before.
• NCG configures NAGIOS to call check_nrpe to run tests via NRPE node.
• NCG also now creates actual NRPE configuration for the NRPE node.
  • Generated in /etc/nagios/nrpe on NAGIOS host!
  • NRPE configuration is needed however on the NRPE node.

• • • The NAGIOS box webserver is now serving. /etc/nagios/nrpe
  • YAIM created /etc/httpd/conf.d/nrpe.conf
  • Located at https://nagios.example.org/nrpe/
  • Access restricted x509 style to CN=nrpe-ui.example.ch
• mirror-nrpe-config command runs on NRPE node.
  • Mirrors in NRPE configuation from NAGIOS host
  • Again cron enabled/disabled with a
    /sbin/service mirror-nrpe-config start/stop
  • A start issues a mirror.

• • • Some tests have "more info" configured in NAGIOS, e.g:
  1. check_ldap, links the NAGIOS documentation for check_ldap
  2. ENOC Links through the the ENOC test results on their web page.
• nagios-run-check
  • Requires a host and nagios service, look it up on web interface.
  • It provides the command line to run the test.

          #  nagios-run-check -v -d -s org.nagios.LocalLogger-PortCheck -H ce1.triumf.ca
          Executing command:
           su nagios -c '"/usr/lib64/nagios/plugins"/"check_tcp" -H "206.12.1.15" -t "60" "-p 9002"'
          # nagios-run-check -s org.nagios.LocalLogger-PortCheck -H ce1.triumf.ca
          CRITICAL - Socket timeout after 60 seconds
          

• • • For various reasons we may need to disable or tune various tests. e.g
  • We test ResourceBDIIs but these need not be visible from a ROC.
  • We test LocalLoggers on CEs but again this need only be visible from the WNs at a site.
• The NCG configuration can be altered with a /etc/ncg/ncg.localdb
  • See ncg.localdb.exampe for some examples.

REMOVE_SERVICE!ce1.triumf.ca!org.glite.LocalLogger
REMOVE_SERVICE!ce2.triumf.ca!org.glite.LocalLogger
REMOVE_SERVICE!srm.triumf.ca!BDII

• This will remove the service checks in NAGIOS service Group org.glite.LocalLogger

• • • NAGIOS probe results are being published via the messaging system.
• You subscribe to them with a stomp client
  • e.g perl(Net::Stomp)

• • • Discovered in last few days, will be fixed shortly.
  • Only voms.cern.ch VOs supported by voms2htpasswd.
  • Two NAGIOS boxes cannot use the same MyProxy
  • Add new YAIM options to:
    1. Disable editing of main nagios configuration files.
       • Easier integration into existing NAGIOS based sites.
    • Disable generation of ncg.conf file.
      • Direct NCG configuration will always be richer than YAIM.
      • Sites in all but simplest case may well end up tuning this.

• • • Homepage: OAT_EGEE_III
• This tutorial: GridMonitoringNcgYaimTutorial2008
• NCG Manual Install guide: GridMonitoringNcg (needs some updating)
• YAIM Install Guide: GridMonitoringNcgYaim (needs some expansion)
• Savannah for Bugs, Requests: https://savannah.cern.ch/projects/sa1tools/
• Discussion Mailing List: egee3-operations-automationdiscuss
• Nagios and NCG Troubleshooting