WLCG Middleware Package Reporter

Author/Developer: Lionel Cons CERN/IT-SDC

Introduction

As discussed during our 4th meeting, volunteer sites must publish the packages they have installed so that they can be compared against reference versions.

We have worked with the Pakiti team and the Pakiti v3 client can now be used to submit package information to the MW Readiness collector server. Since 2015-01-06, the Pakiti v3 client is available from EPEL as the pakiti-client rpm.

Description

In order to collect the list of installed packages at volunteer sites, a lightweight client has to be used. It sends its data (in encrypted form) to a collector server located at CERN. The information collected is protected and only the WLCG MW Officer (and the team working with him) can access it.

Technical details:

• the data is encrypted using openssl smime
• the encrypted data is sent over HTTP
• the received data is stored in a MySQL database with restricted access

Site Name

Collected packages are grouped by site so it is very important to use the correct site name. As discussed elsewhere, the volunteer site name may be different from the official name.

In the instructions below, MY-SITE must be replaced by the agreed site name.

Installation

In order to report the list of installed packages, a single script has to be installed. The easiest solution is to install it as an rpm. It is available from EPEL so, if your machine already uses EPEL, the following command is enough to get it installed:

  # yum install pakiti-client

If you don't install it via EPEL, please make sure that you install the latest version. You must use at least version 3.0.1 that added support to the tag option.

In addition to the script, a configuration file corresponding to the MW Readiness collector must be present. It is attached to this wiki page and can be downloaded on your machine with something like:

  # cd /etc
  # wget https://twiki.cern.ch/twiki/pub/LCG/MiddlewarePackageReporter/wlcg-mwr-pkgdb.conf

Please make sure you keep the configuration file up to date. The current version is from 5-Aug-2015 and has been changed to define the tag to use for middleware readiness (that is MWR).

The recommended way to use this script is daily, via cron. You can add to your crontab something like:

  # pakiti-client --conf /etc/wlcg-mwr-pkgdb.conf --site MY-SITE --rndsleep 60 --tag MWR

Notes:

• the pakiti-client script does not require root privileges
• EGI sites are highly recommended to also configure their Pakiti client to report to the EGI CSIRT

Puppet Installation

For the sites using Puppet, here is what could be used to install and configure the Pakiti v3 client to send information to the MW Readiness collector.

  # package installation
  package { 'pakiti-client':
    ensure => 'installed',
  }
  # script configuration
  $conf = '/etc/wlcg-mwr-pkgdb.conf'
  file { $conf:
    mode   => '0644',
    source => 'puppet:///.../wlcg-mwr-pkgdb.conf',
  }
  # cron definition
  cron { 'wlcg-package-reporter':
    ensure  => 'present',
    command => "pakiti-client --conf ${conf} --site MY-SITE --rndsleep 60 --tag MWR",
    user    => 'nobody',
    hour    => fqdn_rand(24,'pakiti'),
    minute  => fqdn_rand(60,'pakiti'),
  }

This assumes that:

• the Puppet managed machines use EPEL
• the wlcg-mwr-pkgdb.conf file has been copied somewhere in Puppet

CVMFS

The pakiti client is now available also via CVMFS grid.cern.ch .

In order to send data to the MW readiness collector site managers can mount the cvmfs grid.cern.ch and use this command in their cron:

/cvmfs/grid.cern.ch/pakiti/bin/pakiti-client --site <site_name> --conf /cvmfs/grid.cern.ch/pakiti/conf/WLCG-MWR.conf

API

Lionel documented the format used by the pakiti-client script in the script’s man page (in GitHub). Lionel's recommendations:

• If someone wants to write their own submitter tool, I would strongly recommend to use the pakiti-client script to send the report. This can be done with the --input option. The idea would be to:
  • prepare one or more reports to be sent and save them as files
  • for each file, run pakiti-client with the --input option to send the prepared report(s)
• If someone wants to filter out the list of packages report, I would strongly recommend to use the pakiti-client script anyway. The idea would be to:
  • run pakiti-client with the --output option to save the report to file instead of sending it
  • edit the saved file to remove the packages that should not be exposed (think “grep -v”)
  • run pakiti-client with the --input option to send the edited report

Report Format

The pakiti-client 's man page contains a description of the report format used.

References

• for any question: wlcg-package-reporter@cernNOSPAMPLEASE.ch
• pakiti-client 's man page, EPEL page