FTS community tools

This page is a duplicate of - DMFtsAddOns.

This page should index tools either internally produced by the FTS team or externally produced by T1 site admins. The eventual goal for tools should be that they integrate into the FTM monitoring node profile ( the FTM is basically an apache web-server ). We'll publish details of this in due course.

Please feel free to add your work here!

Are these tools Supported by FTS support: not until we say they are

FtsLogParse

Name: FtsLogParse

Provided by: Leif Nixon of NDGF

Brief description: This tool has been used to monitor NDGF's FTS since early summer 2007. As the name suggests, it parses the FTS log files and generates html pages summarising errors by type and by source site over different periods of time.

Location URL:

• ftslogparse.py, ftsstat.py, index.tmpl (the code) (stupid twiki adds .txt to the attached python filenames so you should rename them...)
• http://fts001.nsc.liu.se/ (the NDGF FTS monitor)
• http://fts001.nsc.liu.se/channelload/ (channel load monitor (code not available yet))

Longer description:

ftslogparse.py generates the static html pages showing the error summaries. It should be run regularly as a cron job. ftsstat.py does all the work of collecting statistics and drawing the graphs. It is run from cron once an hour. It counts the number of files younger than 1 hour in /var/tmp/glite-url-copy-sc3/$CHANNEL{failed,completed}/ and also scans the logfiles in $CHANNELcompleted for "filesize at destination" lines, which are summed to compute the throughput. (This information should really be fetched from the database instead.) The data is put into RRD database files (you need to have rrdtool installed) and the graphs are updated.

At the start of the script there is a list of channels to collect statistics for, and which colour they should be given in the graphs.

There are also some hardcoded paths you might need to adapt to your site:

• /var/tmp/glite-url-copy-sc3/ : where to find agent log files
• /root/var/ftsstat/rrds/ : where to put RRD data files
• /var/www/html/ : where to put graphs

You can change the line

for p in ("-1day", "-1week", "-3months", "-1year"):

if you want graphs for different time ranges.

The file index.tmpl is the template for the HTML pages (which are static once they have been generated). You can generate the pages for the different time ranges with something like:

  for p in -1day -1week -3months -1year
  do
    sed "s/PERIOD/$p/" < index.tmpl > index$p.html
  done
  ln -s index-1day.html index.html

For all this to work you need to make sure you always keep the agent log files for at least one hour.

FTS Query / Quality Map

Name: FTS Query / Quality Map

Provided by: Paul Rossman of FNAL

Brief description: This script queries the Oracle FTS DB for number of transfered jobs per channel per hour for the last 24 hours. Produces similar graphic as those used by CMS PhEDEx.

Location URL:

• fts_query_map.py.txt
• fts_query_map.sh.txt (can be used to run from cron)
• fts_query_map.png

Requires:

1. GraphTool - http://t2.unl.edu/documentation/graphtool/installation
2. cx_Oracle - http://www.python.net/crew/atuining/cx_Oracle/
3. Oracle Instant Client (with SDK) - http://www.oracle.com/technology/software/tech/oci/instantclient/index.html

FTS load plugin for Munin

Name: FTS load plugins for the Munin monitoring tool

Provided by: Paul Rossman of FNAL

Brief description: These Munin plugins use glite-transfer-list to generate active and pending transfer statistics on all and/or individual FTS channels.

Location URL:

• fts
• fts_
• http://cmsdcam2.fnal.gov/ftsload/

Installation:

1. Install munin-1.2.5-1.el3.rf and munin-node-1.2.5-1.el3.rf from the Dag repository.
2. Create a directory /etc/munin/plugins.local/ and copy the plugins fts and fts_ there. Make the plugin scripts executable (755).
3. In /etc/munin/plugins/ create links to the fts plugins. The plugin fts_ is a wildcard script.

      fts -> ../plugins.local/fts
      fts_ASGC-FNAL -> ../plugins.local/fts_

4. Create a file /etc/munin/plugin-conf.d/fts with the contents:

      [fts*]
      user root

5. Restart munin-node: /etc/init.d/munin-node restart

Wonderview

Name: Wonderview

Provided by: no one.

Brief description: this tool plumbs the FTS' view of the world directly into your brain. You will feel it as a 6th sense. Kinda like Minority Report, only better.

Location URL: sadly it doesn't exist.

FTS Monitor

name: FTS Monitor

Provided by: IN2P3-CC

Brief description: FTS monitor is a set of PHP pages that provide graphical monitoring view of the FTS service activity. PHP code interacts directly with Oracle DB backend of a FTS2 service.

Location URL: Browse here