Show Children Hide Children

Main FTS Pages
FtsRelease22
Install
Configuration
Administration
Procedures
Operations
Development
Previous FTSes
FtsRelease21
FtsRelease21
All FTS Pages
FtsWikiPages
Last Page Update
LionelSchwarz
2009-10-23

FTS community tools

This page is a duplicate of - DMFtsAddOns.

This page should index tools either internally produced by the FTS team or externally produced by T1 site admins. The eventual goal for tools should be that they integrate into the FTM monitoring node profile ( the FTM is basically an apache web-server ). We'll publish details of this in due course.

Please feel free to add your work here!

Are these tools Supported by FTS support: not until we say they are smile

FtsLogParse

Name: FtsLogParse

Provided by: Leif Nixon of NDGF

Brief description: This tool has been used to monitor NDGF's FTS since early summer 2007. As the name suggests, it parses the FTS log files and generates html pages summarising errors by type and by source site over different periods of time.

Location URL:

Longer description:

ftslogparse.py generates the static html pages showing the error summaries. It should be run regularly as a cron job. ftsstat.py does all the work of collecting statistics and drawing the graphs. It is run from cron once an hour. It counts the number of files younger than 1 hour in /var/tmp/glite-url-copy-sc3/$CHANNEL{failed,completed}/ and also scans the logfiles in $CHANNELcompleted for "filesize at destination" lines, which are summed to compute the throughput. (This information should really be fetched from the database instead.) The data is put into RRD database files (you need to have rrdtool installed) and the graphs are updated.

At the start of the script there is a list of channels to collect statistics for, and which colour they should be given in the graphs.

There are also some hardcoded paths you might need to adapt to your site:

  • /var/tmp/glite-url-copy-sc3/ : where to find agent log files
  • /root/var/ftsstat/rrds/ : where to put RRD data files
  • /var/www/html/ : where to put graphs

You can change the line

for p in ("-1day", "-1week", "-3months", "-1year"):

if you want graphs for different time ranges.

The file index.tmpl is the template for the HTML pages (which are static once they have been generated). You can generate the pages for the different time ranges with something like:

  for p in -1day -1week -3months -1year
  do
    sed "s/PERIOD/$p/" < index.tmpl > index$p.html
  done
  ln -s index-1day.html index.html

For all this to work you need to make sure you always keep the agent log files for at least one hour.

FTS Query / Quality Map

Name: FTS Query / Quality Map

Provided by: Paul Rossman of FNAL

Brief description: This script queries the Oracle FTS DB for number of transfered jobs per channel per hour for the last 24 hours. Produces similar graphic as those used by CMS PhEDEx.

Location URL:

Requires:

  1. GraphTool - http://t2.unl.edu/documentation/graphtool/installation
  2. cx_Oracle - http://www.python.net/crew/atuining/cx_Oracle/
  3. Oracle Instant Client (with SDK) - http://www.oracle.com/technology/software/tech/oci/instantclient/index.html

FTS load plugin for Munin

Name: FTS load plugins for the Munin monitoring tool

Provided by: Paul Rossman of FNAL

Brief description: These Munin plugins use glite-transfer-list to generate active and pending transfer statistics on all and/or individual FTS channels.

Location URL:

Installation:

  1. Install munin-1.2.5-1.el3.rf and munin-node-1.2.5-1.el3.rf from the Dag repository.
  2. Create a directory /etc/munin/plugins.local/ and copy the plugins fts and fts_ there. Make the plugin scripts executable (755).
  3. In /etc/munin/plugins/ create links to the fts plugins. The plugin fts_ is a wildcard script.
       fts -> ../plugins.local/fts
       fts_ASGC-FNAL -> ../plugins.local/fts_
  4. Create a file /etc/munin/plugin-conf.d/fts with the contents:
       [fts*]
       user root
  5. Restart munin-node: /etc/init.d/munin-node restart

Wonderview

Name: Wonderview

Provided by: no one.

Brief description: this tool plumbs the FTS' view of the world directly into your brain. You will feel it as a 6th sense. Kinda like Minority Report, only better.

Location URL: sadly it doesn't exist.

FTS Monitor

name: FTS Monitor

Provided by: IN2P3-CC

Brief description: FTS monitor is a set of PHP pages that provide graphical monitoring view of the FTS service activity. PHP code interacts directly with Oracle DB backend of a FTS2 service.

Location URL: Browse here

Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt fts.txt r1 manage 2.9 K 2007-10-29 - 16:05 PaulRossman FNAL FTS load plugin for Munin
Texttxt fts_.txt r1 manage 3.4 K 2007-10-29 - 16:05 PaulRossman FNAL FTS load plugin for Munin
PNGpng fts_query_map.png r2 r1 manage 97.1 K 2008-02-21 - 16:38 PaulRossman  
Texttxt fts_query_map.py.txt r1 manage 2.7 K 2008-02-19 - 20:44 PaulRossman  
Texttxt fts_query_map.sh.txt r1 manage 0.3 K 2008-02-19 - 20:44 PaulRossman  
Texttxt ftslogparse.py.txt r1 manage 21.1 K 2007-10-18 - 12:56 DavidCameron NDGF FTS monitor log parse script
Texttxt ftsstat.py.txt r1 manage 4.7 K 2007-10-18 - 12:57 DavidCameron NDGF FTS monitor graph producing script
Unknown file formattmpl index.tmpl r1 manage 0.8 K 2007-10-18 - 12:58 DavidCameron NDGF FTS monitor html template
Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r10 - 2009-10-23 - LionelSchwarz
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback