How to use and extend Ganga Robot

This document refers to the Ganga Robot code tagged GangaRobot-0-4 in CVS, which is expected to be included in Ganga 4.4.0 release.

Introduction

Ganga Robot is a tool for running a user-defined list of actions within the context of a Ganga session, where the actions are defined by implementations of an action interface.

The robot is well suited to performing complex repetitive use-cases that involve submitting jobs to the grid, extracting data about the jobs and the grid environment, and reporting statistics on the extracted data. A typical use-case would be to periodically monitor the end-to-end execution of a set of standard jobs submitted via Ganga.

Using Ganga Robot

When the robot is run, it creates a run id based on the current time (UTC) and then sequentially executes each of the actions specified in its configuration, passing the run id to the execute() method of those actions.

Default Configuration

Some generic actions are included with the robot. In particular, there are core actions for submitting exported jobs, waiting for jobs to finish, extracting common data about jobs, and reporting statistics on the extracted data; these core actions are used in the default configuration.

Extract of default configuration GangaRobot/ROBOT.INI:

[Robot]
Driver_Run = ['submit', 20, 'finish', 'extract', 'report']
Driver_Action_submit = GangaRobot.Lib.Core.CoreSubmitter.CoreSubmitter
Driver_Action_finish = GangaRobot.Lib.Core.CoreFinisher.CoreFinisher
Driver_Action_extract = GangaRobot.Lib.Core.CoreExtractor.CoreExtractor
Driver_Action_report = GangaRobot.Lib.Core.CoreReporter.CoreReporter

N.B. GangaRobot/ROBOT.INI also contains some entries to specify the location of the repository and workspace in ~/gangadir/robot to avoid modifying your user repository and workspace.

Usage:

ganga --config-path=GangaRobot/ROBOT.INI robot run [pattern]...

Performs the following actions:

• load and submit 'Hello World' or specified exported jobs
• sleep 20 seconds
• wait for jobs to finish based on job status (timeout 1 hour)
• extract generic data to XML in ~/gangadir/robot/extract/
• generate text and html reports based on extract data in ~/gangadir/robot/report/

Outputs: extract, report

Examples (patterns are absolute or local/codebase relative paths to exported jobs):

  ganga --config-path=GangaRobot/ROBOT.INI robot run
  ganga --config-path=GangaRobot/ROBOT.INI robot run ~/jobs1.txt ~/jobs2.txt
  ganga --config-path=GangaRobot/ROBOT.INI robot run ~/jobs/*.txt
  ganga --config-path=GangaRobot/ROBOT.INI robot run GangaRobot/exports/local-echo-jobs.txt

Custom Configuration

Some utility actions are included with the robot. In particular, there is an action which emails files. Here we see how to modify the robot configuration to add an action which emails the report to specified recipients.

Extract of custom configuration ~/MYROBOT.INI (which should be copied from GangaRobot/ROBOT.INI):

[Robot]
Driver_Run = ['submit', 20, 'finish', 'extract', 'report', 'email']
Driver_Action_email = GangaRobot.Lib.Ext.FileEmailer.FileEmailer

FileEmailer_From = sender@domain.org
FileEmailer_Recipients = recipient@domain.org
FileEmailer_TextFile = ~/gangadir/robot/report/${runid}.txt
FileEmailer_HtmlFile = ~/gangadir/robot/report/${runid}.html

N.B. See GangaRobot/ROBOT.INI for fully documented configuration options.

Usage:

ganga --config=~/MYROBOT.INI robot run [pattern]...

Performs the following actions:

• load and submit 'Hello World' or specified exported jobs
• sleep 20 seconds
• wait for jobs to finish based on job status (timeout 1 hour)
• extract generic data to XML in ~/gangadir/robot/extract/
• generate text and html reports based on extract data in ~/gangadir/robot/report/
• email report to specified recipient.

Examples (patterns are absolute or local/codebase relative paths to exported jobs):

  ganga --config=~/MYROBOT.INI robot run
  ganga --config=~/MYROBOT.INI robot run ~/jobs1.txt ~/jobs2.txt
  ganga --config=~/MYROBOT.INI robot run ~/jobs/*.txt
  ganga --config=~/MYROBOT.INI robot run GangaRobot/exports/dirac-root-jobs.txt

Exceptions

GangaRobotContinueError

Issue a warning but continue with next action

GangaRobotBreakError

Issue a warning and start over from beginning if running in infinite loop. If no infinite loop, the Robot will quit.

GangaRobotFatalError

The Robot will quit.

The default behaviour if any other exception is raised is the same as GangaRobotFatalError but this can be changed in the initialisation. See the ROBOT.INI file for an example.

Extending Ganga Robot

Architecture

The framework is very simple and consists of a Driver class containing a list of IAction implementations.

A set of abstract base action implementations provides a basis for implementing submit / extract / report actions (using the Template Method pattern). Concrete submit / extract / report actions simply extend the base actions and implement the handle...() methods. The core actions are examples of such concrete actions. The Node and Report objects are data models for the extracted data and reports respectively.

See API documentation for more details: GangaRobot-0-4-API.zip.

Adding new actions

Here we see how to add new actions. Our demo actions include an extractor which adds a value to the extracted XML data for each job, and a reporter which adds a line recording the average of these values.

Action code (classes must be in modules in your PYTHONPATH):

class DemoExtractor(BaseExtractor):
    
    def __init__(self):
        self.chain = [CoreExtractor(), self]

    def handlejobnode(self, jobnode, job):
        dn = jobnode.addnode('demo')
        dn.addnode('value', random.randint(0,100))

class DemoReporter(BaseReporter):
    
    def __init__(self):
        self.chain = [CoreReporter(), self]
    
    def handlereport(self, report, runnode):
        report.title = 'Demo Report'
        values = runnode.getvalues('job.demo.value')
        total = 0
        for value in values:
            total += int(value)
        avg = total / max(len(values), 1)
        report.addline('Average value: %d' % avg)

Extract of custom configuration ~/DEMOROBOT.INI (which should be copied from GangaRobot/ROBOT.INI):

[Robot]
Driver_Action_extract = GangaRobot.Lib.Example.Demo.DemoExtractor
Driver_Action_report = GangaRobot.Lib.Example.Demo.DemoReporter

Usage:

ganga --config=~/DEMOROBOT.INI robot run [pattern]...

Outputs: extract, report

• David Tuckett - 20 Jul 2007
• UlrikEgede - 19 Aug 2008