Ganga Testing Framework

Introduction

This page describes Ganga testing framework:

• how the tests are organized
• how to add new tests or modify existing tests
• how to run the Ganga test suite
• how to create HTML reports

Quick HOWTOs

Add and run a bugfix testcase

When fixing a bug reported in Savannah you should add a testcase!

1. Create a GPI test script: Ganga/test/Bugs/SavannahXYZ.gpi
2. Run the test: $ GangaTest/sbin/gangaRunTests Ganga.test.Bugs.SavannahXYZ

Note:

1. if the bug XYZ relates to GangaLHCb or GangaAtlas then put your testcase in GangaLHCb/test/Bugs or GangaAtlas/test/Bugs
2. if you need a special configuration file for your test, then create it in the same directory, e.g.: Ganga/test/Bugs/SavannahXYZ.ini
3. get familiar with other #TypesOfTests
4. if you need some other files for your test, you may create a directory SavannahXYZ and put SavannahXYZ.gpi inside + all the required files.

Add functionality tests

If you have implemented new functionality, then it is good to test it!

For core Ganga functionality use Ganga/test directory. For experiment related functionality, use GangaLHCb/test or GangaAtlas/test

Create a test script in aproperiate directory. Use one of #TypesOfTests.

Ganga/test directory contains the following subdirectories:

• Bugs
• GPI functionality tests use public GPI (majority)
• Internal tests use private ganga API (not in GPI)
• Performance
• Regression: tests check compatibility of the repository across releases
• config: keeps the configuration files

NEW: run the testcase with more control (interactively, with given configuration, etc...)

Running the testcase with =localxml.ini = config: ganga -o[TestingFramework]Config=localxml.ini --test Ganga/test/Bugs/Savannah17080.gpi

Running the same but without testing framework: ganga --config-path=Ganga/test/config/localxml.ini Ganga/test/Bugs/Savannah17080.gpi

Running interactive session with the same configuration: ganga --config-path=Ganga/test/config/localxml.ini

Run test suite for a group of tests

Some examples:

1. Run all bugfix tests in GangaLHCb: $ GangaTest/sbin/gangaRunTests 'GangaLHCb.test.Bugs.*'
2. Run all core performance tests: $ GangaTest/sbin/gangaRunTests 'Ganga.test.Performance.*'
3. Run all public GPI tests: $ GangaTest/sbin/gangaRunTests 'Ganga.test.GPI.*'

If you have a number of related tests, you may create a single directory for all of them.

Produce HTML testing report

During the release process it is usefull to have a complete testing report of the entire test suite:

• Run all tests from different top-level packages (Ganga, GangaAtlas, GangaLHCb) specifying the destination directory for testing reports:

  gangaRunTests --reports_dir=/afs/cern.ch/sw/ganga/reports/ --report_name=4.2.0-beta4 --config:local.ini:remote.ini 'Ganga.test.*'
  gangaRunTests --reports_dir=/afs/cern.ch/sw/ganga/reports/ --report_name=4.2.0-beta4 --config:local.ini:remote.ini 'GangaAtlas.test.*'
  gangaRunTests --reports_dir=/afs/cern.ch/sw/ganga/reports/ --report_name=4.2.0-beta4 --config:local.ini:remote.ini 'GangaLHCb.test.*'

This will produce full HTML reports in /afs/cern.ch/sw/ganga/reports/4.2.0-beta4/html/

Produce XML Difference report

This will produce an xml file, able to be read by the HTML reporter, which will show the differences in test successes and test failures.

• success/success is counted as a success
• failure/failure is counted as a success
• success/failure is counted as a failure and debugging information is provided
• failure/success is counted as a failure and the message “Changed from failure to success in new release” is shown

1. ganga -o[TestingFramework]EnableTestRunner=False EnableXMLDifferencer=True 5.0.4
2. ganga -o[TestingFramework]EnableTestRunner=False EnableXMLDifferencer=True 5.0.4 5.0.2
3. ganga -o[TestingFramework]EnableTestRunner=False EnableXMLDifferencer=True 5.0.4 local local
4. ganga -o[TestingFramework]EnableTestRunner=False EnableXMLDifferencer=True 5.0.4 5.0.2 local local

(1) will difference the current version and the 5.0.4 version, if it's installed. (2) will difference the 5.0.4 and the 5.0.2 version (3) will difference the current version and the 5.0.4 version, if it's installed, using the config=local xml files (4) will difference the 5.0.4 and the 5.0.2 version using the config=local xml files

The idea is to use 1 and 3 in testing, to compare the latest with the last-but-one version, and also to compare the differences of the local config and the Imperial config reports. So:

ganga -o[TestingFramework]EnableTestRunner=False EnableXMLDifferencer=True PREVIOUS_VERSION_NUMBER

and

ganga -o[TestingFramework]EnableTestRunner=False EnableXMLDifferencer=True Current_Version_Number local Imperial

Testing framework

Currently we focus on system testing. This means that we are testing the Ganga system as a whole, including blackbox, integration and functionality tests. All system tests are defined as GPI scripts. Each test is run in a separate Ganga session.

Unit tests cover individual modules/packages avoid dependencies to another part of the system - they will be described later.

The system tests are defined for each top-level Ganga package. CVS links with the latest directory snapshot:

• Ganga/test
• GangaAtlas/test
• GangaLHCb/test

Groups of tests are defined by creating a tree directory structure inside each of the top-level test directories.

Each test-case has a name which reflect the hierarchical structure: e.g. Ganga.test.Bugs.Savannah3 Wildcards may be used for referring to the group of tests: e.g. Ganga.test.Bugs.*

An example of the core Ganga test directory. For up-to-date version, refer to the CVS links above.

ganga/python/
      Ganga/
       test/config #->this subfolder contains different Ganga configurations to be used when running tests
           local.ini 
           remote.ini
           @default->local.ini
        test/Bugs
          Savannah1.gpi  # Plain-GPI test 
          Savannah1.ini    # Specific configuration file for the Savannah1 GPI test
          Savannah2.gpi  
          Savannah3.gpim # GPI Multipass test
         test/JobTree
           CommitUpdate.py  # ->  unittest test file that may define multiple testcases
           Persistency/
              Checkout.gpim
       ...

Types of tests

Plain GPI tests

Simple&plain scripts that make use of GPI commands have .gpi suffix. Just copy and paste commands from Ganga shell into a separate file and the test is created.

Multipass GPI tests

Multipass test (.gpim suffix) automatically run a number of Ganga sessions one after the other and pass some information between them.

Classic example: suppose that you want to make sure that Ganga persistenly commits the changes to the jobs. In the first pass you create the job, in the second pass you modify it, in the third pass you see if you see your modifications.

GPI Multipass tests should derive the pytf.lib.Multipass base class and may define multiple pass-methods. Every step (pass) of the test is required to be completed successfully in order to pass the the whole test. The file containing the test should have .gpim suffix and the name of the class should be the same as the name of the file.

File: MyMP.gmpi

from GangaTest.Framework.tests import MultipassTest

class MyMP(MultipassTest):

    def __init__(self):
        MultipassTest.__init__(self,2)

    def pass1(self):
        """ First pass of this test"""

    def pass2(self):
        """ Second pass of this multipass test"""

Python unittest classes.

Python unittest test-cases (defined in file with Test prefix and having .py extension) allow to group tests into suites in a signle file. Every test method test* is run in a separate Ganga session.

Python unittest test-cases may be defined in the unittest TestCase file (.py extension) by inheriting the GangaGPITestCase base class and having multiple test methods defined in. It is required for the class name for the TestCase to have the same name as the file in which is it is defined

File: TestJobManagement.py

  
from GangaTest.Framework.tests import GangaGPITestCase
#GangaTest.Framework.utils defines some utility methods
#from GangaTest.Framework.utils import sleep_until_completed,sleep_until_state
import unittest

class TestJobManagement(GangaGPITestCase):
     def testOne(self):
         ...
     def testTwo(self):
         ...
     def testThree(self):
         ...

Parallel GPIP tests

GPIP testcases(defined in file with Test prefix and having .gpip extension), the definition of GPIP test type is similar to pyunit test type, but all the testcases defined in a GPIP test file will be run logicially in parallel to improve the multiple(and long) timeout problem(eg. TestRoot.py pyunit testcase)

It is also different from pyunit testcase, all the test methods are run in the same Ganga session. And just like pyunit test type, the class name and GPIP file name must be the same.

File: TestGPIP.gpip

  
from GangaTest.Framework.tests import GangaGPITestCase, ICheckTest

class TestGPIP(GangaGPITestCase):
    def testFailInPreparation(self):
        ...
        return jobTest

    def testTimeoutTest(self):
        ...
        return jobTest

    def testFailInJobTest(self):
        ...
        return failedJobTest 
    ...

class TimeoutJobTest(ICheckTest):
    def __init__(self):
        ...
    def isReadyForCheck(self):
        ...
    def checkTest(self):
        ...

For more detail, please have a look at the GPIP test slides below.

• GPIPTest.pdf: The introduction to GPIP test in ganga testing framework

Testcase configuration files

• Multiple test session configuration files may be placed in /test/config folder. Based on this configurations multiple testing sessions may be launched by invoking the test runner ( gangaRunTests ) with multiple Ganga configuration files.
• Each test-case may set specific configuration parameters in a file having the same name as the test-case and .ini extension. This file "overrides" the session configuration file in the same way as with ganga --config-path command line option

Example:

Configuration file for multipass test MyMP.gpim

File MyMP.ini :

 [Logging]
  _format=TERSE

[TestFramework]
#fail test if it does not terminate in timeout period [sec]
timeout=3000
...

* Note The GPI and GPIM test types should be used when defining new Ganga test-cases, because there is a dirrect mapping between the name of test and the actuals filename in which the test is defined.

Running Tests

Run Ganga system tests

System tests are located in test/ directory of each Ganga top level package (e.g Ganga,GangaLHCb,GangaAtlas). This way of runner invocation is mainly used during the release process - but may be used by the developers/users during their testing phase as well.

Usage:
  ganga --test SELECTION-PATTERN 

SELECTION-PATTERN is relative path pointing to tests under <PACKAGE>/test directory. 
You can use wildcards characters to match a group of tests.

Examples:
1. Run all GPI tests of Ganga Core package:
  ganga --test Ganga/test/GPI

2. Run all GPI tests of Ganga Core package testing the Config object:
  ganga --test Ganga/test/GPI/Config* 

Notes :

1. The config-path used to run each test-case is the following:
TestcaseName.ini:package-config:.gangarc:$GANGA_CONFIG_PATH , where:

• TestcaseName.ini, if exists it defines the test-case specific configuration options
• package-config : is the default testing configuration for a package : /test/config/default.ini. [TestingFramework]Config option may be used to override this.
• the rest of the configuration path is the usual one inherited from the environment

2. In this mode, test-runner generates XML report with the test-case results and automatically calls the HTML report generator.

Run tests from arbitrary locations (local tests)

This is a convenience method mainly used by the test-case author during the test-case developing phase. The parameter for the test-runner can be either a path to a given testcase or a directory containing a set testcases:

 ganga --test TEST_PATH 

Example:
1. Run Mytestcase.gpi
   ganga --test /home/adim/mytests/MyTestcase.gpi
2. Run the single test testTwo inside the TestJobManagement.py Python unitest class 
   ganga --test /home/adim/mytests/TestJobManagement.testTwo

Notes :

1. The config-path used to run each test-case is the following:
TestcaseName.ini:.gangarc:$GANGA_CONFIG_PATH , where:

• TestcaseName.ini, if exists it defines the test-case specific configuration options
• the rest of the configuration path is the usual one inherited from the environment

2. (Recommendation) By default the testing framework calls cleanup methods (i.e job and templates registries/repository/workspace are purged) after each testcase run to make sure that test-cases are not interfering each other. SO make sure you are invoking the testing framework using a different Ganga configuration (i.e different Local workspace,repository username,etc) in order to avoid losing your precious jobs.

3. In this mode, the test-runner DOES NOT generates XML/HTML reports, the output being displayed only to the console.

(Re)generating the HTML reports

When running ganga system tests the test-runner produces an XML report in ReportsOutputDir with the test-cases failure/success statuses. These XML reports together with the coverage analysis reports are automatically translated into HTML pages. However, there might be some cases when HTML reports regeneration is needed, for example when testing different packages on different machines and a final merge is required. To do this you need to merge the ReportsOutputDir and LogOutputDir dirs first and, then, run the HTML reports generator using [TestingFramework]ReportsOnly option of the testing framework:

ganga --test -o[TestingFramework]EnableHTMLReporter=True -oEnableTestRunner=False

Test runner configuration

Bellow is a verbatim copy of the configuration section used to set control the test-runner behavior:

[TestingFramework]

## LOCAL TESTS:
# Enable running of local tests from arbitrary locations
# Default : true
#EnableLocalTests = True

## SYSTEM TESTS:
# Enable running of Ganga system tests located in PACKAGE/test directories (e.g GangaLHCb/test)
# Default : True
#EnableSystemTests = True

# When running system tests, multiple base configurations may be used by setting a colon separated
# list of these files
# These files are searched in PACKAGE/test/config directories
# Default: default.ini
#Config = 'default.ini'

# Unique identifier for the current testing session
# Default: latest
#RunID = 'latest'

# Reports directories
#LogOutputDir = '/path/reports/latest/output'
#ReportsOutputDir = '/path/reports'

# Skip the test-runner and run only the reporting tool
# Default: False
#ReportsOnly = False

#enable/disable test-runner
EnableTestRunner=True
# enable/disable html reporter
EnableHTMLReporter=False
#enable/disable xml differencer
EnableXMLDifferencer=False

-- UlrikEgede - 12 Aug 2008