Ganga Testing Framework

This page documents the re-engineering of the Ganga Testing Framework.

The existing framework is a hack and a patchwork on top of pyunit: http://isscvs.cern.ch/cgi-bin/cvsweb.cgi/ganga/python/Ganga/test/?cvsroot=ganga

It is integrated in the release process and executed from within the ganga-release script: http://isscvs.cern.ch/cgi-bin/cvsweb.cgi/ganga/release/tools/?cvsroot=ganga

Requirements

Configuration

We should be able to do a full validation of a release,using all possible configurations (local,remote repository etc) and have a HTML,XML or TEXT report to be published.

Unit vs System/Integration tests

There are two types of tests:

  • Tests-cases for individual modules (embedded in code as PyUnit tests) used by developers to test specific functionality separately
  • Test-suites (groups of tests) that should do functional testing of components (i,e: Ganga,Core, Ganga.LHCb, Ganga.Atlas)
    • these groups of tests should be organized based on a consistent schema (naming conventions,hierachical structure? )

 
      CoreTests.Bugs
               .Persistency
               .Submission
               .JobTree
               .GPIInvariants
               .Internal
               .Repository
               .Regression
      LHCbTests.Bugs
               .Submission
               .Regression
      AtlasTests.Bugs
                .Submission
                .Regression

Integration with the release process

Multiple criteria for executing groups of tests and testing configurations: For example: "run LHCb.* --config=local.cfg"

In order to validate the release fully, three parts must be executed independently:

    - Core validation  (by Release Manager) 
    - LHCb validation  (by GangaLHCb maintainer) 
    - Atlas validation (by GangaAtlas maintainer)

Reporting

Have a HTML,XML or TEXT report to be published.

The output of different parts should be stored in some shared place (e.g. in the pre-release area) and if multiple groups of tests are ran the output should be"merged" into one, coherent report. It means that each of the persons should be able to say "run full test suite for [Ganga.Core, Ganga.LHCb or Ganga.Atlas]"

Developer's participation

The framework should be designed in such a way which encourages developers to add new tests and which is easy/flexible to use by the release persons.

  • probably the easiest is to have each test in a separate file

Special requirements

  1. Special test cases
    • Multiple run tests - a test case must be executed in a number of ganga sessions, passing some parameters between them
      • example: test_bug_10006.py must be invoked twice to check if the persistent repository commit was succesfull
      • currently it is supported by a home-made class MultipassTest defined in Ganga.test.multipass
    • Regression tests - checking jobs repository backwards compatiblity (reference data = job repository with all types of objects dumped)
      • generation of new reference data must be explicitly controlled (and not automatically triggered!)
      • on the other hand, the validation against existing reference data should be automatic
  2. It should be possible to specify if a given set of tests should be executed in a single ganga sessions (as far as possible, not possible for multipass tests) or if each test should be executed in a separate session. We note that results in both cases are not always equivalent.

TestOOB

Currently we are investigating the TestOOB Python testing framework http://testoob.sourceforge.net which is based on PyUnit and offers additional functionality :
  • test-suites running based on reg-exp (could help to run a entire set of tests automatically, i.e: test Core.*)
  • reporting the test results (TXT,XML,HTML)
  • run tests in parallel in different processes with the ��processes� command-line option.
  • support for non-deterministic tests (pass with probability) (Repeat each test a number of times)

Implementation proposal

1. Every Ganga package (e.g: Core,GangaLHCb,GangaAtlas) will have a special test area (/test subfolder)

 
  ganga/python/
      Ganga/
        test/config #->this subfolder contains different Ganga configurations to be used when running tests
           local.ini 
           remote.ini
           @default->local.ini    
        test/
          testBugs_1.py # -> GangaCore/Bugs/ PyUnit testcase for bug 1
          testBugs_1.ini # -> configuration file for this test-case
          testBugs_nnn.gpi #-> Plain-GPI test 
          testBugs_mmm.gmp #-> GPI Multipass test
          ...
          testPersistency_t1.py
          testPersistency_t1.py
          ...
      GangaLHCb/
        test/config #->this subfolder contains different Ganga configurations to be used when running tests
           local.ini 
           remote.ini
           @default->local.ini    
        test/
           testBugs_nnn.py
           testBugs_mmm.py
           ...
           testSubmission_t1.py
           testSubmission_t2.py
           ...
           testSubmission_t1.py
           testSubmission_t2.py
           ...
           testRegression_nnn.py

       GangaAtlas/
         test/config
          ...
         test/
          ...

2. There are three types of tests, so far:

  • "classic" PyUnit test-cases
  • Plain-GPI testcases (the test is automatically run in a ganga session)
  • Multipass Test-cases (every pass will be automatically run in a separate ganga session)

The following conventions should be used when defining/naming test-case files:

  • Every test-case is defined in a separate file named test<Subcategory>_<test_case_name>.<test_case_type>
    • test_case_type can be py for PyUnit tests, gpi for GPI plain-tests or gmp for GPI multipass tests
  • Every test-case may have a configuration file associated (test<Subcategory>_<test_case_name>.ini) which can override the default testing configuration (see below)

3 .Different test session configuration files resides in .Ganga<Component>/test/config directory. Based on this configurations multiple testing sessions may be launched using a generic testing-driver which will collect and run the tests based on a filter-pattern:

 e.g
    #launch a single testing session (@default.ini) , running just test-cases for Bugs from GangaLHCb
    runTests GangaLHCg.Bugs 
    
    # run all the tests from GangaLHCb using both the local.ini and remote.ini configuration files      
    runTests --config=local.ini,remote.ini  GamgaLHCb.* 

4. Testing Framework:

  • the testing framework itself is a separate package (ganga/python/GangaTest/bin/)
  • the Testing Driver (runTests) will be a python application that collects all the supported testcases (based on a pattern), runs the test-cases, and reports the results.

Usage:

  runTests.py [options] package
   package                              : pattern for selection of tests (e.g.: Core.*)
   [--configs=<list_of_testing_config>] : optional; default to Package/test/config/default.ini; may be a list of files
   [--session=single|shared]      : specify if the selected set of tests should be executed in a single ganga sessions (as far as possible, not possible for multipass tests)
                                          or if each test should be executed in a separate session (default: shared)
   [--report-(text|html|xml)]           : specify to save a the test report (xml|html|xml)
   [--report-outputdir=outputprefix]    : specify an destination directory where the report in generated (default: ./reports)
   [--runid=id]                         : specify an additional information about the run (e.g realease version (4.2.0))

5 Based on reporting options the report is generated in
outputprefix/runid/package
For example:
    reports/4.2.0/GangaCore.Bugs.xml
    reports/4.2.0/GangaLHCb.Bugs.xml
    reports/4.2.0/GangaLHCb.Submission.xml
    reports/4.2.0/GangaAtlas.Bugs.xml
    reports/4.2.0/GangaAtlas.Submission.xml
  

A summarizing tool is needed to generate public reports. The proposed format is the following:

  config1.ini config2.ini
Package PASSED FAILED PASSED FAILED
GangaCore* 10 4 8 2
GangaLHCb 21 1 4 4
GangaAtlas 19 2 1 3
ALL 50 7 13 9
Category PASSED FAILED PASSED FAILED
Bugs 30 4 25 5

* acces to the comprehensive report in which every test is detailed

How to use the new framework

Please check this mini-Guide: GangaTestingFrameworkGuide

-- JakubMoscicki - 14 Jun 2006

-- AdrianMuraru - 27 Jun 2006

Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r13 - 2007-02-27 - AdrianMuraru
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ArdaGrid All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback