Introduction

StatTest is a package to perform automatic regression testing on the content of a ROOT file. It has been developed by andrea.dotti@cernNOSPAMPLEASE.ch for the physics validation of Geant4 and integration in the Geant4 Ctest/CDash system, but it is a standalone application that can be used to perform regression testing of any ROOT file. Currently supported objects to test are one dimensional histograms (TH1) and TTrees.

The system works performing a statistical comparison between the same object (histogram -binned distribution- or branch in a tree -unbinned distribution-) in two different files (the current file against a reference file). Which ROOT objects to test, the type of test, the limits to accept/reject a test are specified in a simple configuration file. Differently of other similar systems for automatic data-quality the system is intended to be as simple as possible.

An introduction to the tool can be seen here: http://indico.cern.ch/getFile.py/access?contribId=3&sessionId=0&resId=0&materialId=slides&confId=186631 The software is available in Geant4 SVN g4tests repository: https://svnweb.cern.ch/cern/wsvn/g4tests/trunk/verification/StatTest/? (requires authentication)

Since I've received some requests from outside G4 collaboration to get a copy of the tool, I have cloned it on github: https://github.com/andreadotti/StatTest.

The software is written in python and uses pyROOT. It requires the correct version of ROOT and python to be configured. A script that works on lxplus is provided to setup everything: setup-lxplus.sh

Instructions

The package, even if intended to be run integrated with CTest/CDash can be used in standalone on any ROOT file. To use the software a configuration file has to be created that specifies: which histograms/TTrees to consider for testing, which statistical comparison should be applied and what are the thresholds to accept/reject a test. The subdirectory: example contains an extended commented example of the syntax of this configuration file. For example, to test ALL histograms in a file with the Chi2 test and consider the test FAILED if p-value<0.1 while ACCEPTED it if p-value>0.5 (it will be in NOTPASSED state case in other cases; i.e. intermediate, or "not-sure" state) the configuration file looks like:

Binned = {
    'DefaultTestName' : 'BinnedWeighted1DChi2Test',
    'DefaultThresholds' : [ 0.1,0.5],
    'DefaultReferenceFile' : 'reference.root',
    'Histos' : [
        { 'Name' : '.*', },        
        ] 
    }

#No TTrees to check
UnBinned = {
    'DataSet' : [ {'Name':''} ]
    } 
}

Use in stand-alone mode

Assuming you want to compare file

myfile.root

against the reference file

areferencefile.root

with the configuration file

conf.qa

just type:

python runtests.py conf.qa myfile.root areference.root

Use: python runtests.py -h for a list of options. To produce a PDF file with distributions add the

-g report.pdf

command line option. To obtain a list of supported statistical tests use the

--list

command line option.

Use in Ctest/CDash for Geant4

StatTest can be used in a Geant4 ctest. Assuming you have a test (for example test30), producing a ROOT output you can modify the CMakeLists.txt file of the test as follow to introduce extra ctests (in this case in the PhysicsChecks CDash group) that take advantage of StatTest:

#---Physics Validaiton-------------------------------------------------------------------------------
# Tests executed only in ctest/cdash PhysicsChecks group
# Testing for PhysicsChecks is done in three steps:
# 1- A test trying to build the application
# 2- A simulation job
# 3- A step in which the output of the simulation is checked against a reference

#Step1 Build executable test
GEANT4_ADD_TEST(test30-build BUILD test30 LABELS PhysicsChecks)

#Step 2 Perform simulation test: runs all models together
GEANT4_ADD_TEST(test30-Bi 
                          COMMAND ${CMAKE_CURRENT_BINARY_DIR}/test30 ${CMAKE_CURRENT_SOURCE_DIR}/physval/macro.in 
                          DEPENDS test3-build
                          LABELS PhysicsChecks
                          ENVIRONMENT ${GEANT4_TEST_ENVIRONMENT})

#Step 3 Execute regression testing only if StatTest program is available
#Each "model" is a separate test
find_package(StatTest QUIET)
if(STATTEST_FOUND)
        foreach( _model bertini binary bertini_preco)
                 STATTEST_ADD_TEST( test30-Bi-${_model}-checkOutput
                                G4TEST test30-Bi
                                CONFIG ${CMAKE_CURRENT_SOURCE_DIR}/physval/testconf.qa
                                INPUT ${_model}.root
                                REFERENCE ${CMAKE_CURRENT_SOURCE_DIR}/physval/${_model}-ref.root
                                IMG ${_model}.pdf
                                LABELS PhysicsChecks)
        endforeach()
endif()

-- AndreaDotti - 11-Jul-2012