Introduction
StatTest is a package to perform automatic regression testing on the content of a ROOT file.
It has been developed by
andrea.dotti@cernNOSPAMPLEASE.ch for the physics validation of Geant4 and integration in the Geant4 Ctest/CDash system, but it is a standalone application that can be used to perform regression testing of any ROOT file. Currently supported objects to test are one dimensional histograms (TH1) and TTrees.
The system works performing a statistical comparison between the same object (histogram -binned distribution- or branch in a tree -unbinned distribution-) in two different files (the current file against a reference file). Which ROOT objects to test, the type of test, the limits to accept/reject a test are specified in a simple configuration file. Differently of other similar systems for automatic data-quality the system is intended to be as simple as possible.
An introduction to the tool can be seen here:
http://indico.cern.ch/getFile.py/access?contribId=3&sessionId=0&resId=0&materialId=slides&confId=186631
The software is available in Geant4 SVN g4tests repository:
https://svnweb.cern.ch/cern/wsvn/g4tests/trunk/verification/StatTest/?
(requires authentication)
Since I've received some requests from outside G4 collaboration to get a copy of the tool, I have cloned it on github:
https://github.com/andreadotti/StatTest
.
The software is written in python and uses pyROOT. It requires the correct version of ROOT and python to be configured. A script that works on lxplus is provided to setup everything: setup-lxplus.sh
Instructions
The package, even if intended to be run integrated with CTest/CDash can be used in standalone on
any ROOT file.
To use the software a configuration file has to be created that specifies: which histograms/TTrees to consider for testing, which statistical comparison should be applied and what are the thresholds to accept/reject a test. The subdirectory: example contains an extended commented example of the syntax of this configuration file.
For example, to test
ALL histograms in a file with the Chi2 test and consider the test
FAILED if p-value<0.1 while
ACCEPTED it if p-value>0.5 (it will be in
NOTPASSED state case in other cases; i.e. intermediate, or "not-sure" state) the configuration file looks like:
Binned = {
'DefaultTestName' : 'BinnedWeighted1DChi2Test',
'DefaultThresholds' : [ 0.1,0.5],
'DefaultReferenceFile' : 'reference.root',
'Histos' : [
{ 'Name' : '.*', },
]
}
#No TTrees to check
UnBinned = {
'DataSet' : [ {'Name':''} ]
}
}
Use in stand-alone mode
Assuming you want to compare file
myfile.root
against the reference file
areferencefile.root
with the configuration file
conf.qa
just type:
python runtests.py conf.qa myfile.root areference.root
Use: python runtests.py -h for a list of options. To produce a PDF file with distributions add the
-g report.pdf
command line option. To obtain a list of supported statistical tests use the
--list
command line option.
Use in Ctest/CDash for Geant4
StatTest can be used in a Geant4 ctest. Assuming you have a test (for example test30), producing a ROOT output you can modify the CMakeLists.txt file of the test as follow to introduce extra ctests (in this case in the PhysicsChecks CDash group) that take advantage of
StatTest:
#---Physics Validaiton-------------------------------------------------------------------------------
# Tests executed only in ctest/cdash PhysicsChecks group
# Testing for PhysicsChecks is done in three steps:
# 1- A test trying to build the application
# 2- A simulation job
# 3- A step in which the output of the simulation is checked against a reference
#Step1 Build executable test
GEANT4_ADD_TEST(test30-build BUILD test30 LABELS PhysicsChecks)
#Step 2 Perform simulation test: runs all models together
GEANT4_ADD_TEST(test30-Bi
COMMAND ${CMAKE_CURRENT_BINARY_DIR}/test30 ${CMAKE_CURRENT_SOURCE_DIR}/physval/macro.in
DEPENDS test3-build
LABELS PhysicsChecks
ENVIRONMENT ${GEANT4_TEST_ENVIRONMENT})
#Step 3 Execute regression testing only if StatTest program is available
#Each "model" is a separate test
find_package(StatTest QUIET)
if(STATTEST_FOUND)
foreach( _model bertini binary bertini_preco)
STATTEST_ADD_TEST( test30-Bi-${_model}-checkOutput
G4TEST test30-Bi
CONFIG ${CMAKE_CURRENT_SOURCE_DIR}/physval/testconf.qa
INPUT ${_model}.root
REFERENCE ${CMAKE_CURRENT_SOURCE_DIR}/physval/${_model}-ref.root
IMG ${_model}.pdf
LABELS PhysicsChecks)
endforeach()
endif()
--
AndreaDotti - 11-Jul-2012