RelValReport

Complete: 3

RelValReport_v2

Motivation

A procedure to automate the reports production is foreseen in the mission of the Performance Taskforce. RelValReport is meant to fulfill this requirement easing the creation of reports by means of the PerfReport tool and optionally, the PyReleaseValidation Package. It is a tool able to run one process or several processes specified in a ASCII file, benchmark them with any of the profiling tools (IgProf, ValgrindFCE and EdmSize) and produce the performance reports in form of static html pages or in form of a database. All the options can be specified from commandline.

Goal of the page

The goal of this page is to show the user how to produce with an automated procedure performance profiles and reports by means of RelValReport.

Quick start

First of all, check out the script from the CMSSW repository. It is under Configuration/PyReleaseValidation/test, starting from tag V00-05-00.
project CMSSW
cvs co -r V00-05-00 Configuration/PyReleaseValidation/test/relvalreport_v2.py

Therefore, install PerfReport . For the time being, an installation on afs is also present in:

  • PR2.1:
    /afs/cern.ch/user/d/dpiparo/w0/perfreport2.1installation
  • PR3 -- EXPERIMENTAL!:
    /afs/cern.ch/user/d/dpiparo/w0/perfreport3installation

Then, you should insert in the settings of the directories of your PR installation(s), if they are different from the ones above. You can find the settings just on the top of the script.

#! /usr/bin/env python
r'''
Relvalreport_v2: a script to run performance tests and produce reports in a automated way.
'''
# Configuration parameters:#############################################################
# Perfreport 3 and 2 coordinates:
PR3_BASE='/afs/cern.ch/user/d/dpiparo/w0/perfreport3installation/'
PR3=PR3_BASE+'/bin/perfreport'# executable
PERFREPORT3_PATH=PR3_BASE+'/share/perfreport' #path to xmls

PR2_BASE='/afs/cern.ch/user/d/dpiparo/w0/perfreport2.1installation/'
PR2=PR2_BASE+'/bin/perfreport'# executable
PERFREPORT2_PATH=PR2_BASE+'/share/perfreport' #path to xmls
########################################################################################
Now set the CMSSW environment. You are ready to go!

Let's start with a report of the process described in rhe hypotetical myconfig.cfg, using IgProf in performance mode:

relvalreport_v2.py -c'cmsRun myconfig.cfg' -R -P -pIgProf_perf  -yPERF_TICKS -o MyReportDir -n MyProfile 
Ok, now you can browse your brand-new profile in IgProf_perf. Let's suppose now you want a memory profile together with its report:
relvalreport_v2.py -c'cmsRun myconfig.cfg' -R -P -pIgProf_mem  -yMEM_TOTAL -o MyReportDir_Memtotal -n MyProfile_Memtotal 
Easy. Now you want to make another memory profile, using a different counter, let's say MEM_LIVE. You don't need to produce another profile, you have it already. Therefore:
relvalreport_v2.py -c'cmsRun myconfig.cfg' -R  -pIgProf_mem  -yMEM_LIVE -o MyReportDir_Memlive -n MyProfile_Memlive
As you can notice, the -P switch disappeared.

Now you are ready to run the program. You can type, as usual, for more advice:

relvalreport_v2.py -h

Detailed description of the options available:

The full list of options available for relvalreport is the following:
  • -h, --help
  • -pPROFILER, --profiler=PROFILER
    • -yIGPROF_COUNTER
  • -iINFILE, --infile=INFILE
  • -cCOMMAND, --command=COMMAND
  • -R, --Report
  • -P, --Profile
  • -nPROFILE_NAME, --profile_name=PROFILE_NAME
  • -oOUTPUT_DIR, --output=OUTPUT_DIR
  • --executable=EXECUTABLE
  • --db EXPERIMENTAL!!!!!!!!
But let's dive into a detailed description of every one of them.

-h, --help

Shows the help.

-pPROFILER, --profiler=PROFILER

Set the profiler to use. Currently, the available profilers are: ValgrindFCE, IgProf_perf, IgProf_mem, Edm_Size. You can also set the counter you want to use in case you select IgProf_mem. This operation is carried out with the -y option. If the profiler is different from IgProf, the -y is ignored.

-iINFILE, --infile=INFILE

The most interesting feature of RelvalReport_v2. This options allows the user to specify an ASCII file where the processes to benchmark are collected: this mode is the principal usecase of relvalreport_v2.py. Let's start with an example:
 
# My list of candles:
 
cmsDriver.py MU- -sSIM  -e10_20 @@@ IgProf_perf.PERF_TICKS @@@ QCD_sim_IgProfperf
# The PyRelval Executable is launched 
cmsDriver.py MU- -sRECO -e10_20 @@@ ValgrindFCE @@@ QCD_reco_Valgrind
cmsRun mycfg.cfg @@@  CMS.IgProf_mem.MEM_TOTAL @@@ Mycfg
The file is divided in 3 columns separated by '@@@':
  1. The process to benchmark
  2. The profiler to use
  3. The metadata relative to the procedure
All the comments begin with '#'. At this point you should just type:
relvalreport2_v2.py -i my_candles_list.txt -R -P
And the processes will be run. You don't need to specify anything with the -c switch: the commands are contained in the ASCII file! Points 2 and 3 need some more detailed explainations. Since from an IgProf profile one can produce several reports according to the counter selected, you can specify the counter that suits your needs with a '.'. Therefore if you want an IgProf profile and a report according to the MEM_TOTAL counter you will write IgProf_mem.MEM_TOTAL in the second column. The metadata are used to create automatically directories and profiles names. If the -o or -n options are specified as well, the program will combine the two in an intelligent way. This is useful if you want to use the same set of candle processes in different contexts (e.g different machines). For what concerns the EdmSize profiler, the name of the process is replaced by the name of the rootfile.

-cCOMMAND, --command=COMMAND

Here you set the program you want to benchmark. If you need more than one word to describe it (i.e. cmsRun myconfig.cfg), type
-c 'program.exe  -with options.txt'
See Quick Start for more details.

-R, --Report

The switch to enable the report production.

-P, --Profile

The switch to enable the profile production. You might need to produce several reports out from a single profile (e.g. IgProf_mem)

-nPROFILE_NAME, --profile_name=PROFILE_NAME

The name of the input (output) profile. This is the name of the file that will be read by perfreport or/and written by the profiler.

-oOUTPUT, --output=OUTPUT

Specifies the name of the output directory for the html static report.

--executable=EXECUTABLE Experimental

The CMSSW user should not be interested in this switch. This switch tells IgProf which executable to benchmark. It is present since no particular reason forces a user to use the machinery developed by the Performance Taskforce just for cmsRun..

--db -- Experimental

This switch enables the use of perfreport3 and the filling of a db instead of the production of static html pages. At the moment is experimental since PR3 needs some more tests by the Perfreport Team. The name of the db will be the one specified by the -o switch.

Contacts

If you experience any kind of problems with RelValReport, please contact me immediately. Write to: danilo.piparo@NOSPAMcernNOSPAMPLEASE.ch

Older Documentation

The old, now-deprecated version of this page has been moved to: SWGuideRelvalReportOld

Review Status

Reviewer/Editor and date Comments
DaniloPiparo - 30 May 2007 page created
JennyWilliams - 31 Oct 2007 moved old info into archive page

Responsible: Main. Danilo Piparo
Last reviewed by: DaniloPiparo - 30 October 2007

Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r7 - 2007-11-01 - DaniloPiparo



 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback