CMS Performance Suite Guide

Complete:

Introduction

This page is meant to document the performance suite of tools developed (originally for the simulation group) for CMSSW. The guide to the previous revisions of the performance suite are here

The profiling tools currently used are CMS.IgProf, Valgrind (Callgrind and MemCheck) and EdmEventSize, but also simple parsers that use the output of framework services like Timing, SimpleMemoryCheck, MessageLogger (CMS.TimeReport).

Goal of the Performance Suite

The goal of this suite is to automate the performance profiling of CMSSW applications and to facilitate the analysis of performance measurements, including the ability to perform regression analysis across different CMSSW releases.

It is especially aimed at becoming a standard tool to be run regularly by the Release Validation team for the Step1, Step2 and GEN-SIM,DIGI step-by-step. Besides its use in the Release Validation process it can be used by developers to quickly run performance tests on various candles in a 1 command line fashion.

Changes

The previous performance suite was mostly written in Perl; using Python integrates it better into CMSSW, enables us to use PyROOT and improves robustness. The suite has been rewritten into Python and includes several new tools that will be useful for us. Previously, Performance Suite profiled each step until the required step(s) for profiling had been profiled. To avoid this inefficient process the suite has been modified so that profiling is only performed for those steps that are required to be profiled. Prerequisite steps are not profiled but are run in one-shot using cmsDriver.py. In addition, using a "-" between steps when specifying those to be profiled by cmsPerfSuite.py, will perform one-shot profiling that profiles those steps as a whole rather than profiling each step individually. The performance suite now also includes mechanisms for producing performance change information between releases (see below). Performance report 2.0.1 is also used and, when performing regression, changes in performance above a certain threshold (defined in an XML file) can be highlight.

The major change in the last version of the Performance Suite (from V01-04-02) is that the suite has been re-designed to work on a command-line driven approach to provide the most flexible configuration via options and help with the maintenance.

How to run the Performance Suite

After setting up the current release with

cmsenv

and changing to the preferred working directory, the performance suite help can be launched with:

cmsPerfSuite.py -h

Note that while the examples in the usage may not be up to date in the future, the options documentation will automatically be kept up to date. Please notify Gabriele Benelli if you notice anything going out of synch here.

There are a number of use cases (described below in more detail), from RelVal DataOperations operator (see instructions here), to individual developer testing a particular step in 1 candle only.

Here's an example the typical commands used in the RelVal fashion, for example for Step1 in CMSSW_2_2_1:

cmsPerfSuite.py --step GEN-HLT --timesize 100 --igprof 5 --memcheck 5 --RunTimeSize MinBias,TTbar --RunIgProf TTbar --RunMemcheck TTbar --RunDigiPileUp TTbar --PUInputFile "/store/relval/CMSSW_2_2_1/RelValMinBias/GEN-SIM-DIGI-RAW-HLTDEBUG/IDEAL_V9_v2/0001/101C84AF-56C4-DD11-A90D-001D09F24EC0.root" --cmsdriver="--conditions FrontierConditions_CMS.GlobalTag,IDEAL_V9::All --eventcontent RAWSIM" >& cmsPerfSuite.log&

(Note: you are advised to redirect the output to a file named "cmsPerfSuite.log" so that later scripts can pick up additional information)

This command will profile Step1 (GEN,SIM,DIGI,L1,DIGI2RAW,HLT). The TimeSize tests (Timing, SimpleMemory and EdmSize) will be run on 100 events, for the MinBias and TTbar candles, the IgProf tests (IgProf with Perf, MemTotal and MemLive counters) will be run on 5 events for candle TTbar, Valgrind Memcheck tests will be run on 5 events for candle TTbar. All the tests above that are run on candle TTbar will also be re-run with PILE UP, using as input file for the pile up events the file specified by the --PUInputFile option. All the information will be placed in the current working directory (subdirectory structure according to candle_test). The publication of these results to allow for html browsing is done with a separate script (cmsPerfPublish.py).

The necessary options to be set are the --step one, one of the --RunTimeSize, --RunIgProf, --RunMemcheck, --RunCallgrind options, with its respective --timesize, --igprof, --memcheck, --callgrind options. In case of --RunDigiPileUp option the --PUInputFile is also necessary. Finally the --cmsdriver option is also necessary, this option sets necessary cmsDriver.py options and offer the ability to add future cmsDriver.py option that may become necessary in the future. At the moment the cmsPerfSuite.py --cmsdriver option is necessary since the cmsDriver.py --conditions and --eventcontent options need to be set.

All the tests above will be performed by default on cpu1, while submitting an infinite loop of cmsScimark benchmarks on the spare cores (for a 4 core machine they would be cpu0,cpu2,cpu3) to ensure the load conditions are always the same and the results can be compared when running on the same machine. The number of cores of the machine, by default, is read directly from the /proc/cpuinfo file of the machine . On cpu1, the tests selected by the user will be, by default, preceded and followed by running 10 times the cmsScimark and 10 times the cmsScimarkLarge benchmark, in order to keep track of the machine performance of the core selected for running. The behaviour in terms of pre and post cmsScimark running can be modified with the --cmsScimark and --cmsScimarkLarge options that can be set to 0 to avoid running these tests on the core on which the measurement will be done. For the number of cores and the running of infinite loops of cmsScimark on them, they can be also configured with the options --cores (that sets the number of cores on the machines), --cpu (that specifies on which core we want to run, e.g. --cpu 2 will have the suite run on cpu2), --notrunspare or -n (not to run on the spare cores).

Another default behavior is to create a tarball at the end of the running that contains all the results of the suite in the current directory. It is possible to set the --archive or -a option to point to a CASTOR directory where to archive the tarball, in case the archiving is successful the tarball is deleted from the local directory. By default the suite will try to archive it for RelVal in an area were only RelVal can write, so if the -a option is not set by non/RelVal users, the tarball will be left in the local directory.

There is also a regression option -r or --prevrel that allows to run a regression analysis between the current run of the performance suite and a previous one. The option specifies the working directory of the previous run of the performance suite. There are separate scripts that allow the regression to be done in a second step, and also tools that allow the regression between individual tests, without comparing the whole performance suite run (see below).

A few use cases

Release Validation

As part of the Release Validation effort the performance suite is run on Step1, Step2 and step-by-step on GEN-SIM,DIGI to measure and keep track of the software performance. The RelVal operator instructions to run the performance suite are kept here, and this constitutes one of the main use cases of the performance suite. The results of it are published here split by directory name. There is also a Regression directory designed to collect a few regression analyses run across releases.

Release Integration

As part of the Release Integration effort the performance suite is used for two types of tests:

• Check that the processing chain in cmsDriver.py is not broken, i.e. if one uses the output of the GEN-SIM step as input of DIGI, and the output of each step as input of the successive step the chain does not crash (this is necessary for testing the performance of individual steps)
• Once a week to test the performance of the currently available integration build, comparing it with the one in the same series from the previous week.

Current IB tests:

Test	Step	TimeSize	TimeSize PileUp	CMS.IgProf	CMS.IgProf PileUp	Memcheck	Memcheck PileUp	Callgrind	Callgrind PileUp	Typical run time (hrs)
GEN-DIGI2RAW	GEN-SIM-DIGI-L1-DIGI2RAW	600 MinBias, 100 TTbar	100 TTbar	51 TTbar	51 TTbar	1 TTbar	1 TTbar	N/A	N/A	N/A
HLT	HLT	6000 MinBias, 1000 TTbar	1000 TTbar	201 TTbar	201 TTbar	50 TTbar	50 TTbar	N/A	N/A	N/A

Developer's tests

The suite can be used with its simple command line interface to set up a variety of performance tests that are routinely ran by several developers in the software development and release integration efforts. It can be used to run only TimeSize type of tests on a large number of events, or to run IgProf tests on multiple candles, with and without some development code, and a regression analysis can be run between the results to measure the performance effect of the new code, the results being published in html format.

Architecture benchmarking

There are a number of additional scripts contained in the Validation/Perfomance packaged that use the suite as an architecture benchmarking suite. The idea is to use CMSSW to load different architecture machines and use it to benchmark them directly with HEP code, instead of using industry standard benchmarks (SPEC). The suite is not yet complete, but the performance suite as it is can be used (and has been used in the context of the HEPiX cpu benchmark working group) to benchmark architectures.

Command line options description

Let's start with the command line description:

• --step

  This option is used to specify on which processing steps should the performance profiling be done. The actual allowed steps are maintained in the cmsPerfCommons.py file (Validation/Performance/scripts), at the time of writing they are: GEN-SIM, DIGI, L1, DIGI2RAW, HLT, RAW2DIGI-RECO. Combinations of these steps are allowed. Possible options are:

  • --step DIGI
    

    In this case the performance suite will only profile the DIGI step and will run the GEN-SIM step necessary as input, without profiling it.

  • --step GEN-HLT
    

    Using the hyphen notation, all steps between the GEN step and the HLT one will be run in one shot and profiled as one (i.e. the cmsDriver.py commands will be run with --step GEN,SIM,DIGI,L1,DIGI2RAW,HLT)

  • --step GEN-SIM,DIGI,L1,DIGI2RAW,HLT
    

    Specifying all steps individually, the performance suite will profile them step by step, i.e. it will profile GEN-SIM, take its output as input for the DIGI step, profile DIGI, take its output for the L1 step etc until HLT is profiled.

  • --step GEN-SIM,L1
    

    In this case, the suite will run the GEN and SIM step at once and it will profile it, then it will run the DIGI step on its output, without profiling it, and finally it will run the L1 step profiling it. This is useful when only some steps need to be profiled, while the intermediate steps don't need to be.
  Note two potential problems:
  • by design cmsDriver.py does not allow to run HLT and RECO steps in the same cmsDriver.py command!
  • for historical reasons it is not possible to profile the GEN step alone at the moment (this might change in the near future).
• --RunTimeSize

  This options is used to select among the allowed candles the ones on which to run the "TimeSize" profiling. TimeSize profiling consists of Timing and SimpleMemory service profiling (including the TimeReport Summary) and EdmEventSize analysis of the root files produced. These are the most basic tests that give a first measurement of CPU Time and memory (VSIZE, RSS) performance together with an analysis of the size of the products stored (file size on disk). The number of events for this kind of testing is configurable by the -t or --timesize option below. Possible values for --RunTimeSize are:

  • --RunTimeSize CMS.MinBias
    

    Run TimeSize tests on MinBias candle.

  • --RunTimeSize CMS.MinBias,SingleMuMinusPt10,TTbar
    

    Run TimeSize tests on the three candles above. Note how they are simply separated by a comma ",". Of course these candles need to be among the allowed ones, maintained in Validation/Performance/scripts/cmsPerfCommons.py.
• --timesize (or -t)

  This option sets the number of events for the TimeSize tests. For example:

  • --timesize 100
    

    Run 100 events for all TimeSize tests configured by --RunTimeSize option above.
• --RunIgProf

  This option sets the candles (among the allowed ones, as specified in RunTimeSize), on which to run IgProf tests. IgProf tests consist of Perf, MemTotal and MemLive tests and an analysis of MemTotal and MemLive (IgProfAnalyse). The IgProf tests give information on CPU performance at the function level, pointing to the code (Perf), and on peak memory allocation (MemTotal) and allocated memory on job exit (MemLive). The IgProf results are reported using PerfReport into static html pages that allow for very detailed inspection of the results (by function and callstack for example). The Perf tests don't imply significant penalty in running time, while the memory profiling (MemTotal, MemLive) can impact running time by over 20%. The number of events for these tests is set with the option -i or --igprof below. The use of this command is similar to --RunTimeSize:

  • --RunIgProf TTbar
    

    Run IgProf tests on TTbar candle only.

  • --RunIgProf CMS.MinBias,CMS.HiggsZZ4LM200
    

    Run IgProf tests on the multiple candles above.
• --igprof (or -i)

  This option sets the number of events for the IgProf tests. For example:

  • --igprof 5
    

    Run 5 events for all IgProf tests configured by --RunIgProf option above.
• --RunMemcheck

  This option sets the candles on which to run the Valgrind Memcheck tool test. This test used to investigate memory leaks and errors that are reported in a rather long log file. Besides producing the logfile, the performance suite also uses a script that parses the output to get the memory leaks displayed nicely (using Giovanni Petrucciani's script). This test is expensive in terms of time, so normally it is not run on a large number of events. It is particularly useful to investigate crashes due to illegal memory operations. The number of events is set by the option -m or --memcheck below. Example of use of this option:

  • --RunMemcheck SingleElectronE1000
    

    Run Valgrind Memcheck tests on SingleElectronE1000 candle only.

  • --RunMemcheck SinglePiMinusE1000,QCD_80_120
    

    Run Valgrind Memcheck tests on the multiple candles above.
• --memcheck (or -m)

  This option sets the number of events for the Valgrind Memcheck tests. For example:

  • --memcheck 5
    

    Run 5 events for all Valgrind Memcheck tests configured by --RunMemcheck option above.
• --RunCallgrind

  This option sets the candles on which to run the Valgrind Callgrind FCE tool test. This test used to investigate performance issues by doing a full simulation of the cache. Besides producing a very long log file, the performance suite also reports the results with PerfReport to allow detailed navigation of the results. This test is the most expensive in terms of time, so normally it is run on a small number of events (often one only!). It is particularly useful to investigate crashes due to illegal memory operations. The number of events is set by the option -m or --memcheck below. Example of use of this option:

  • --RunCallgrind SingleMuMinusPt10
    

    Run Valgrind Callgrind FCE tests on SingleMuMinusPt10 candle only.

  • --RunCallgrind SingleElectronE1000,CMS.MinBias
    

    Run Valgrind Callgrind FCE tests on the multiple candles above.
• --callgrind (or -c)

  This option sets the number of events for the Valgrind Callgrind FCE tests. For example:

  • --callgrind 1
    

    Run 1 event for all Valgrind Callgrind FCE tests configured by --RunCallgrind option above.
• --RunDigiPileUp

  This option is used to specify the candle(s) on which to run DIGI PILEUP, repeating all tests set to run on that candle (with the options RunTimeSize, RunIgProf, RunMemcheck, RunCallgrind above). This option assumes that some of the tests above are going to be run, if that's not the case, the RunTimeSizePU, RunIgProfPU, RunMemcheckPU, RunCallgrindPU options below should be used instead (it is basically a shortcut for the RelVal use case). For this option to work the --PUInputFile option below needs to be specified so that the suite knows where to get the events for pileup. Example of use:

  • --RunDigiPileUp TTbar
    

    This would re-run with DIGI PILE UP whichever tests (TimeSize, IgProf, Memcheck, Callgrind) that are set to run on TTbar candle.

  • --RunDigiPileUp TTbar,QCD_80_120
    It is also possible to specify multiple candles on which to re-run all tests with PILE UP.
    

• --PUInputFile

  This option sets the input file for the PILE UP events to be used. The option is necessary in case PILE UP is to be run. If the option is omitted, the suite will look in the working directory for a file named INPUT_PILEUP_EVENTS.root, if it does not find it it will quit with a request for a pile up input file to be specified. The option can take as input a plain LFN from DBS, a full path filename, or a simple filename in case the file is in the current working directory. Usually it is a good idea to take as input file a root file from the RelValMinBias sample for the release one is working on. Example of use of this option:

  • --PUInputFile "store/relval/CMSSW_3_0_0_pre10/RelValMinBias/GEN-SIM-DIGI-RAW-HLTDEBUG/IDEAL_30X_v1/0000/209AB4E9-FEFF-DD11-A400-000423D98E54.root"
    

    In this case the suite will rfcp the file above from CASTOR to the local working directory as INPUT_PILEUP_EVENTS.root. This file will be used to get events for PILE UP.

  • --PUInputFile "/build/yourusername/CMSWW_3_0_0_pre10/work/yourPUInputFile.root"
    

    The file /build/yourusername/CMSWW_3_0_0_pre10/work/yourPUInputFile.root will be used as PILE UP input.
• --RunTimeSizePU

  This option is equivalent to the RunTimeSize one above, with the difference that the TimeSize tests will be run with PILE UP events. With respect to --RunDigiPileUP it gives the possibility to run only TimeSize tests on the candle specified, avoiding for example running unwanted Callgrind FCE tests with PILE UP that could be set to run on the same candle. It also allows the user to run only DIGI PILE UP tests without running any other test. This option requires the --PUInputFile option to be also set, so that the file with PILE UP events is known to the performance suite. Example of use of this option:

  • --RunTimeSizePU TTbar
    

    This will run TimeSize tests on TTbar candle with PILE UP events (taken from the file specified by --PUInputFile option). The number of events will be the one specified by --timesize (or -t) option. Like for the other options above, multiple candles can be input as comma separated lists (e.g. TTbar,CMS.MinBias,QCD_80_120).
• --RunIgProfPU

  This option is equivalent to the RunIgProf one above, with the difference that the IgProf tests will be run with PILE UP events. With respect to --RunDigiPileUP it gives the possibility to run only IgProf tests on the candle specified, avoiding running any other tests that could be set to run on the same candle without PILE UP. It also allows the user to run only DIGI PILE UP tests without running any other test on the same candle. This option requires the --PUInputFile option to be also set, so that the file with PILE UP events is known to the performance suite. Example of use of this option:

  • --RunIgProfPU QCD_80_120
    

    This will run IgProf tests on TTbar candle with PILE UP events (taken from the file specified by --PUInputFile option). The number of events will be the one specified by --igprof (or -i) option. Like for the other options above, multiple candles can be input as comma separated lists (e.g. TTbar,CMS.MinBias,QCD_80_120).
• --RunMemcheckPU

  This option is equivalent to the RunMemcheck one above, with the difference that the Valgrind Memcheck test will be run with PILE UP events. With respect to --RunDigiPileUP it gives the possibility to run only the Valgrind Memcheck test on the candle specified, avoiding running any other tests that could be set to run on the same candle without PILE UP. It also allows the user to run only DIGI PILE UP tests without running any other test on the same candle. This option requires the --PUInputFile option to be also set, so that the file with PILE UP events is known to the performance suite. Example of use of this option:

  • --RunMemcheckPU QCD_80_120
    

    This will run the Valgrind Memcheck test on TTbar candle with PILE UP events (taken from the file specified by --PUInputFile option). The number of events will be the one specified by --memcheck (or -m) option. Like for the other options above, multiple candles can be input as comma separated lists (e.g. TTbar,CMS.MinBias,QCD_80_120).
• --RunCallgrindPU

  This option is equivalent to the RunCallgrind one above, with the difference that the Valgrind Callgrind FCE test will be run with PILE UP events. With respect to --RunDigiPileUP it gives the possibility to run only the Valgrind Callgrind FCE test on the candle specified, avoiding running any other tests that could be set to run on the same candle without PILE UP. It also allows the user to run only DIGI PILE UP tests without running any other test on the same candle. This option requires the --PUInputFile option to be also set, so that the file with PILE UP events is known to the performance suite. Example of use of this option:

  • --RunCallgrindPU QCD_80_120
    

    This will run the Valgrind Callgrind FCE test on TTbar candle with PILE UP events (taken from the file specified by --PUInputFile option). The number of events will be the one specified by --callgrind (or -c) option. Like for the other options above, multiple candles can be input as comma separated lists (e.g. TTbar,CMS.MinBias,QCD_80_120).
• --cmsdriver

  This option is NECESSARY to be able to run the performance suite. It sets whatever extra options are necessary for cmsDriver.py, that is the underlying script used by the suite to run cmsRun jobs. At the moment of writing for example the cmsDriver.py --conditions and --eventcontent options have to be specified in order to run, and the user can do so by using the --cmsdriver option. In general to know what to use for this options a good place to look into for a given release is /Configuration/PyReleaseValidation/data/cmsDriver_highstats_hlt.txt. For example, for CMSSW_2_2_1 and Step2:

  • --cmsdriver "--conditions FrontierConditions_CMS.GlobalTag,IDEAL_V9::All --eventcontent RECOSIM"
    

    This will make sure that every cmsDriver.py command run by the performance suite will use these cmsDriver.py options.
• --cores

  This options sets the number of cores of the machines. It is by default read from the /proc/cpuinfo file of the machine, but can be set by the user too. Associated with the --cpu option below regulates the running of cmsScimark on spare cores. E.g. --cores 4, would set the number of cores to 4.

• --cpu

  This option sets the cpu core on which the performance suite is to be run. The default is set to 1, --cpu 1, that means the suite will run the measurements on cpu1.

• --cmsScimark

  This option sets the number of cmsScimark benchmarks to run before and after the performane measurement is carried out (on cpu1 by default).

• --cmsScimarkLarge

  This option sets the number of cmsScimarkLarge benchmarks to run before and after the performance measurement is carried out (on cpu1 by default).

• --archive or -a

  This option sets the CASTOR directory where to archive the tarball of the performance suite results that is automatically created. By default it is set to /castor/cern.ch/cms/store/relval/performance/ so that it automatically archives the tarball for RelVal. For a regular user that did not set the -a option the rfcp to the RelVal area will fail, so the tarball will be left in the working directory. When the archiving is successful the local copy of the tarball is automatically removed. Example:

  • --archive /castor/cern.ch/user/y/yourusername/yourperformancedirectory
    

    The suite upon termination, will tarball all results and rfcp them to /castor/cern.ch/user/y/yourusername/yourperformancedirectory, it will then delete the local copy of the tarball if the rfcp was successful.
• --prevrel or -r

  This option is used to specify the directory where the output of a previously run performance suite is stored, against which the user wants to run a regression analysis. The regression itself can be done by hand with a separate script (cmsRelRegress.py) once the performance suite is done running (this is preferable in official running of the suite for RelVal, to avoid storing in CASTOR regression information that could pollute future regression analysis of the same release). Furthermore, in case the user is only interested in the regression analysis of a particular test, this can be done with a dedicated script (cmsPerfRegress.py). Example of the use of -r:

  • -r /build/relval/CMSSW_2_2_1/work
    

    Once the performance suite is done running, it will run a regression analysis of all test results that are present in both directories.

Available candles

The candles are maintained in the Validation/Performance/scripts/cmsPerfCommons.py and any standard Relval candle can be added with little effort. At the time of writing the available candles are:

• MinBias
• HiggsZZ4LM200
• SingleElectronE1000
• SingleMuMinusPt10
• SinglePiMinusE1000
• TTbar
• QCD_80_120
• QCD_3000_3500

As it is reported in the Release Validation Operator instructions for the execution ofe the performance suite, at the moment MinBias and TTbar have been chosen for the standard RelVal running of the performance suite at each (pre-)release.

Available Profilers

EdmSize, TimeReport, Timing, SimpleMemoryCheck.

IgProfPerf, IgProfMem.

Valgrind callgrind, Valgrind memcheck, [Valgrind massif, currently not used]

Details of cmsPerfPublish.py

All the information produced by the performance suite would be tedious to analyse by eye so a publishing script, cmsPerfPublish.py, is used to generate nice HTML pages. It can be run in two ways: to get the results published in a local directory (local or AFS) or on a remote directory (i.e. on a directory on a remote host). To publish to a local directory (you can create one, say mkdir Results):

cmsPerfPublish.py -v Results

The script will gather the profiling information from the current directory and it will produce the profiling information web pages in $CMSSW_BASE/work/Results/<RELEASE_NAME>/index.html

Then you can conveniently browse the results with your favorite local browser. The -v (verbose) option is recommended to help debugging potential publication problems.

A remote directory on a remote server can be specified using:

cmsSimPerfPublish.py -v somehost:/some/remote/dir

Rsync and ssh may need to be properly configured for this feature to operate correctly.

Another input directory can be specified as an argument like so:

cmsPerfPublish.py -v --input=/some/other/dir

In the case a regression analysis with respect to a previous release has been run in the current directory cmsPerfPublish.py will automatically detect the information contained within the cmsPerfSuite directory and produce a series of tables and graphs that are useful for regression performance analysis.

Details of cmsRelvalReportInput.py

cmsRelvalReportInput.py creates a configuration file for cmsRelvalreport.py that describes the order in which cmsDriver.py should be run with which candles, steps and profiling so that time spent running cmsDriver.py is minimised. Without this correct ordering we would have to re-run parts of the profiling steps. The first argument is the number of events to be run, the second is the candle type (in the format above) and the third is the profiling type(s). For example, to perform Timing Report, Time Report and SimpleMemoryCheck profiling for all candles and 10 events, we would run:

cmsRelvalreportInput.py 10 AllCandles 012

The profile codes (multiple codes can be used) follow: Code Profile Type 0 TimingReport 1 TimeReport 2 SimpleMemoryCheck 3 EdmSize 4 IgProfPerf 5 IgProfMemTotal 6 IgProfMemLive 7 IgProfAnalyse 8 ValgrindFCE 9 ValgrindMemCheck

It is possible to use custom steps (--usersteps) and custom cmsDriver commands (--cmsdriver).

Details of cmsRelvalReport.py

RelValreport is a tool for automation of benchmarking and report generation.

For example, to execute the candles profiling detailed in the file candles_150.txt, and report its results in directory 150_report:

cmsRelvalreport.py -i candles_150.txt -R -P -o 150_report

Details of cmsRelRegress.py

cmsRelRegress.py compares two cmsPerfSuite.py directories to produce regression information about the difference in profiling information between releases without having to run cmsPerfSuite.py again. One can create regression information in the latest release cmsPerfSuite directory (/new/rel/dir/cms3000) when compared with the previous release cmsPerfSuite directory (/old/rel/dir/cms2.0_beta) by using:

cmsRelRegress.py /old/rel/dir/cms2.0_beta /new/rel/dir/cms3000

In this case cms2.0_beta is the previous release and cms3000 is the latest release. The regression information will be placed in the latest release (in this case it will be in cms3000). This information will be detected by cmsPerfPublish.py is run, then a series of additional tables and graphs will be created in addition to the regular performance profiling ones.

Details of cmsPerfRegress.py

cmsPerfRegress.py compares two individual logfiles that are produced during the cmsPerfSuite.py profiling. This is useful in case you wish to perform regression analysis with only two logfiles. Pass the two logfiles as arguments the first one being the previous release logfile and the second being the latest release and specify the type of logfile. Types can be "timing", "simplememory", "edmsize", "igprof" or "callgrind", callgrind logfiles must be gzip'd. For example

cmsPerfRegress.py -t timing /old/logfile /new/logfile

Details of cmsSimplememchecker_parser.py

This is a utility script (available command line in the release) to parse and plot the VSIZE and RSS information from the SimpleMemory Service from cmsRun logfiles. Here's its usage:

cmsSimplememchecker_parser.py -i YourLogFile.log -o YourDirectory

The script will parse the information from the lofgile YourFile.log and it will create, in the directory YourDirectory a file YourLogFile_SimpleMemReport.html that contains the VSIZE, RSS and deltaVSIZE, deltaRSS plots per event. The plots themselves are also dumped in a root file in the same directory and as gif files.

Details of cmsTiming_parser.py

This is a utility script (available command line in the release) to parse and plot the timing information from the Timing Service from cmsRun logfiles. Here's its usage:

cmsTiming_parser.py -i YourLogFile.log -o YourDirectory

The script will parse the information from the lofgile YourFile.log and it will create, in the directory YourDirectory a file YourLogFile_Timing.html that contains the graph of time per event and the histogram of the times. The plots themselves are also dumped in a root file in the same directory and as gif files.

Details of cmsTimeReport.pl

This is a utility script (available command line in the release) to parse the TimeReport summary at the end of cmsRun job logfiles that had the "wantSummary" option set to true and it saves it in html format. This is meant to help in publishing on the web such information. Here's its usage:

cmsTimeReport.pl YourLogFile.log YourDirectory

The script will parse the TimeReport summary present in YourLogFile.log and it will create a file TimeReport.html in the directory YourDirectory.

Details of cmsIgProf_Analysis.py

This is a utility script (available command line in the release) to execute some IgProf analysis (on the beginJob and beginEvent segments and on the MEM_LIVE and MEM_TOTAL counters). The script produces, in the case of MEM_LIVE and MEM_TOTAL, VERY long text html output, such that could crash or hang some web browsers (you are warned). Here's the usage:

=cmsIgProf_Analysis.py -i YourIgProfMemReport -o YourDirectory

The script will extract the MEM_LIVE and MEM_TOTAL information from the IgProf report and it will perform 4 standard analyses, creating the corresponding 4 html files (doBeginJob_output.html, doEvent_output.html, mem_live.html, mem_total.html) in the directory YourDirectory.

Results (Publication page)

The official results for the RelVal execution of the performance suite are published here. As you can see there are several directories, Step1, Step2, GEN-SIM,DIGI, L1-RECO for the different steps run by RelVal at each (pre-)release. There is also a directory Regression, that contains the same subdirectory structure, designated to publish the results of regression analysis between releases already available.

Of course individual users can publish their results in other servers/locations using the publishing script above.

CVS Repositories Involved

The performance suite code has been consolidated into Validation/Performance package. The suite also uses the valgrindMemcheckParser.pl script that lives in Utilities/ReleaseScripts/scripts. All performance suite scripts are available command line in the release. At the moment the suite is using PerfReport from a private AFS directory, but the tool should be soon added to the CMSSW externals, as matplotlib, that is necessary to have the pie-charts produced by PerfReport.

GabrieleBenelli - 03 Mar 2009 - page author

Responsible: GabrieleBenelli