Using PerfReport

THIS DOCUMENTATION DOES NOT APPLY TO CURRENT CMSSW RELEASES

Goal of the page

This page is intended to explain how to use PerfReport to generate static HTML reports from performance profiles.

Contacts

If you experience any kind of problems with PerfReport, please contact me immediately. Write to: robin.alexander.moser@cernNOSPAMPLEASE.ch

Introduction

PerfReport generates static HTML reports from performance profiles in various formats.

The most recent released stable version of PerfReport is 2.0.0.
It creates HTML reports from profile data output by Callgrind FCE (Full Callstacks Extension) and imports IgProf profiles natively. It can also produce reports in the same look and feel from the output generated by the edmEventSize tool. PerfReport 2.0.0 is available as an RPM in the CMS repository. Check instructions on how to configure and run below.

PerfReport 1.0.0 is now deprecated and available only in the form of a source tarball. You can use PerfReport 1.0.0 if you need to create HTML reports from standard Callgrind files. For all other purposes, please use PerfReport 2.0.0. If you want to profile with Callgrind, please consider using the patched version (FCE). Or else, consider using IgProf instead.

Callgrind FCE is currently being finalized and is foreseen to be published in the form of an RPM in the near future. For the time being, Callgrind FCE has to be built from its sources (see download section).

Please find separate manuals for 2.0.0 below and 1.0.0 here.

PerfReport 2

Getting started quickly

Your first steps with PerfReport 2 can be done as follows:

Installing and configuring

Initialize PerfReport in your environment by doing (depending on shell)

source $VO_CMS_SW_DIR/slc4_ia32_gcc345/external/perfreport/2.0.0/profile.d/init.sh

source $VO_CMS_SW_DIR/slc4_ia32_gcc345/external/perfreport/2.0.0/profile.d/init.csh

On LXPlus (64bit SLC4), a working command to get perfreport in your path is

source /afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/perfreport/2.0.0/etc/profile.d/init.sh

A simple report from IgProf data

IgProf normally outputs a gzipped ASCII profile. Make sure you

gunzip

perfreport -fi -i (your igprof file) -o (output directory)

A simple report from Callgrind FCE

Callgrind FCE is currently being finalized and will be released as an updated Valgrind package in the near future. As long as this is not done, you can use the current test version of Callgrind FCE located in my public AFS directory. Since Callgrind FCE is loaded as a Valgrind module, it can run with your local Valgrind installation which does not need to be patched. Simply type

setenv VALGRIND_LIB /afs/cern.ch/user/m/moserro/public/vgfcelib

export VALGRIND_LIB=/afs/cern.ch/user/m/moserro/public/vgfcelib

valgrind --tool=callgrind --fce=(outputfile).fce cmsRun (...your configuration...)

valgrind --tool=callgrind --fce=(outputfile).fce --instr-atstart=no cmsRun (...your configuration...)

perfreport -ff -i (your fce output file) -o (output directory)

A simple report from edmEventSize

After obtaining an ASCII profile from edmEventSize, type

perfreport -fe -i (your edmeventsize file) -o (output directory)

A simple regression report for either IgProf or Callgrind FCE

To create a regression difference analysis between an older and a newer profile, type

perfreport -fi -i (newer profile) -r (older profile) -o (output directory)

perfreport -ff -i (newer profile) -r (older profile) -o (output directory)

More sophisticated analyses

PerfReport offers a big variety of configuration options and filters you can use to select from the profile the data you are interested in, remove unwanted parts, aggregate costs in various ways, collapse set of functions, inline costs into callers, specify renamings in the case of regression analysis, etc. To make use of these features, please take the time to read the complete manual below.

The command line

To invoke PerfReport, please type

  perfreport -f(f|i|e) -i (profile)

The switches -f and -i are mandatory. If omitted, the program halts:

  -f(f|i|e)

is used to specify the format of the input profiles:
• f: FCE
• i: IgProf
• e: edmEventSize

  -i (inputfile)

is used to specify the input file. If regression analysis is requested, this is the more recent profile.

In addition, there is a number of optional switches:

  -c (regconfig)

specify the filename of the regression config XML. If omitted, the standard config is used. If -r is omitted, -c is ignored.

  -d (reportdesc)

specify the filename of the report description XML. If omitted, the standard description is used.

  -o (outdir)

specify the output directory for HTML/PNG production. If omitted, the current directory is used.

  -r (profile)

specify the filename of the second profile. This is the older preofile for regression analysis. If omitted, no regression analysis is performed.

  -t (tempdir)

specify a temporary directory for intermediate analysis data. If omitted, '/tmp/' is used.

  -m 'key,value'

add the given metadata pair. This information will be displayed on all main pages. Repeat the switch to add multiple pairs.

  -y (counter)

specify the counter to be read from IgProf profiles. Supported counters are:
• PERF_TICKS
• MEM_TOTAL
• MEM_LIVE
• MEM_LIVE_PEAK and
• MEM_MAX.
The switch is ignored if a format other than -fi is used.

See https://twiki.cern.ch/twiki/bin/view/CMS/IgProfAnalysis for explanations about the various IgProf counters.

  -p

disable graphics production with Python/MatplotLib and store Python scripts (.py) in the output directory instead for later conversion to graphics.

Authoring PerfReport 2 XML report descriptions

1. Obtaining copies of the standard descriptions

The initialization scripts for PerfReport automatically set an environment variable called PERFREPORT_PATH which will point to the directory where the standard XML report descriptions that PerfReport uses are located. If you type

ls $PERFREPORT_PATH/*.xml

• $PERFREPORT_PATH/pr2allstd.xml. This is the standard report description used to produce reports for IgProf and Callgrind FCE.

  
  

• $PERFREPORT_PATH/pr2edmeventsize.xml. This is the standard report description used to produce reports for edmEventSize.

  
  

• $PERFREPORT_PATH/pr2cmsswfltmsg.xml. This is an alternative description suitable for producing IgProf and Callgrind FCE reports. It has an additional filter activated which prunes all the branches below any function called edm::Message*. You might want to use this description to filter out unwanted contributions arising from message logging as present in the nightly builds of CMSSW. To use this report description, simply type

  perfreport ... -d $PERFREPORT_PATH/pr2cmsswfltmsg.xml ...

  and combine with any other options you like.

  
  

• $PERFREPORT_PATH/pr2stdregconf.xml. This is the standard regression configuration file which PerfReport uses whenever you omit the config switch -c. See the section below about regression config files.

If you want to write your own filter descriptions to make a different kind of analysis, the easiest way is to copy the corresponding standard XML file (in most cases pr2allstd.xml) to some directory of yours and to modify it there accordingly, following the instructions about XML descriptions you can find below. Then, invoke PerfReport adding the -d switch and the file name of your modified XML.

You can even edit the XML files in the $PERFREPORT_PATH to change the standard behaviour of PerfReport without having to specify the -d switch. However, you are discouraged to do that as you might get confused by your own changes at a later point in time and cleaning them up will require reinstalling PerfReport.

2. Top level schema and the <Report> element

An XML report description has the following layout.

<Report>

 ...(options) ...

 <ReportSection>
  ...
 </ReportSection>

 <ReportSection>
  ...
 </ReportSection>

 ...
</Report>

Every report section defines a specific view of the data represented in the report and has its own entry page together with, possibly, subpages linked from there.

To specify global report options, you can use the following tags:

<GlobalFilters>
  <Filter>
    ...
  </Filter>
  <Filter>
    ...
  </Filter>
  ...
</GlobalFilters>

3. The <ReportSection> element

The <ReportSection> element represents a main page within the report, consisting of a table of (possibly aggregated) report items and, optionally, graphical representations of that data. Subpages which give detailed information about the items listed may be attached to such a page.

All child elements of a <ReportSection> are optional and can be omitted. The simplest meaningful XML report description therefore is

<Report>
 <ReportSection>
 </ReportSection>
</Report>

It will produce a standard report with default options which shows the top 20 functions overall in a table and a pie chart.

Optionally however, you can add any of the following subtags:

Whenever the utility builds a table within this report section, it will display a set of columns, according to which of the following options you activate:

You may also configure the way in which function names are displayed. To this end, add a <NamingDiscipline> element to the <ReportSection> and put one of the following values inside:

If you want to control the ordering used to sort the items in the list, add a <SortBy> element to the <ReportSection> and put one of the following values inside:

Please be aware that the sorting takes place before truncation of the list. So if you set the displayitems property to 20 and the sorting discipline to functionname, you will end up with a list containing the first 20 functions w.r.t. alphabetical ordering.

If you are creating a report with many subsections or with subsections that are logically related, it usually makes sense to add links to the pages that connect them to one another. To this end, the tool will produce a header on every page where all those links are collected and hierarchically grouped. For this feature to work you will have to provide information on where to position the current reportsection in the hierarchy. Use the following tags to do this:

4. Describing a data filter using a <Filter> element

Filters in PR2 allow for more generic operations on the profile data than just separating wanted from unwanted elements (as in PR1). There exist two types of filters, the ones which need an <ItemSet> parameter and the ones which don't. In the first case, the syntax is

<Filter>
  <Type>...</Type>
  <ItemSet>
    ...
  </ItemSet>

</Filter>

and in the other case simply

<Filter>
  <Type>...</Type>
</Filter>

Within the <Type> tag, specify the kind of filter you want to apply. The following exist:

5. Describing a set of functions (/report items) using an <ItemSet> element

The <ItemSet> tag is built in the follwing manner:

<ItemSet>
 <Type> ... </Type>
 ...
</ItemSet>

Depending on the type of set you select inside the <Type> subtag, there are different options you can select below.

The following item set catgeories are available:

<ItemSet>
 <Type>And</type>
 <LeftOperand>
  <ItemSet>
   ...
  </ItemSet>
 </LeftOperand>
 <RightOperand>
  <ItemSet>
   ...
  </ItemSet>
 </RightOperand>
</ItemSet>

<ItemSet>
 <Type>Not</Type>
 <Operand>
  <ItemSet>
   ...
  </ItemSet>
  </Operand>
</ItemSet>

For the first two ItemSet types, the following proterties can be chosen inside a <Property>-tag:

Authoring PerfReport 2 XML regression configuration files

PerfReport 2 XML regression configuration files have two ingredients: the two quantitative parameters used for regression and the renamings that should be applied to function names when matching symbols of the two profiles. The global structure of a regression config file is a follows:

<RegressionConfig>
 <ThresholdDecrease>...</ThresholdDecrease>
 <ThresholdIncrease>...</ThresholdIncrease>
 <Renamings>
    ...
 </Renamings>
</RegressionConfig>

Setting quantitative regression parameters

When PerfReport creates a regression report, comparing an older with a newer profile, it can report for every report item

• the absolute amount of change in cost in every cost column (inclusive of relative columns which will yield the amount of change in relation to the other items)
• the relative amount of change in cost in every cost column
• the movement the item has done within the complete ranking of cost-intensive functions

Describing renamings

Before PerfReport can calculate the changes in cost and then decide (according to the above parameters) whether it should display regression information and what information to display, PerfReport has to match each symbol appearing in the older profile with the corresponding symbol appearing in the newer profile. This is most trivial if the symbol and library names are the same; PerfReport will use a (symbol name, library name) key to identify each function. Imagine however that from one to the next release of CMSSW it is decided to change the name of some class. In this case, PerfReport does not know the old and the new classes are related and displays only the new one, claiming it has disappeared from the cost list.

To prevent this from happening, you can provide PerfReport with the information that the name has changed and it will consider this when computing the correct matching. The description of such renamings has to be placed in the regression config XML in the <Renamings> section. The <Renamings> section may contain an arbitrary number of <Renaming> subelements. These in turn have the following structure:

<Renaming>
 <Search>...</Search>
 <Replace>...</Replace>
 <Part>...</Part>
 <Lookup>...</Lookup>
</Renaming>

The various elements have the following meaning and usage:

The following graphic illustrates the process of matching symbol names in PerfReport. There are prospective changes from A to Ax and from func2 to func_2, as well as retrospective changes from Bx to B and from func_1 to func1.

PerfReport 1

WARNING

PerfReport version 1 is deprecated. Use it exclusively if you are forced to analyse profiles generated with the standard unpatched version of Callgrind. In any other case please use PerfReport 2.

Instructions for using PerfReport 1

Instructions for using PerfReport version 1

Review status

Responsible: VincenzoInnocente
Last reviewed by: RobinMoser - 22 Mar 2007