EMI Metrics

Introduction

This twiki page explains EMI metrics in the following areas:

1. Process related metrics: They are related to the way software changes are managed within the project. It uses the information stored in the tracking tools used by the different middlewares.
2. Software related metrics: The static analysis software related requirements to configure the metrics software. This information is required by each product team.
3. GGUS related metrics: Liaising with the GGUS people to obtain 3rd level incident metrics (2 of which relate to the KPIs in DoW)

The metrics formulas are described in the Software Quality Assurance Procedure (SQAP) plan. For a fuller description including questionnaires, all metrics in the SQAP, optional metrics, McCall Quality factors, XSD and XML examples, see the internal reference document.

Process related Metrics

In EMI, a number of different bug trackers exist. Each of them extract the metrics in a different way, but all of them must return the results following the same XML format, so that the information per bug tracking ticket can be interpreted and visualised within the ETICS system.

Weekly generated metric reports (EMT, Detailed Report, EU Report)

Full set of reports (.html)	http://cern.ch/emiqa/reports/latest
Charts for QC and SA1 (.html)	http://cern.ch/emiqa/reports/latest/chart.html
EMT (.pdf)	http://etics-repository.cern.ch/repository/reports/name/emi-qa-reports/-/reports/EticsQAReports/EMT_report.pdf
Detailed Report (.pdf)	http://etics-repository.cern.ch/repository/reports/name/emi-qa-reports/-/reports/EticsQAReports/Detailed_report.pdf
EU Report (.odt)	http://etics-repository.cern.ch/repository/reports/name/emi-qa-reports/-/reports/EticsQAReports/EMI-METRICS-REPORT_EU.odt

How to obtain bug trackers metrics

The flow of information from the bug-tracker to ETICS required XML format is described below:

XML Schema: Different Bug-tracking Information presented Uniformally

Location of Schema's, XML examples and conversion scripts (under development)

A subversion repository is viewable at: https://svnweb.cern.ch/trac/emisa2/browser/metrics/trunk or can be checked out using:

svn co http://svnweb.cern.ch/guest/emisa2/metrics

Definition of Schema's and associated XML files

Schema Name	Description/Function	Required Schema's	XML using the schema	XML Description
BugListing.xsd	Defines an unbounded set of defects/features existing in each middleware bug-tracker.	SQAPDefinitions.xsd	<middleware>-BugListing.xml	Complete bug listing extracted from each middlewares bug-tracker, each defect/feature being defined in one XML element: <RfC>
SQAPDefinitions.xsd	Contains information related to the Change Management Guidelines [2]. This XSD defines the required information for each defect/feature ticket.	None	No XML should use it directly, use BugListing.xsd instead	None
BugMapping.xsd	This is optional! This intermediate schema validates <middleware>-BugMapping.xml. The idea being that the conversion script (usually run daily) can remain object oriented and unchanged, whilst the BugMapping.xml changes to reflect changes in the internal workings of the bug-tracker.	None	<middleware>-BugMapping.xml	Useful for internal changes in bug-tracker, and as a documented list of these changes

First Steps to Producing a Uniform Framework

Each middleware should primarily concern themselves in the first instance with how to form the

<middleware>-BugMapping.xml conforming to BugMapping.xsd

and:

<middleware>-BugListing.xml conforming to Buglisting.xsd

Again, the first is totally optional, but both of these XSD's and corresponding XML files (produced per middleware) conform to the request for change guidelines in the software quality assurance plan (SQAP). Each XML file must be returned to the metrics group for validation and syntax checking. Please send to emi-sa23@eu-emiNOSPAMPLEASE.eu when the XML is revised or ready.

Request for Change Conformance

The SQAPDefinitions.xsd is a schema file that conforms to the Change Management guidelines. BugListing.xsd states that the XML to be validated must contain many RfC items and a timestamp. The per middleware, <middleware>-BugListing.xml should therefore contain items for each opened defect tracking ticket of the form:

  <RfC>
    <StatusChangeTimestamps>
      <accepted>dateTime</accepted>                    <!-- possible elements: open, accepted, fixed, tested, nottested, rejected, closed -->
      <fixed>dateTime</fixed>                                  <!-- possible values: values are always in the form of dateTime: yy-mm-ddThh:mm:ss -->
      <tested>dateTime</tested>
    </StatusChangeTimestamps>
    <Status>The current status</Status>                <!-- possible values: open, accepted, fixed, tested, nottested, rejected, closed -->
    <EndedOn>dateTime</EndedOn>                    <!-- this can be UNSET or dateTime -->
    <Description>relates to the first comment, restrict to 65 characters</Description>
    <Title>The main summary of the bug goes here</Title>
    <SubmittedOn>dateTime</SubmittedOn>
    <Component>specified in SQAPDefintions.xsd</Component>
    <DetectionArea>specified in SQAPDefintions.xsd</DetectionArea>    <!-- Possible Values: development, production, testing. Note 'testing' was added June 2011 -->
    <Priority>specified in SQAPDefintions.xsd</Priority>
    <AffectedEMIMajorRelease>Affected EMI Release Number</AffectedEMIMajorRelease>   <!-- Possible values EMI-1, EMI-2, EMI-3, OlderMiddleware -->
    <TargetEMIMajorRelease>Affected EMI Release Number</TargetEMIMajorRelease>   <!-- Possible values: EMI-1, EMI-2, EMI-3, OlderMiddleware -->
    <AffectedPlatforms>Platform Name (and Version)</AffectedPlatforms>    <!-- Possible values: SL 5, SL 6, Debian 6, All Linux, All, No Architecture -->
    <AssociatedTest>Associated Test for a defect or feature</AssociatedTest>    <!-- Possible values: Yes, No, N/A -->
    <GGUSTicket>URL of originating GGUS ticket</GGUSTicket>
    <Severity>specified in SQAPDefintions.xsd</Severity>
    <UID>URL pointing to the ticket in the bug-tracker (was previously an integer, revised June 2011)</UID>
  </RfC>

Example obtained from revised gLite conversion (tested open defect with no GGUS ticket and an associated test):

  <RfC>
    <StatusChangeTimestamps>
      <accepted>2011-06-29T10:13:22</accepted>
      <fixed>2011-06-29T14:50:18</fixed>
      <tested>2011-07-04T17:20:09</tested>
    </StatusChangeTimestamps>
    <TargetEMIMajorRelease>UNSET</TargetEMIMajorRelease>
    <EndedOn>UNSET</EndedOn>
    <Status>tested</Status>
    <GGUSTicket></GGUSTicket>
    <Description>...</Description>
    <Title>There's a memory leak in GRSTx509MakeProxyCert function of gridsite component</Title>
    <SubmittedOn>2011-06-29T09:52:15</SubmittedOn>
    <Component>Gridsite</Component>
    <DetectionArea>development</DetectionArea>
    <Priority>medium</Priority>
    <AffectedPlatforms>SL 5</AffectedPlatforms>
    <AffectedEMIMajorRelease>EMI-1</AffectedEMIMajorRelease>
    <AssociatedTest>Yes</AssociatedTest>
    <Severity>defect</Severity>
    <UID>https://savannah.cern.ch/bugs/?83773</UID>
  </RfC>

Where dateTime="YYYY-MM-DDThh:mm:ss", e.g: "2010-12-09T14:46:50"

Validation of Per Middleware Scripts

The <middleware>-BugMapping.xml should be visualised using main-BugMapping.py, typing the name of the middleware into The <middleware>-BugListing.xml should be validated using xmllint as in the example:

validateAll.sh

Persistent Bug-Tracking Collection

Middleware Name	URL locations
ARC	http://bugzilla.nordugrid.org/ARC-BugListing.xml
gLite	http://emi-rfc.cern.ch/xml/gLite-BugListing.xml
UNICORE	http://emi-rfc.cern.ch/xml/UNICORE-BugListing.xml
dCache	http://xen-ep-emi-tb-se-3.desy.de:2880/pnfs/desy.de/data/testers.eu-emi.eu/dCache-BugListing.xml
MPI	http://devel.ifca.es/mpi-start/report/6?format=rss&emixml=true
StoRM	https://storm.cnaf.infn.it:8443/redmine/buglisting/StoRM-BugListing.xml

Open Issues - UNICORE

Actually about UNICORE SourceForge BugTracking system there are some open issues, in particular with reference to:

1. BugStatus mapping: In UNICORE the status is represented by a 'status' field and a 'resolution' field which should be mapped into emi states. At the moment no agreement has been reached about that, this concerns also priority definitions.

2. Component names: Component names actually are extracted from the 'category' field of the original bugListing.xml of SourceForge. These names are not univoke since what it seems, a correct mapping between SF categories and univoke component names in emi must be adopted.

3. Starting date for metrics publishing: actually no date has been established to start publishing metrics, we have not seen yet any standard presentation format or web application where data will be print.

Issues above must be solved at a first instance before to proceed with further improvements:

Next steps:

3. Solve the StatusMapping xml file problem: status mapping at the moment is also wired in the code, becouse of the double states association between emi and SF is not possible at the moment to configure it.

4. Solve the StatusChanging issue: SF has a change log which renders us difficult to perform bug status changes mapping (also becouse of the mapping between status resolution and emi states), SF logs only for events in a sequential way, setting the corresponding date, but it does not associate elements to a single change. So this information must be derived from the context.

Software related metrics

Work in Progress: Needs more input from SA2.4 to provide product teams with a mechanism to correctly produce metric reports with each ETICS build report.

ETICS plugin information

Middleware Name URL locations ARC http://bugzilla.nordugrid.org/ARC-BugListing.xml gLite http://emi-rfc.cern.ch/xml/gLite-BugListing.xml UNICORE http://emi-rfc.cern.ch/xml/UNICORE-BugListing.xml dCache http://xen-ep-emi-tb-se-3.desy.de:2880/pnfs/desy.de/data/testers.eu-emi.eu/dCache-BugListing.xml

Information about ETICS plugins: https://web.infn.it/etics-support/index.php/component/content/category/22-the-plugin-framework

The structure for software related metrics currently used in the ETICS system is: MetricsSchema. The template is available on that page and also here (ETICS CVS link): metrics.xsd

Setting up the FindBugs Metric for Java Code

• The ETICS guide to using FindBugs must be constrained to a particular set of options to ensure that the Metrics are calculated identically across all Product Teams.
• If you have Java code in your software stack this will show up in the SLOCCOUNT (software lines of code count) and therefore should also show up as a FindBugs metrics result.
• FindBugs will look for Java class files in the following directories by default:

     ${src.location}/classes
     ${src.location}/build/classes
     ${src.location}/bin/classes
     ${src.location}/target/classes 
     ${src.location}/build
     

• If your code is not in one of these directories FindBugs will fail to run if unless you have specified findbugs.class.location or java.class.location (applies to all Java metric plugins). You have to set the property as part of your ETICS checkout and build as follows:

     --property findbugs.class.location='${src.location}/some/subdirectory/containing/built/classes'
     --property java.class.location='${src.location}/some/subdirectory/containing/built/classes'
     

• It is also possible to specify an jar file instead or in addition to the class files. This can be specified with findbugs.jar.location or java.jar.location (applies to all Java metric plugins). You have to set the property as part of your ETICS checkout and build as follows:

     --property findbugs.class.location='${src.location}/some/subdirectory/containing/jar/file'
     --property java.class.location='${src.location}/some/subdirectory/containing/jar/file'
     

  • Note that any externals included in the jar file will be included in the FindBugs count.

• Findbugs is generated during the 'pretest' ant task. If your code is compiled after this task, FindBugs can not find any class files when it is running.

Setting up the PMD Metric for Java code

• Using a non-standard location for the source code results in PMD not finding the required java files. To remedy this it is necessary to define a property java.src.location. You have to set the property as part of your ETICS checkout and build as follows:

     --property java.src.location='${src.location}/some/directory/containing/the/java/files'
    or
     --property pmd.src.location='${src.location}/some/directory/containing/the/java/files'
     

• If the java.src.location, pmd.src.location or src.location=src does exist, there is a possibility that PMD will run successfully but return no result when the subdirectory does not contain any Java files.

Setting up the JCheckStyle Metric for Java code

• Using a non-standard location for the source code results in checkstyle not finding the required java files. To remedy this it is necessary to define a property java.src.location. You have to set the property as part of your ETICS checkout and build as follows:

     --property java.src.location='${src.location}/some/directory/containing/the/java/files'
    or
     --property checkstyle.src.location='${src.location}/some/directory/containing/the/java/files'
     

• If the java.src.location, checkstyle.src.location or src.location=src does exist, there is a possibility that JCheckStyle will run successfully but return no result when the subdirectory does not contain any Java files.

Using ETICS metrics

Quick Guide to Obtaining Software Metrics

To enable all ETICS metric plugins, you should append the property: --autoprofile to the checkout and build of any project, subsystem or component build.

Specific Metric Result Generation

To enable ETICS plugins, you have to append to the requirements this: -p default.profile=pluginName

Examples:

• -p default.profile=ipv6,wsi
• -p default.profile=ckjm,aqcm,pmd,checkstyle

The results of the metrics generation can be found inside the report, in the main links column in the left of the page, as a new entry called "Metrics":

GGUS related Metrics

Still under discussion with the GGUS team. 2 of 4 metrics are currently calculable already in GGUS.

Contacts

General Metrics Related Enquiries

Contact info: emi-sa23@eu-emiNOSPAMPLEASE.eu

Involvement	Name (Institution)	Contact Information	Reason for Involvement
Lead:	Eamonn Kenny (TCD)	Contact info: EamonnKenny	SA2.3 Metrics Definitions
Participants:	Andres Abad Rodriguez (CERN)	Contact info: AndresAbadRodriguez	SA2.4 Metrics Schema's
	Duarte De Meneses (CERN)	Contact Info: dmeneses@cernNOSPAMPLEASE.ch	SA2.4 Metrics Visualisation
	Christian Bernardt (DESY)	Contact Info: christian.bernardt@desyNOSPAMPLEASE.de	dCache bug-tracking contact point
	Anders Waananen (NBI)	Contact Info: waananen@nbiNOSPAMPLEASE.dk	ARC bug-tracking contact point
	Michele Carpene (CINECA)	Contact Info: m.carpen@cinecaNOSPAMPLEASE.it	UNICORE bug-tracking point
	Björn Hagemeier	Contact Info: b.hagemeier@fz-juelichNOSPAMPLEASE.de	UNICORE bug-tracking point
	Andrea Ceccanti (INFN)	Contact Info: andrea.ceccanti@cnafNOSPAMPLEASE.infn.it	SA1 Quality Control contact point
	Giuseppe Fiameni (CINECA)	Contact Info: g.fiameni@cinecaNOSPAMPLEASE.it	SA1 Quality Control contact point

References

1. The internal Metrics Document, https://svnweb.cern.ch/trac/emisa2/browser/metrics/trunk/doc/Metrics-Report-v0.38.docx, https://svnweb.cern.ch/trac/emisa2/browser/metrics/trunk/doc/Internal-Metrics-Report.pdf
2. The Change Management Guidelines, https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2ChangePolicy