EMI QA Verification Dashboard

Introduction

EMI releases need to be verified from the quality assurance perspective according to the EMI Production Release Criteria. Before a new product version can be released to production, the different SA2 policies are checked by SA2, in particular the Quality Control (QC) task. Some of the checks are done automatically, and some others, like the documentation review, are done manually. A dashboard displaying all this information for each EMI release assits QC to carry out this task.

The EMI QA Verification Dashboard automates the release verification activity extracting information from ETICS, Savannah and the certification reports. It integrates documentation review results as well, making it easier to analyse review information and produce reports. It also allows other people, like the release manager, to easily access and process this information.

Overview of the architecture

The Dashboard will be accessible through a web page created using the Django web framework. A MySQL database will contain all the necessary metadata associated to a given EMI release, product, policy, etc.

Data will be extracted from different sources:

• Savannah release tasks:
  • List of RfCs and their corresponding status (this will imply interacting with other trackers or with the SA2.3 XML files to generate metrics, which is something to be investigated).
  • List of binaries/tarballs.
  • Test plan link.
  • Certification report
    • Documentation availability (this information will be confirmed with the results of the documentation review, performed manually)
      • Functional description
      • User Guide
      • Service Reference Card
      • ...
    • In-house testing
    • License information
  • Test report
    • Unit tests
    • Deployment tests
    • Functionality tests
    • ...
• ETICS:
  • Subsystem build information.
  • Binary/Source packages and tarballs available in the repository.

In the end, we should be able to see the following type of information per EMI release, supported platform and product:

We should be able to export this information in suitable formats to be able to generate statistics and reports. We should also be able to generate the verification reports that are attached to the release tasks.

Dashboard checks

The different checks performed by the Dashboard are the following:

Certification Report Template v3.0/3.1

Check Number	QA Artifact	Information extracted from	Information compared to	SA2 Check
General checks
1	RfCs are listed	Answer for 1.2 in Certification Report	List of RfCs in Savannah release task	IF ANSWER_1.2=YES AND List of RfCs NOT NULL in the Release Task THEN SA2_CHECK=PASS ELSE SA2_CHECK=FAILED
2	All RfCs are "Tested/Not Tested"	Answer for 1.3 in Certification Report	XML files	Manual check for the time being
3	Relevant RfCs have an associated regression test	-	-	Manual check requested by QC
4	Licence information	Answer for 8 in Certification Report	License in Savannah release task	IF ANSWER_8=YES AND License NOT NULL then SA2_CHECK=OK ELSE SA2_CHECK=FAIL
5	VCS Tag available	Answer for 2 in Certification Report	ETICS	Manual check for the time being
6	Certification report is compliant with SA2	-	-	Manual check requested by QC
7	Certification report is complete in all sections	-	-	Manual check requested by QC
ETICS checks
8	Subsystem configuration is locked	Answer for 3.1 + ETICS Subsystem Configuration Name in Certification Report	ETICS	Manual check for the time being
9	Subsystem configuration builds without errors	Answer for 3.2 + ETICS Subsystem Configuration Name in Certification Report	ETICS	Manual check for the time being
10	ETICS configuration correctly listed	-	-	Manual check requested by QC
11	Binary rpms available in the repository	Answer for 4.1 in Certification Report + Package List in Savannah release task	ETICS	Manual check for the time being
12	Binary tarballs available in the repository	Answer for 4.2 in Certification Report + Package List in Savannah release task	ETICS	Manual check for the time being
13	Source rpms available in the repository	Answer for 5.1 in Certification Report + Package List in Savannah release task	ETICS	Manual check for the time being
14	Source tarballs available in the repository	Answer for 5.2 in Certification Report + Package List in Savannah release task	ETICS	Manual check for the time being
15	Binary packages specified in the release task	Answer for 6 in Certification Report	Package List in Savannah release task	Manual check for the time being
16	Source packages specified in the release task	Answer for 7 in Certification Report	Package List in Savannah release task	Manual check for the time being
Testing related checks
17	Successful in-house deployment of the component from the EMI repository	Answer for 14 in Certification Report	AND of all test results reported in the Summary of Test report	IF ANSWER_15=YES AND (Clean Installation=PASS or NA)AND(Upgrade Installation=PASS or NA)AND(Unit Tests=PASS)AND(Basic Func Tests=PASS) AND (Regression Tests=PASS or NA) THEN SA2_CHECK=OK ELSE SA2_CHECK=FAIL
18	Test Plan exists and is specified in the release task	Answer for 10 in Certification Report	Test Plan Link in Savannah release task	IF ANSWER_10=YES AND Test Plan Link in the Release Task is a valid link then SA2_CHECK=OK ELSE SA2_CHECK_FAIL
19	Test report is specified in the release task	Answer for 11 in Certification Report	Latest attachment in Savannah release task where the word test + report are present.	IF ANSWER_11 AND Attachment in the Release Task THEN SA2_CHECK=OK ELSE SA2_CHECK=FAIL
20	Test report is compliant with SA2	-	-	Manual check requested by QC
21	Test checks	Answer for 10.1 to 10.8 in Certification Report	Summary of the Test report	See Test Checks
22	Regression test execution correctly linked to tracked RfCs	-	-	Manual check requested by QC
23	Product deployment in the testbed	Testbed Result in Savannah release task	-	IF Testbed Result is PASS THEN SA2_CHECK=PASS ELSE IF Testbed Result is FAIL then SA2_CHECK=FAIL ELSE SA2_CHECK="No Testbed result available yet"
Documentation related checks
24	Release Notes specified in the Release task	Answer for 13 in Certification Report	Component Release Notes Savannah release task	IF ANSWER_13=YES and Component Release Notes NOT NULL then SA2_CHECK=OK ELSE SA2_CHECK=FAIL
25	Release Notes compliant with SA2	-	-	Manual check requested by QC
26	Are there configuration changes in this release?	Answer for 14 in Certification Report	-	-
27	If yes, are the changes reported in he Component Release Notes	Answer for 15 in Certification Report	Component Release Notes in Savannah release task	-
28	Documentation checks	Answer for 16.1 to 16.5 in Certification Report	Manual evaluation	See Documentation Checks

Certification Report Template v2.0

Check Number	QA Artifact	Information extracted from	Information compared to	SA2 Check
General checks
1	RfCs are listed	Answer for 1.2 in Certification Report	List of RfCs in Savannah release task	IF ANSWER_1.2=YES AND List of RfCs NOT NULL in the Release Task THEN SA2_CHECK=PASS ELSE SA2_CHECK=FAILED
2	Licence information	Answer for 15 in Certification Report	License in Savannah release task	IF ANSWER_15=YES AND License NOT NULL then SA2_CHECK=OK ELSE SA2_CHECK=FAIL
3	VCS Tag available	Answer for 2 in Certification Report	ETICS	Manual check for the time being
ETICS checks
4	Subsystem configuration builds without errors	Answer for 4 + ETICS Subsystem Configuration Name in Certification Report	ETICS	Manual check for the time being
5	Binary rpms available in the repository	Answer for 5.1 in Certification Report + Package List in Savannah release task	ETICS	Manual check for the time being
6	Binary tarballs available in the repository	Answer for 5.2 in Certification Report + Package List in Savannah release task	ETICS	Manual check for the time being
7	Source rpms available in the repository	Answer for 6.1 in Certification Report + Package List in Savannah release task	ETICS	Manual check for the time being
8	Source tarballs available in the repository	Answer for 6.2 in Certification Report + Package List in Savannah release task	ETICS	Manual check for the time being
9	Binary packages specified in the release task	Answer for 7 in Certification Report	Package List in Savannah release task	Manual check for the time being
10	Source packages specified in the release task	Answer for 8 in Certification Report	Package List in Savannah release task	Manual check for the time being
Testing related checks
11	Successful in-house deployment of the component from the EMI repository	Answer for 14 in Certification Report	AND of all test results reported in the Summary of Test report	IF ANSWER_14=YES and (Clean Installation=PASS or NA)AND(Upgrade Installation=PASS or NA)AND(Unit Tests=PASS)AND(Basic Func Tests=PASS) AND (Regression Tests=PASS or NA) THEN SA2_CHECK=OK ELSE SA2_CHECK=FAIL
12	Test Plan exists and is specified in the release task	Answer for 9 in Certification Report	Test Plan Link in Savannah release task	IF ANSWER_9=YES and Test Plan Link in the Release Task is a valid link then SA2_CHECK=OK ELSE SA2_CHECK_FAIL
13	Test report is specified in the release task	-	Latest attachment in Savannah release task where the word test + report are present.	IF Attachment in the Release Task THEN SA2_CHECK=OK ELSE SA2_CHECK=FAIL
14	Test results	Answer for 10.1 to 10.8 in Certification Report	Summary of the Test report	See Test Checks
Documentation related checks
15	Release Notes specified in the Release Task	Answer for 12 in Certification Report	Component Release Notes Savannah release task	IF ANSWER_12=YES and Component Release Notes NOT NULL then SA2_CHECK=OK ELSE SA2_CHECK=FAIL
16	Documentation checks	Answer for 11.1 to 11.5 in Certification Report	Manual evaluation	See Documentation Checks

Documentation Checks

The following check should be performed for each Document Type:

Document specific answer in the cert report	Documentation State according to the reviewer	SA2_check	Problems/errors field in the Dashboard	Verification report answer
YES	Updated	PASS	-	Updated (reviewed on DD.MM.YYYY)
	Missing	FAIL	Inconsistent information: Document is missing according to the documentation review performed on DD.MM.YYYY	Missing (reviewed on DD.MM.YYYY)
	NA	FAIL	Inconsistent information: Document is not applicable according to the documentation review performed on DD.MM.YYYY	NA (reviewed on DD.MM.YYYY)
	Outdated	FAIL	Inconsistent information: Document is outdated according to the documentation review performed on DD.MM.YYYY	Outdated (reviewed on DD.MM.YYYY)
	Unknown	FAIL	Not reviewed	Not reviewed
NO	Updated	FAIL	Inconsistent information: Document is outdated according to the PT	Updated (reviewed on DD.MM.YYYY)
	Missing	FAIL	-	Missing (reviewed on DD.MM.YYYY)
	NA	FAIL	Inconsistent information: Document is not applicable according to the documentation review performed on DD.MM.YYYY	NA (reviewed on DD.MM.YYYY)
	Outdated	FAIL	-	Outdated (reviewed on DD.MM.YYYY)
	Unknown	FAIL	Not reviewed	Not reviewed
NA	Updated	FAIL	Inconsistent information: Document is not applicable according to the PT	NA
	Missing	FAIL	Inconsistent information: Document is not applicable according to the PT	NA
	NA	PASS	-	NA
	Outdated	FAIL	Inconsistent information: Document is not applicable according to the PT	NA
	Unknown	FAIL	Not reviewed	Not reviewed

Test Checks

The following check should be performed for each Test Type:

Test specific answer in the cert report	Test result according to the Test report	SA2_check	Problems/errors field in the Dashboard	Verification report answer
YES	PASS	PASS	-	YES
	FAIL	FAIL	Inconsistent Information in the reports	??
	NA	FAIL	Inconsistent Information in the reports	??
NO	PASS	FAIL	Inconsistent Information in the reports	??
	FAIL	FAIL	Test has failed	YES
	NA	FAIL	Inconsistent Information in the reports	??
NA	PASS	FAIL	Inconsistent Information in the reports	??
	FAIL	FAIL	Inconsistent Information in the reports	??
	NA	PASS	-	NA

Dashboard comment fields

QC would like to add some comments in their verification report. The following editable fields need to be present in the Dashboard for QC to add their comments:

• Task verification remarks
• Test report verification remarks
• Certification report verification remarks
• General remarks

Verification Report

The dashboard should generate the following Verification Report.

Database schema

Requirements from SA2

Requirements from SA1 QC

Requirements from JRA1 QC

Development Plan

Developers

See DevEMIVerificationDashboard

-- MariaALANDESPRADILLO - 09-Jun-2011