EMI QA Verification Dashboard

Introduction

EMI releases need to be verified from the quality assurance perspective according to the EMI Production Release Criteria. Before a new product version can be released to production, the different SA2 policies are checked by SA2, in particular the Quality Control (QC) task. Some of the checks are done automatically, and some others, like the documentation review, are done manually. A dashboard displaying all this information for each EMI release assits QC to carry out this task.

The EMI QA Verification Dashboard automates the release verification activity extracting information from ETICS, Savannah and the certification reports. It integrates documentation review results as well, making it easier to analyse review information and produce reports. It also allows other people, like the release manager, to easily access and process this information.

Overview of the architecture

The Dashboard will be accessible through a web page created using the Django web framework. A MySQL database will contain all the necessary metadata associated to a given EMI release, product, policy, etc.

Data will be extracted from different sources:

  • Savannah release tasks:
    • List of RfCs and their corresponding status (this will imply interacting with other trackers or with the SA2.3 XML files to generate metrics, which is something to be investigated).
    • List of binaries/tarballs.
    • Test plan link.
    • Certification report
      • Documentation availability (this information will be confirmed with the results of the documentation review, performed manually)
        • Functional description
        • User Guide
        • Service Reference Card
        • ...
      • In-house testing
      • License information
    • Test report
      • Unit tests
      • Deployment tests
      • Functionality tests
      • ...
  • ETICS:
    • Subsystem build information.
    • Binary/Source packages and tarballs available in the repository.

Dashboard_Architecture.jpg

In the end, we should be able to see the following type of information per EMI release, supported platform and product:

QA_Dashborad.jpg

We should be able to export this information in suitable formats to be able to generate statistics and reports. We should also be able to generate the verification reports that are attached to the release tasks.

Dashboard checks

The different checks performed by the Dashboard are the following:

Certification Report Template v3.0/3.1

Check Number QA Artifact Information extracted from Information compared to SA2 Check
General checks
1 RfCs are listed Answer for 1.2 in Certification Report List of RfCs in Savannah release task IF ANSWER_1.2=YES AND List of RfCs NOT NULL in the Release Task THEN SA2_CHECK=PASS
ELSE SA2_CHECK=FAILED
2 ALERT! All RfCs are "Tested/Not Tested" Answer for 1.3 in Certification Report XML files Manual check for the time being
3 ALERT! Relevant RfCs have an associated regression test - - Manual check requested by QC
4 Licence information Answer for 8 in Certification Report License in Savannah release task IF ANSWER_8=YES AND License NOT NULL then SA2_CHECK=OK ELSE SA2_CHECK=FAIL
5 VCS Tag available Answer for 2 in Certification Report ETICS Manual check for the time being
6 ALERT! Certification report is compliant with SA2 - - Manual check requested by QC
7 ALERT! Certification report is complete in all sections - - Manual check requested by QC
ETICS checks
8 ALERT! Subsystem configuration is locked Answer for 3.1 + ETICS Subsystem Configuration Name in Certification Report ETICS Manual check for the time being
9 Subsystem configuration builds without errors Answer for 3.2 + ETICS Subsystem Configuration Name in Certification Report ETICS Manual check for the time being
10 ETICS configuration correctly listed - - Manual check requested by QC
11 Binary rpms available in the repository Answer for 4.1 in Certification Report + Package List in Savannah release task ETICS Manual check for the time being
12 Binary tarballs available in the repository Answer for 4.2 in Certification Report + Package List in Savannah release task ETICS Manual check for the time being
13 Source rpms available in the repository Answer for 5.1 in Certification Report + Package List in Savannah release task ETICS Manual check for the time being
14 Source tarballs available in the repository Answer for 5.2 in Certification Report + Package List in Savannah release task ETICS Manual check for the time being
15 Binary packages specified in the release task Answer for 6 in Certification Report Package List in Savannah release task Manual check for the time being
16 Source packages specified in the release task Answer for 7 in Certification Report Package List in Savannah release task Manual check for the time being
Testing related checks
17 Successful in-house deployment of the component from the EMI repository Answer for 14 in Certification Report AND of all test results reported in the Summary of Test report IF ANSWER_15=YES AND (Clean Installation=PASS or NA)AND(Upgrade Installation=PASS or NA)AND(Unit Tests=PASS)AND(Basic Func Tests=PASS) AND (Regression Tests=PASS or NA) THEN SA2_CHECK=OK ELSE SA2_CHECK=FAIL
18 Test Plan exists and is specified in the release task Answer for 10 in Certification Report Test Plan Link in Savannah release task IF ANSWER_10=YES AND Test Plan Link in the Release Task is a valid link then SA2_CHECK=OK ELSE SA2_CHECK_FAIL
19 Test report is specified in the release task Answer for 11 in Certification Report Latest attachment in Savannah release task where the word test + report are present. IF ANSWER_11 AND Attachment in the Release Task THEN SA2_CHECK=OK ELSE SA2_CHECK=FAIL
20 ALERT! Test report is compliant with SA2 - - Manual check requested by QC
21 Test checks Answer for 10.1 to 10.8 in Certification Report Summary of the Test report See Test Checks
22 ALERT! Regression test execution correctly linked to tracked RfCs - - Manual check requested by QC
23 ALERT! Product deployment in the testbed Testbed Result in Savannah release task - IF Testbed Result is PASS THEN SA2_CHECK=PASS ELSE IF Testbed Result is FAIL then SA2_CHECK=FAIL ELSE SA2_CHECK="No Testbed result available yet"
Documentation related checks
24 Release Notes specified in the Release task Answer for 13 in Certification Report Component Release Notes Savannah release task IF ANSWER_13=YES and Component Release Notes NOT NULL then SA2_CHECK=OK ELSE SA2_CHECK=FAIL
25 ALERT! Release Notes compliant with SA2 - - Manual check requested by QC
26 ALERT! Are there configuration changes in this release? Answer for 14 in Certification Report - -
27 ALERT! If yes, are the changes reported in he Component Release Notes Answer for 15 in Certification Report Component Release Notes in Savannah release task -
28 Documentation checks Answer for 16.1 to 16.5 in Certification Report Manual evaluation See Documentation Checks

Certification Report Template v2.0

Check Number QA Artifact Information extracted from Information compared to SA2 Check
General checks
1 RfCs are listed Answer for 1.2 in Certification Report List of RfCs in Savannah release task IF ANSWER_1.2=YES AND List of RfCs NOT NULL in the Release Task THEN SA2_CHECK=PASS
ELSE SA2_CHECK=FAILED
2 Licence information Answer for 15 in Certification Report License in Savannah release task IF ANSWER_15=YES AND License NOT NULL then SA2_CHECK=OK ELSE SA2_CHECK=FAIL
3 VCS Tag available Answer for 2 in Certification Report ETICS Manual check for the time being
ETICS checks
4 Subsystem configuration builds without errors Answer for 4 + ETICS Subsystem Configuration Name in Certification Report ETICS Manual check for the time being
5 Binary rpms available in the repository Answer for 5.1 in Certification Report + Package List in Savannah release task ETICS Manual check for the time being
6 Binary tarballs available in the repository Answer for 5.2 in Certification Report + Package List in Savannah release task ETICS Manual check for the time being
7 Source rpms available in the repository Answer for 6.1 in Certification Report + Package List in Savannah release task ETICS Manual check for the time being
8 Source tarballs available in the repository Answer for 6.2 in Certification Report + Package List in Savannah release task ETICS Manual check for the time being
9 Binary packages specified in the release task Answer for 7 in Certification Report Package List in Savannah release task Manual check for the time being
10 Source packages specified in the release task Answer for 8 in Certification Report Package List in Savannah release task Manual check for the time being
Testing related checks
11 Successful in-house deployment of the component from the EMI repository Answer for 14 in Certification Report AND of all test results reported in the Summary of Test report IF ANSWER_14=YES and (Clean Installation=PASS or NA)AND(Upgrade Installation=PASS or NA)AND(Unit Tests=PASS)AND(Basic Func Tests=PASS) AND (Regression Tests=PASS or NA) THEN SA2_CHECK=OK ELSE SA2_CHECK=FAIL
12 Test Plan exists and is specified in the release task Answer for 9 in Certification Report Test Plan Link in Savannah release task IF ANSWER_9=YES and Test Plan Link in the Release Task is a valid link then SA2_CHECK=OK ELSE SA2_CHECK_FAIL
13 Test report is specified in the release task - Latest attachment in Savannah release task where the word test + report are present. IF Attachment in the Release Task THEN SA2_CHECK=OK ELSE SA2_CHECK=FAIL
14 Test results Answer for 10.1 to 10.8 in Certification Report Summary of the Test report See Test Checks
Documentation related checks
15 Release Notes specified in the Release Task Answer for 12 in Certification Report Component Release Notes Savannah release task IF ANSWER_12=YES and Component Release Notes NOT NULL then SA2_CHECK=OK ELSE SA2_CHECK=FAIL
16 Documentation checks Answer for 11.1 to 11.5 in Certification Report Manual evaluation See Documentation Checks

Documentation Checks

The following check should be performed for each Document Type:

Document specific answer in the cert report Documentation State according to the reviewer SA2_check Problems/errors field in the Dashboard Verification report answer
YES Updated DONE PASS - Updated (reviewed on DD.MM.YYYY)
Missing ALERT! FAIL Inconsistent information: Document is missing according to the documentation review performed on DD.MM.YYYY Missing (reviewed on DD.MM.YYYY)
NA ALERT! FAIL Inconsistent information: Document is not applicable according to the documentation review performed on DD.MM.YYYY NA (reviewed on DD.MM.YYYY)
Outdated ALERT! FAIL Inconsistent information: Document is outdated according to the documentation review performed on DD.MM.YYYY Outdated (reviewed on DD.MM.YYYY)
Unknown ALERT! FAIL Not reviewed Not reviewed
NO Updated ALERT! FAIL Inconsistent information: Document is outdated according to the PT Updated (reviewed on DD.MM.YYYY)
Missing ALERT! FAIL - Missing (reviewed on DD.MM.YYYY)
NA ALERT! FAIL Inconsistent information: Document is not applicable according to the documentation review performed on DD.MM.YYYY NA (reviewed on DD.MM.YYYY)
Outdated ALERT! FAIL - Outdated (reviewed on DD.MM.YYYY)
Unknown ALERT! FAIL Not reviewed Not reviewed
NA Updated ALERT! FAIL Inconsistent information: Document is not applicable according to the PT NA
Missing ALERT! FAIL Inconsistent information: Document is not applicable according to the PT NA
NA DONE PASS - NA
Outdated ALERT! FAIL Inconsistent information: Document is not applicable according to the PT NA
Unknown ALERT! FAIL Not reviewed Not reviewed

Test Checks

The following check should be performed for each Test Type:

Test specific answer in the cert report Test result according to the Test report SA2_check Problems/errors field in the Dashboard Verification report answer
YES PASS DONE PASS - YES
FAIL ALERT! FAIL Inconsistent Information in the reports ??
NA ALERT! FAIL Inconsistent Information in the reports ??
NO PASS ALERT! FAIL Inconsistent Information in the reports ??
FAIL ALERT! FAIL Test has failed YES
NA ALERT! FAIL Inconsistent Information in the reports ??
NA PASS ALERT! FAIL Inconsistent Information in the reports ??
FAIL ALERT! FAIL Inconsistent Information in the reports ??
NA DONEPASS - NA

Dashboard comment fields

QC would like to add some comments in their verification report. The following editable fields need to be present in the Dashboard for QC to add their comments:

  • Task verification remarks
  • Test report verification remarks
  • Certification report verification remarks
  • General remarks

Verification Report

The dashboard should generate the following Verification Report.

Database schema

Dashboard_DB_Schema.jpg

Requirements from SA2

Requirement Comments from SA2 Comments from Developer Implemented?
The dashboard should allow to select results per EMI release and supported platform. This will probably imply to store the dates on which different updates are done. Maybe an EMI_release entity is needed in the DB. OK per EMI release In progress
The dashboard should allow to select resuls per EMI product   OK In progress

Requirements from SA1 QC

Requirement Comments from SA1 QC Comments from Developer Implemented?
Add a new check: Successful deployment of the product in the testbed QC suggested to have a new field in Savannah that the testbed manager will use to report about succesful deployment in the testbed. Discussion started with Cristina and Danilo. To be added as soon as v3.1 of Change Management Policy is approved.  
Add a new check: Succesful inter-component testing Clarification on how inter-component testing is performed is needed first. JRA1 has been contacted to clarify this. No feedback from JRA1. We assume this will be part of the tests performed by the Testbed Team.  
Add a new check: Whether fixed RfCs have an associated regression test This needs a new field in either the tracker or the information provided in the XML files that would indicate whether an RfC has an associated regression test. Under discussion at the EMT.    

Requirements from JRA1 QC

Requirement Comments from SA1 QC Comments from Developer Implemented?
Add a new check: Immediate/High priority RfCs have a corresponding release task This is nowadays partly provided by the EMT metrics report, but it will be indeed very useful to integrate this type of information in the Verification Dashboard for a future version    

Development Plan

Planning.jpg

Developers

See DevEMIVerificationDashboard

-- MariaALANDESPRADILLO - 09-Jun-2011

Topic attachments
I Attachment History Action Size Date Who Comment
JPEGjpg Dashboard_Architecture.jpg r2 r1 manage 129.2 K 2011-06-09 - 18:06 MariaALANDESPRADILLO  
JPEGjpg Dashboard_DB_Schema.jpg r1 manage 99.1 K 2011-06-14 - 15:38 MariaALANDESPRADILLO  
Texttxt EMI_Certification_Report_template_v2.0.txt r1 manage 2.5 K 2011-07-28 - 12:19 MariaALANDESPRADILLO  
JPEGjpg Planning.jpg r1 manage 45.2 K 2011-06-14 - 16:57 MariaALANDESPRADILLO  
JPEGjpg QA_Dashborad.jpg r2 r1 manage 213.5 K 2011-07-07 - 14:13 MariaALANDESPRADILLO  
Edit | Attach | Watch | Print version | History: r18 < r17 < r16 < r15 < r14 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r18 - 2012-06-14 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback