Continuous Integration and Testing

Objective

The continuous execution of builds and tests on newly developed software is a key activity in any software development project. To properly implement such procedure it is fundamental to automatize all tasks related to the build and test. Having such automatization in place, the continuous integration and testing procedure can then be applied in different phases of the software development process.

This page documents the work done by the Data Management product team to implement a continuous integration and testing procedure for the Data Management software products: DPM/LFC, FTS, and GFAL/lcg_util.

Tools

The execution of continuous build and tests is only possible when based on a number of tools. The DM PT adopted the following tools:

• ETICS - used to manage the submission of builds/tests exploiting a remote infrastructure that provides access to different platform types;
• YUM - used as the repository containing the RPMs generated by an etics remote build;
• YAIM - used to deploy the RPMs retrieved for the YUM repository and to configure the deployed services;
• YAIMGEN - used to execute a number of pre-defined certification tests and to generate reports;

Execution Plan

In order to have a complete continuous integration and testing procedure it's important to integrate the execution of source code builds with the execution of different types of tests under a common continuous execution plan. The current execution plan defines the following steps:

1. Source code build - Build the source code from SVN;
2. Repository creation - Create a YUM repository containing all generated RPMs;
3. Deployment tests execution - Test the upgrade of a production-level node with the generated RPMs;
4. Configuration tests execution - Test that the configuration of the previously installed services works correctly;
5. Certification tests execution - Test the execution of a number of pre-defined certification tests;
6. Report creation - Generate summary reports that are send to the product team.

The execution of these steps is fully automatize and the entire procedure runs in few minutes. The procedure is based on the ETICS remote build and remote test functionality which is invoked by a number of cronjobs. The following list provides a detailed description of how the steps above are grouped in different executions and what are the intermediate tasks automatically executed.

1. [cronjob] launches n etics remote builds (one for each platform)
   • [etics] selects the appropriate platforms
   • [etics] checkout all relevant SVN code
   • [etics] builds the code
   • [etics] packages the code in RPMs
   • [etics] creates a YUM repository with the created RPMs
   • [etics] generates the build report
2. [cronjob] launches n etics remote tests (one for each platform/metapackage)
   • [etics] (using yaim) deploys the metapackage RPMs from the recently created YUM repository
   • [etics] (using yaim) configures the deployed services
   • [etics] (using yaimgen) runs certification tests
   • [etics] (using yaimgen) generates the deployment report and test report
3. [cronjob] prepares a summary report of the build/test execution
   • [summary_report] downloads all reports of the build, deployment, and testing activities
   • [summary_report] prepares a summary report with all results for all platforms/metapackages
   • [summary_report] sends the summary report to the DM product team mailing list

As introduced before, this procedure can then be applied in different phases of the software development process. Currently the DM product team runs every night this procedure on the latest code produced by developers. As such, all changes committed by developers during the day are immediately built and tested every night. Furthermore reports are automatically produced and sent to developers informing them on the impact of their changes. As a consequence the quality of the code continuously increases. The same execution plan can also be executed in later stages of the software development process such as after the creation of a release tag. In this case the advantage is not on the continuous verification of the code quality but rather on the easiness to verify if new release tags build and pass all existing tests. This directly contributes to a decrease of the time needed to complete the release process.

Current Status

The DM product team is responsible for three subsystems (LCGDM, gliteFTS, LCGUTIL) which are stored in three SVN repositories. These repositories contain several source code components and generate different metapackages for distribution.

The continuous integration and testing procedure is currently being executed on all these components as follows:

• Source Code: based on the SVN trunk
• Execution Time: daily between 01:00-09:00
• Build Coverage: all LCGDM, gliteFTS, and LCGUTIL components
• Deployment Tests Coverage: all LCDGM metapackages
• Certification Tests Coverage: all LCDGM metapackages

In particular the etics remote builds/tests previously described are executed as follows:

• 01h00m - submission of LCGDM remote build for SL5/64, SL5/32, SLC4/64, SL4/32
• 02h00m - submission of LCGUTIL remote build for SL5/64, SL5/32, SLC4/64
• 03h00m - submission of gliteFTS build for SL5/64, SL5/32, SLC4/64
• 04h00m - submission of LCGDM remote (deployment and certification) tests for SL5/64, SL5/32, SLC4/64, SL4/32
• 08h00m - preparation of summary report and notification to PT mailing list

Below are two examples of reports automatically generated every night:

Future Plans

There are several improvements planned for the continuous integration and testing procedure. These improvements cover different areas:

Certification Tests

• Add to the current execution plan the certification tests for FTS metapackages
• Enhance the existing LCGDM certification tests with tests requiring a gLiteUI
• Finalize the TRAC list of certification tests (to know which tests can be automatized)

Report

• Improve the current email summary report
• Store the execution plan results to have historical information
• Move the report tool to a more powerful tool also capable to submit etics tests/builds that could replace the existing cronjobs

Other

• Define and execute regression tests for all metapackages
• Align with the ETICS team the usage of a common virtual machine for vnode and etics nodes
• Exploit the ETICS multi-node testing for more advanced test scenarios
• Start using the continuous integration and testing procedure not only for SVN trunk but also for SVN release tags (before the PT certification activity)

-- PedroAndrade - 08-Jun-2010