ETICS-2 Requirements from SA3

Summary

The requirements have been divided into 'build', 'testing' and 'release'. As SA3 already has systems in place for 'testing' and 'release' these are of lower priority than 'build' requirements where we rely entirely on ETICS.

Agreed (subject to further detail where necessary)

Requirement	bug #	Date for delivery
Multi package configurations (one configuration has multiple packages)	48053	end Jan 09 (as announced by the ETICS team here), not yet delivered
Support packaging for the Mac OS X platform	48143
Multiple locking against different platforms and configurations	48104

To be finalised between SA3 and JRA1;

Requirement	bug #	Date for delivery
OS native installation of build dependencies, build machines as VMs/chroot	48049
Possibility to inject/build source rpms	48105
Signed packages	47958
Service delivery based on SLAs	48051
New subsystems (tag paradigm, one component in multiple subsystems)	48054	Feb 09 (as announced by the ETICS team here), not yet delivered

Requirements already fulfilled:

Requirement	bug #	Date for delivery
Creation of source tarballs and rpms	48050	first version delivered in Feb 09
Deployment (installability) tests	48052	new version received Jan 09 (previous versions didn't work)

Build

Performance. By common agreement this is priority number one and nothing else should be worked on until this is fixed. For a number of tasks ETICS is currently unusable (eg managing the project configuration).

• Improve performance to an acceptable level - DONE!

ETICS service stability

• Problems are seen with the stability of the ETICS service: build nodes being not in good shape (AFS not properly working), unavailability of build results. ETICS stability has to be ensured.

To address next

• Add "create source tarballs" as product of the build
  • Provide a switch which means "create source tarballs wherever possible, but do not fail otherwise"
• Produce signed rpms (the ETICS signature proves that the rpms have not been altered since the build, it is not a validation/certification from ETICS).

Porting activity

• Partition of privileges on the basis of platform to allow porters to make progress on non-core platforms without affecting the main build
• Source patch management
• Easy builds with alternative compilers and python versions, eg SL4/gcc4.1/python2.5

Requirements shared with developers

• easily build multiple subsystems against the project
• easily build multiple components against the project
• compatibility between the cache and merge options
• multiple subsystems in source mode with merge

Triggering rebuilds

• utility to increase age across a subsystem to force a rebuild
• utility to manage the rebuild of dependent components when a library changes

Source rpms

• establish exactly what the constraints are on creating source rpms. Ideally, ETICS should decouple source creation and build - the process should be in two halves;
  1. ETICS produces source.tgz files of source and these and only these source.tgz are used to create binary.tgz.
  2. ETICS produces .src.rpm and only .src.rpm are ever used to generate .i386.rpm with input coming from nowhere else.

Chroot environments

• Support build support in chroot environments (virtual machines) so the dependencies are installed as they have been configured to expect at runtime (ie at --configure=prefix location). This is also an issue with external dependencies. E.g. mysql-devel has a sybmolic link to mysql. Both of them are registered in ETICS, but using them as dependencies for building a component does not work. ETICS install both externals into different directories and thus the symbolic link is broken.

Other

• apt/yum support on the ETICS permanent repository
• support for files under /etc in packager
• support for multiple packages from one component/spec , eg to produce mylib and mylib-devel

Locking

• Review of locking. The current implementation of 'resolved' locking introduces a number of complexities which may not justify the advantages.

Testing

We are mainly interested in deployment tests. When these are working we can consider doing system tests,

Deployment Tests

Installation of the middleware without any configuration.

Immediate Deployment Test after a build by a developer

For any set of artifacts produced by a developer (typically in the volatile repository) this test should check if well defined versions (production, pps or cert) of the affected gLite services can be updated with these artifacts. The following points have to be considered

• It should be easy for a developer to launch the deployment test for selected artefacts (project, subsystem or component) of a given build.
• The user has to be able to specify against which gLite service and which version (production, pps, certification) the test should be run.
• ETICS must be able to produce repositories (yum, apt, ...) containing artefacts.
• ETICS must be able to install (as root) production, pps as well as cert gLite services.

Deployment tests for patches

For a given patch, based on the "Affected Metapackages" we want to install these node types and upgrade them with the patch. For this ETICS would have to be able to dynamically run sub-tests.

Regular Deployment Test for a complete gLite build

For a complete build of gLite a deployment of the various gLite services should be done.

• Deployment of a service from scratch as well as updating of existing services should be tested.
• For deployment from scratch ETICS has to know about the software that has to be installed for a particular service. This is on the level of individual rpms. Lists of rpms for revery service are being maintained by the release team. A way has to be found how ETICS can get/maintain this information.
• It has to be decided on which build of gLite (HEAD etc.) to do the deployment tests.

System Tests

Simple regression tests

By "simple" we mean tests that can be done on one gLite node. The node is configured to be part of an existing grid infrastructure (e.g. the certification testbed). In this category we do not consider tests that require the installation and configuration of several gLite services. The following points have to be considered:

• ETICS must be able to install, update and configure gLite services with versions as defined by the release management (production, pre-production, certification).
• A useful application would be regression testing for patches: ETICS should be able to parse a Savannah patch, configure and install the affected node types and run the corresponding regression tests (provided by SA3).

Release

This section is about requirements related to package level integration, meta-packages, rpm lists, repository management, deployment tests etc. For SA3, these issues are not seen as a priority as they are already handled internally. If our current tooling can be reproduced in ETICS there will be advantages to managing all information in one place.

• Easy identification necessary information via package names
  • The process trigger is a set of package names associated in a patch
• Bulk operations on multiple metapackages
  • Package operations on the distribution will affect one or more metapackages, they should not have to be treated individually
• Metapackage generation with version files
  • This is the option to add some files in /opt/glite/version/<package-name> in each metapackage
• Metapackage maintenance through multiple stages with different names
  • Metapackages need to be named differently for PPS and production to avoid problems triggered by patches staying in PPS for varying lengths of time
• Decoupling of build lists and deployment lists
  • The project definition for build is not the project definition for release.
• Provision of AFS repo for AA
  • Documented here
• Support for yum repositories, with release, externals and updates
  • Updated packages go into updates, the release dir should not be changed
• Per metapackage repositories
  • To allow for divergence in dependencies, we need to maintain a different repository for each metapackage. In reality, these are currently all symlinked to a generic repo.
• Fast metapackage production
  • It is essential that metapackage generation is very fast
• Metapackages with full >= dependencies (not just 'first level' deps)
  • This enables tracking of whether machines are fully updated

Release Use Cases

• Patch -> certification
  • Repo creation for the patch
  • Optional meta-package
  • NB a patch is a list of rpm names, not necessarily configurations
• N Patches -> PPS
  • Add patch contents to PPS repo (respecting Release, updates and externals)
  • Identify and update all relevant metapackages
  • Verify integrity of repo
• N Patches -> production
  • Update of prod repo on AFS ready for mirroring to linuxsoft
  • Management of web content on the basis of patch info (... I'm guessing ETICS is not interested in doing this)
  • Generation of rpm lists (inc '2nd level' deps) for each service (for use with quattor etc)
• Addition of new rpm to release
• Temporary deviation from standard rpm set by one metapackage
  • eg condor is upgraded on WMS but not on other components
• Patch rejection on x86_64 while equivalent for i386 proceeds
• Query the version of package X in particular metapackage
• Query current status of a patch (pps, prod etc)

Note

• There is no need foreseen to build a metapackage + all dependencies

ETICS schedule

ETICS internal planning is publicised at https://etics.web.cern.ch/etics/collaboration/WP1/project.asp