Etics build system overview.

Etics is the build system used by EGEE. The main goal of ETICS is to handle every aspect of the software development workflow, i.e. build, release and testing, in such a way that every operation has to be completely independent from the platforms, the languages and the tools involved in the process. The content of an ETICS site is structured as a set of hierarchies of entities, with one hierarchy for each project registered into the service; an entity can be a subsystem of the project or a component of a given subsystem. Every entity groups different configurations; a configuration contains all the information and instructions required for building, packaging and testing the software on one or more platforms. In the following it is referred to the set of information and instructions of a configuration as metadata. All glite components and subsystems should be in the project glite Middleware project. Interaction with Etics can be done via the web interface or command line tools. For more in-depth details and tutorials on Etics please see the Etics documentation.

ETICS and user permissions.

Any user must register to ETICS in order to have the correct set of permissions granted to its role. The registration can be performed at the following page: https://etics.cern.ch:8443/eticsAdmin/public/registration/requestRegistration.jsp

Since the authentication mechanism of ETICS is based upon X509 public key infrastructure (PKI) before sending the request for registration the user must be sure that his credentials, the private key and the related certificate, have been correctly installed into the web browser. ETICS grants access even to unregistered users, providing a special "guest" account in this case; the system allows those users to perform just read-only operations.

ETICS provides a fine-grained permission management; rights can be set up for the entire project or for each subsystem or component and also for a given configuration of subsystem or component. For further details about the permission management refer to: Understanding Security in ETICS

Installation, configuration and set up of the ETICS command line client

The steps required for the installation of the ETICS client are:

Download the Etics workspace setup script

wget  "http://eticssoft.web.cern.ch/eticssoft/repository/etics-client-setup.py -O etics-client-setup

Run the Etics workspace setup script

python etics-client-setup

Set the ETICS_HOME env variable

export ETICS_HOME=`pwd`/etics

Add the Etics commands to your PATH

export PATH=$PATH:$ETICS_HOME/bin

All the configuration parameters for the ETICS client are defined into a configuration file whose default location is $HOME/.etics.conf. Most of the parameters listed into the configuration file can be also redefined in the command line. The basic properties for the account setting are:

the path of the user certificate

x509_user_cert

the path of the user private key

x509_user_key

they are required in order to establish the secure connection and to authenticate against the ETICS server and are contained into the user section. The required parameter for the connection is server, which is the fully qualified domain name of the service to contact and is contained into the system section, the main service running is located at etics.cern.ch. The ETICS client is usually able to detect the correct platform of the system where it is running on, whever this is not possible the user can force the platform definition with the platform property listed into the generic properties section. Every ETICS command can work only inside a well formatted workspace, i.e. a directory of the local filesystem which contains the project metadata files. The commands that must be invoked for setting up the workspace are:

for the initialization

etics-workspace-setup

for downloading the project metadata

etics-get-project org.glite

Creation of an ETICS entity

The creation of a subsystem is carried out by the project administrator; the creation of a component is carried out by the project administrator or by the module administrator of the subsystem that component belongs to. The suitable command for this operation is etics-module:

add a subsystem to the project

etics-module add --parent org.glite --subsystem subsystem

add a component to a subsystem

etics-module add --parent subsystem --component component

By default the system creates a configuration for both the subsystem and the component whose name is org.glite.subsystem.HEAD or org.glite.subsystem.component.HEAD respectively.

The naming convention for subsystem and components should follow the same rules defined for the CVS module name, discussed in section 2

the subsystem name should be

org.glite.subsystem_name

the component name should be

org.glite.subsystem_name.component_name

There's no relation between the name of the entity and the name of the CVS module whom the entity refers to.

Every operation on a subsystem or component configuration can be performed using the command etics-configuration:

add a configuration to a component

etics-configuration add -c component_config -i input_file component

modify metadata of a configuration

etics-configuration modify -c component_config -i input_file component

clone a configuration

etics-configuration clone -c target-configuration source-configuration component

list the existing configuration for a component.

etics-list-configuration component

Most of the commands described above requires that a set of metadata must be defined into a file. This file, called in the following "ini-file", is divided into sections and must contain

• the main definition for a configuration:

[Configuration-glite-mysubsystem-mycomponent_R_1_0_0_0]
moduleName = org.glite.mysubsystem.mycomponent
displayName = Brief description of mycomponent
description = Full description of mycomponent
projectName = org.glite
age = 0
tag = glite-mysubsystem-mycomponent_R_1_0_0_0
version = 1.0.0
path = ${projectName}/${moduleName}/${version}/noarch/

• the section that defines the CVS commands for the default platform:

[Platform-default:VcsCommand]
displayName = Brief description of the CVS commands of mycomponent
description = Full description of the CVS commands of mycomponent
tag = cvs -d ${vcsroot} tag -R ${tag} ${moduleName}
checkout = cvs -d ${vcsroot} co -r ${tag} -P ${moduleName}

• the section that defines the build commands for the default platform:

[Platform-default:BuildCommand]
postpublish = None
packaging = None
displayName = Brief description of the build commands of mycomponent
description = Full description of the build commands of mycomponent
doc = None
prepublish = None
publish = None
compile = None
init = None
install = None
clean = None
test = None
configure = None

for further details concerning the syntax of an ini-file refer to the ETICS user guide.

Dependencies for an ETICS entity and the lock operation

One powerful feature of ETICS is the capability of handling the software dependencies, such as libraries and tools, required by a given subsystem or component with a minimum amount of effort from integrators and developers. Any dependency or resource must be registered in the system; in this way any entity can be accessed by a project, any possible combination is allowed, a subsystem can be a dependency for a project or a subsystem contained into it, a component of a project can be a dependency for another component of a different project.

The most important example concerns the project externals which collects all the resources published by external projects. It is strongly recommended to use only the external resource contained into that project in order to exploit all the features of the system during the release step.

According to the way an ETICS entity uses a given dependency the system distinguishes two different criterions:

• static or dynamic dependency: the former defines a fixed release version for the resource inside an entity configuration, the latter allows the entity to inherit the release version from the parent entities; they're mutually exclusive.
• build-time or run-time dependency: the former specifies that the resource is required only during the build or testing step, the latter instructs the system in order to have a dependency when the artifact produced is used, typically putting such requirement into the package; the two definitions can be combined together.

When an entity defines a dynamic dependency it delegates the system to select the correct version of the resource, in general the version decided for the entire project. It is strongly recommended to make use of this feature, especially for the external dependencies, if no particular requirements must be considered.

Dynamic dependencies are not able to guarantee that a given configuration for an entity can be reproduced at any time. It is necessary to freeze, or lock, the configuration of that entity together with all the configurations of the sub-entities and dependencies. The lock operation is performed with the command etics-configuration lock.

Checking out and building a component

The steps required for downloading and building the code of a component are:

Check out a specific configuration

etics-checkout --config component_configuration_name --project-config project_configuration_name component_name

Locally build the code

etics-build component_name

Integrators and developers can schedule a remote build for the configuration of the managed component with the following command

etics-submit build --config component_configuration_name --platforms selected_platforms component_name

The system can publish the artifacts, such as tarballs or packages (rpm, deb), either into a temporary area, called "volatile" repository, or into the permanent repository. The volatile repository is suitable only for development purposes, the permanent repository must be used for publishing the official artifacts of a release and it can be populated only if the configuration is locked. The results and every reference to artifacts produced by the remote build are contained into the ETICS reports site.

All the operations above can be applied to an entire subsystem replacing the component name and the configuration with subsystem name and the corresponding configuration.

ETICS and the software release management

For each subsystem and component registered in ETICS it is possible to distinguish two different types of configuration:

• the ETICS configuration for a branch
• the ETICS configuration for a release

The former is suitable for development purposes, it contains references to CVS branches or to the HEAD main branch and cannot be locked as any definition can be changed according to the development requirements. The latter represents a frozen release thus it must be guaranteed it cannot be changed; it contains only references to CVS "sticky" tag and must be locked in order to freeze the entire dependency structure and the set of commands, properties and variables. The naming convention for a frozen release configuration should be the same as the CVS tag, thus it should follow the notation:

for a subsystem

glite-subsystem_R_major_minor_revision_age

for a component

glite-subsystem-component_R_major_minor_revision_age

If the configuration refers to an internal release, suitable for tests only purpose for example, any notation is allowed but it is strongly recommended to put in the tag the name of the component or the subsystem together with a string summarizing the goal of the release. The naming convention for a branch configuration states that the configuration name should contain the prefix

for a subsystem

glite-subsystem_branch

for a component

glite-subsystem-component_branch

followed by a string that identifies the branch. The only exception allowed is the HEAD main branch for which the ETICS provides the default configuration name org.glite.subsystem.HEAD or org.glite.subsystem.component.HEAD.

A release configuration can be obtained from a branch configuration with the ETICS clone operation of the web application or taking the ini-files of the branch configuration as templates for the etics-tag command of the command line client.

-- PaulAndreetto - 09 Oct 2008