gLite Developer's Guide
Applicable to gLite as of version 3.1
December 10, 2009
1.Introduction
1.1 Purpose
This guide is for anyone developing or modifying EGEE Middleware. Aim of the document is to provide developers with a quick reference guide to quickly become familiar with development tools, software integration process and release management. It does not contain material about coding standards and naming schemas. The interested reader is advised to consult the
EGEE II Developer Guide
[R2] where these topics have been treated.
1.2 Document Organization
This guide should be considered as a live document. The guide is written in a wiki to allow faster and easier changes, contributions and document management.
Twiki2Pdf Add-On will be used to provide a pdf version as required for official documents by
EGEE Document Management Procedure [R1]. The document contains two appendices with technical information about the usage of environment variables, daemons and services in gLite as well as a statements on the license used by gLite and IPV6 compliance.
The URL for this document is
https://twiki.cern.ch/twiki/bin/view/EGEE/DevelopersGuide.
1.3 Application area
This document applies to the all software process, including development, integration and testing activities, seen from the point of view of developers.
In several sections
ETICS [R7] command line commands are used. At the time of writing this document the new
ETICS client was not yet available. Thus
ETICS commands given in this document are subject to change.
1.4 References
1.5 Feedback
For feedback or questions on this guide please contact Maria Alandes at CERN. Her contact details can be found in the CERN directory
http://consult.cern.ch/xwho.
1.6 Mailing lists
Developers are encouraged to subscribe to the EMT mailing list
project-eu-egee-middleware-emt to follow discussions and announcements. Subscribe using
CERN E-Groups [R22].
1.7 Weekly meetings (telephone conference)
Two telephone conferences are held on Mondays at 14h30 and on Wednesdays at 16h30 every week. The agenda and the call for the meeting is announced in the EMT mailing list. Agendas and minutes of previous meetings can be found in Indico category
Home > Departments > IT > GT > EMT. Developers are encouraged to participate in this meeting. They can propose discussion topics by sending them to the EMT mailing list. The meeting is chaired by the release manager and attended by product team representatives. Some other people participating regularly in the meetings are
ETICS representatives, JRA1 section leader, SA1 representatives and LCG representatives.
The EMT also performs a bug triage, making an objective assessment of a bug's
priority. Once a week the EMT will
- Check all bugs of severity (which is set by the submitter) major & critical submitted in the last 7 days
- Assess their severity and consequent priority
- Update the bug's priority, with a comment to make it clear that the priority has been set by the EMT
Typically the severity will not be changed as it records the submitter's assessment
Background and assessment criteria can be found in
Problem Management and Change Management in gLite
2. Software Configuration and Version Control
2.1 Overview
CVS (Concurrent Versions System (CVS), also known as Concurrent Versioning System) and
ETICS (eInfrastructure for Testing, Integration and Configuration of Software) are used to manage software complexity and to improve the software quality.
CVS is used to allow several developers to collaborate on a software module development in a distributed (in space and time) environment. CVS also keeps track of all work and all changes in the source code.
ETICS is used to
- build software modules managing complex dependencies and software integration with configuration concept
- build software modules on different platforms
- manage software configuration versioning
- manage software release process
2.2 Repository
The gLite CVS repository is hosted by the
CERN Central CVS Service [R4]. It can be accessed via the generic
CVSROOT:
glite.cvs.cern.ch:/cvs/glite
For up to date information on how to access the CERN CVS have a look at the
CERN Central CVS Service [R4] webpage.
2.3 Read Only Access gLite CVS Repository
The CVS repository can be accessed, in read only mode, two ways:
export CVSROOT=:pserver:anonymous@glite.cvs.cern.ch:/cvs/glite
The full documentation for accessing CERN CVS can be found on the
CERN CVS page [R18].
2.4 Authentication: Read/Write Access
In order to have CVS write access, it is necessary to have a CERN AFS account. Instructions to request an account are available on the
CERN account webpage [R5].
2.4.1 Read/Write Access using Kerberos V
export CVSROOT=':gserver:glite.cvs.cern.ch:/cvs/glite'
2.4.2 Read/Write Access using SSH
export CVSROOT=:ext:<afs-user-name>@glite.cvs.cern.ch:/cvs/glite
export CVS_RSH=ssh
To set up SSH, follow the instructions for
Configuring SSH access from Linux/Unix on the CERN CVS page.
2.5 Components, Subsystems and CVS Directories
A component is a CVS directory that contains the source code for a given functionality and a subsystem is a logical group of components. The name of CVS directory for a component should be in the form of
org.glite.subsystem.component. Dots should not be used in the subsystem or component name. Each component should create a package where the package name is in the format
glite-subsystem-component. Anyone with write access to CVS should be able to create a components directory in CVS. After creating this directory you will need to obtain write access by emailing a request to
middleware-integration. The CVS directory for the component should contain the following directories.
src: contains the source code
doc: contains the documentation
etc: configuration files
tests: contains test material
build: not need in CVS but should be use for building the code.
2.6 CVS tags
When a component is ready to be released, the CVS componet directory should be tagged. The format of the tag should be
glite-subsystem-component_R_x_y_z_w where x, y, z and w are the major, minor, revision and age numbers, respectively, for the release.
2.7 Mail notification
Developers are encouraged to subscribe to mailing lists that provide an automatic notification on every CVS commit for the component of interest. Currently mailing lists for the following components are available: build.common-cpp, testing, ce, data, datamgt (
DPM, LCG-DM), dgas, lb (jp), sd (service-discovery), security, wms-utils, wms-ui, wms, yaim. For subscription to these mailing lists go to the
CERN E-Groups [R22] page and search for groups starting with the string
project-eu-egee-cvs.
Every developement team must have resp. set up a mailing list providing an automatic notification no matter which revision control system is being used.
2.8. Access request
If you need to have write access to a certain CVS module, please send a mail to the EMT mailing list specifying the cvs module(s) name(s) and the CERN AFS login of the user(s) who need to have access.
3. The build system
ETICS [R7] is the build system used by EGEE. The main goal of
ETICS is to handle every aspect of the software development workflow, i.e. build, release and testing, in such a way that every operation has to be completely independent from the platforms, the languages and the tools involved in the process. The content of an
ETICS site is structured as a set of hierarchies of entities, with one hierarchy for each project registered into the service; an entity can be a subsystem of the project or a component of a given subsystem. Every entity groups different configurations; a configuration contains all the information and instructions required for building, packaging and testing the software on one or more platforms. In the following it is referred to the set of information and instructions of a configuration as metadata. All gLite components and subsystems should be in the project gLite Middleware project. Interaction with
ETICS can be done via the
web interface [R7] or command line tools. For more in-depth details and tutorials on
ETICS please see the
Etics documentation [R6].
Note that it is difficult to build the middleware without the use of
ETICS.
3.1 ETICS and user permissions
Any user must register to
ETICS in order to have the correct set of permissions granted to its role. The registration can be performed at the following page:
https://etics.cern.ch:8443/eticsAdmin/public/registration/requestRegistration.jsp
Since the authentication mechanism of
ETICS is based upon X509 public key infrastructure (PKI) before sending the request for registration the user must be sure that his credentials, the private key and the related certificate, have been correctly installed into the web browser.
ETICS grants access even to unregistered users, providing a special "guest" account in this case; the system allows those users to perform just read-only operations.
ETICS provides a fine-grained permission management; rights can be set up for the entire project or for each subsystem or component and also for a given configuration of subsystem or component. For further details about the permission management refer to:
Understanding Security in ETICS
3.2 Installation, configuration and set up of the ETICS command line client
The steps required for the installation of the
ETICS client are:
- Download the Etics workspace setup script
-
wget "http://eticssoft.web.cern.ch/eticssoft/repository/etics-client-setup.py -O etics-client-setup
- Run the Etics workspace setup script
-
python etics-client-setup
- Set the ETICS_HOME env variable
-
export ETICS_HOME=`pwd`/etics
- Add the Etics commands to your PATH
-
export PATH=$PATH:$ETICS_HOME/bin
All the configuration parameters for the
ETICS client are defined into a configuration file whose default location is $HOME/.etics.conf.
Most of the parameters listed into the configuration file can be also redefined in the command line.
The basic properties for the account setting are:
- the path of the user certificate
-
x509_user_cert
- the path of the user private key
-
x509_user_key
they are required in order to establish the secure connection and to authenticate against the
ETICS server and are contained into the
user section.
The required parameter for the connection is
server, which is the fully qualified domain name of the service to contact and is contained into the
system section, the main service running is located at
etics.cern.ch.
The
ETICS client is usually able to detect the correct platform of the system where it is running on, whever this is not possible the user can force the platform definition with the
platform property listed into the generic
properties section.
Every
ETICS command can work only inside a well formatted workspace, i.e. a directory of the local filesystem which contains the project metadata files. The commands that must be invoked for setting up the workspace are:
- for the initialization
-
etics-workspace-setup
- for downloading the project metadata
-
etics-get-project org.glite
3.3 Creation of an ETICS entity
The creation of a subsystem is carried out by the project administrator; the creation of a component is carried out by the project administrator or by the module administrator of the subsystem that component belongs to. The suitable command for this operation is
etics-module:
- add a subsystem to the project
-
etics-module add --parent org.glite --subsystem subsystem
- add a component to a subsystem
-
etics-module add --parent subsystem --component component
By default the system creates a configuration for both the subsystem and the component whose name is
org.glite.subsystem.HEAD or
org.glite.subsystem.component.HEAD respectively.
The naming convention for subsystem and components should follow the same rules defined for the CVS module name, discussed in section 2
- the subsystem name should be
-
org.glite.subsystem_name
- the component name should be
-
org.glite.subsystem_name.component_name
There's no relation between the name of the entity and the name of the CVS module whom the entity refers to.
Every operation on a subsystem or component configuration can be performed using the command
etics-configuration:
- add a configuration to a component
-
etics-configuration add -c component_config -i input_file component
- modify metadata of a configuration
-
etics-configuration modify -c component_config -i input_file component
- clone a configuration
-
etics-configuration clone -c target-configuration source-configuration component
- list the existing configuration for a component.
-
etics-list-configuration component
Most of the commands described above requires that a set of metadata must be defined into a file. This file, called in the following
"ini-file", is divided into sections and must contain
- the main definition for a configuration:
[Configuration-glite-mysubsystem-mycomponent_R_1_0_0_0]
moduleName = org.glite.mysubsystem.mycomponent
displayName = Brief description of mycomponent
description = Full description of mycomponent
projectName = org.glite
age = 0
tag = glite-mysubsystem-mycomponent_R_1_0_0_0
version = 1.0.0
path = ${projectName}/${moduleName}/${version}/noarch/
- the section that defines the CVS commands for the default platform:
[Platform-default:VcsCommand]
displayName = Brief description of the CVS commands of mycomponent
description = Full description of the CVS commands of mycomponent
tag = cvs -d ${vcsroot} tag -R ${tag} ${moduleName}
checkout = cvs -d ${vcsroot} co -r ${tag} -P ${moduleName}
- the section that defines the build commands for the default platform:
[Platform-default:BuildCommand]
postpublish = None
packaging = None
displayName = Brief description of the build commands of mycomponent
description = Full description of the build commands of mycomponent
doc = None
prepublish = None
publish = None
compile = None
init = None
install = None
clean = None
test = None
configure = None
for further details concerning the syntax of an ini-file refer to the
ETICS user guide [R8].
3.4 Dependencies for an ETICS entity and the lock operation
One powerful feature of
ETICS is the capability of handling the software dependencies, such as libraries and tools, required by a given subsystem or component with a minimum amount of effort from integrators and developers. Any dependency or resource must be registered in the system; in this way any entity can be accessed by a project, any possible combination is allowed, a subsystem can be a dependency for a project or a subsystem contained into it, a component of a project can be a dependency for another component of a different project.
The most important example concerns the project
externals which collects all the resources published by external projects. It is strongly recommended to use only the external resource contained into that project in order to exploit all the features of the system during the release step.
According to the way an
ETICS entity uses a given dependency the system distinguishes two different criterions:
- static or dynamic dependency: the former defines a fixed release version for the resource inside an entity configuration, the latter allows the entity to inherit the release version from the parent entities; they're mutually exclusive.
- build-time or run-time dependency: the former specifies that the resource is required only during the build or testing step, the latter instructs the system in order to have a dependency when the artifact produced is used, typically putting such requirement into the package; the two definitions can be combined together.
When an entity defines a dynamic dependency it delegates the system to select the correct version of the resource, in general the version decided for the entire project. It is strongly recommended to make use of this feature, especially for the external dependencies, if no particular requirements must be considered.
Dynamic dependencies are not able to guarantee that a given configuration for an entity can be reproduced at any time. It is necessary to freeze, or lock, the configuration of that entity together with all the configurations of the sub-entities and dependencies. The
lock operation is performed with the command
etics-configuration lock, as in the following example:
etics-configuration lock --parent-config glite_3_2_cert -c component_configuration component_name
3.5 Checking out and building a component
The steps required for downloading and building the code of a component are:
- Check out a specific configuration
-
etics-checkout --config component_configuration_name --project-config project_configuration_name component_name
- Locally build the code
-
etics-build component_name
- Integrators and developers can schedule a remote build for the configuration of the managed component with the following command
-
etics-submit build --config component_configuration_name --platforms slc4_ia32_gcc346,slc4_x86_64_gcc346,sl5_ia32_gcc412,sl5_x86_64_gcc412 component_name
- Also a release build against a locked component can be scheduled
-
etics-submit build --config locked_component_configuration_name --platforms slc4_ia32_gcc346,slc4_x86_64_gcc346 --register component_name
The system can publish the artifacts, such as tarballs or packages (rpm, deb), either into a temporary area, called
"volatile" repository, or into the permanent repository. The volatile repository is suitable only for development purposes, the permanent repository must be used for publishing the official artifacts of a release and it can be populated only if the configuration is locked. The results and every reference to artifacts produced by the remote build are contained into the
ETICS reports site [R7].
All the operations above can be applied to an entire subsystem replacing the component name and the configuration with subsystem name and the corresponding configuration.
3.6 ETICS and the software release management
For each subsystem and component registered in
ETICS it is possible to distinguish two different types of configuration:
- the ETICS configuration for a branch
- the ETICS configuration for a release
The former is suitable for development purposes, it contains references to CVS branches or to the HEAD main branch and cannot be locked as any definition can be changed according to the development requirements.
The naming convention for a branch configuration should be as in the following:
- for a subsystem
-
glite-subsystem_branch_major_minor_revision
- for a component
-
glite-subsystem-component_branch_major_minor_revision
The latter represents a frozen release thus it must be guaranteed it cannot be changed; it contains only references to CVS tag and must be locked in order to freeze the entire dependency structure and the set of commands, properties and variables.
The naming convention for a frozen release configuration should be the same as the CVS tag, thus it should follow the notation:
- for a subsystem
-
glite-subsystem_R_major_minor_revision_age
- for a component
-
glite-subsystem-component_R_major_minor_revision_age
If the configuration refers to an internal release, suitable for tests only purpose for example, any notation is allowed but it is strongly recommended to put in the tag the name of the component or the subsystem together with a string summarizing the goal of the release.
The naming convention for a branch configuration states that the configuration name should contain the prefix
- for a subsystem
-
glite-subsystem_branch
- for a component
-
glite-subsystem-component_branch
followed by a string that identifies the branch. The only exception allowed is the HEAD main branch for which the
ETICS provides the default configuration name
org.glite.subsystem.HEAD or
org.glite.subsystem.component.HEAD.
A release configuration can be obtained from a branch configuration with the
ETICS clone operation of the web application or taking the ini-files of the branch configuration as templates for the etics-tag command of the command line client.
3.7 How to write ETICS configurations
This section gives guidelines on how to write ETICS configurations. As much information as possible should be defined in the project configuration.
Which components should be grouped into subsystems?
- A subsystem is a set of components with significant relationship at development/build time
- Two components go in the same subsystem if and only if they need to be developed/built together
- A single component should be buildable without the need to build the whole subsystem. One must thus avoid to define information (e.g. properties) on the subsystem level
Component Configurations
- Specify only dynamic dependencies in the Default platform for direct dependencies (i.e. something of that component is used)
- Do not override properties
- Do not specify .DEFAULT properties
- Should only have the default platform attached. Exceptions have to be communicated
- About
globus and vdt: Please, use globus package for build time only dependencies and vdt_globus_essentials package for run time only dependencies.
Subsystem Configurations
- Are empty apart from the list of component configurations
Project Configurations
- Specify .DEFAULT properties per platform
4. Release Workflow
4.1 Understanding the release
Since the beginning of 2010 a new release process is in place in order to prepare for the transition to EGI, the follow up project to EGEE. An overall description of the new release process can be found in the
EGEE-III gLite Product Team Release Process [R25] document. The process is based on product teams being entirely responsible for their products (development, testing, integration and release).
ETICS is used as a build system as well as a tool to create metapackages. Please refer to the
Product Team Integration Guide [R24] for a detailed explanation on how product teams should use
ETICS for the release process.
Both
ETICS and CVS are arranged on a subsystem basis. To work best, a subsystem should
- Group together components with synchronised evolution, where updates are frequently correlated.
- Contain no other unrelated components, as this imposes an artificial connection and prevents independent evolution
Unfortunately, the gLite subsystems as currently defined often break both these rules, as for historical reasons they are arranged along administrative lines. Nevertheless the scenario where updates of a component only affects the other components in the same subsystem can be regarded as the default scenario for gLite integration.
ETICS Project configurations
The release team maintains two project configurations in
ETICS for both gLite 3.1 and gLite 3.2:
- glite_3_X_cert
- glite_branch_3_X_0_dev
The configuration
glite_branch_3_X_0_dev is maintained to hold the latest tags of interest to developers, i.e. the configurations currently 'in work' and which are likely to subsequently be locked and used for a release candidate build. Any build related bugs can be considered fixed in this build. This build will not be used to create releasable software but provides an automatic build of the latest tags against new versions of dependencies, enabling anticipation of problems. Note that the tags contained in
glite_branch_3_X_0_dev will be used to create releasable software, when locked and built against
glite_3_X_cert. Developers can always build any configuration against
glite_3_X_cert in the usual way.
When a release is required, the 'in work' configuration in
glite_branch_3_X_0_dev should be built against
glite_3_X_cert and locked when everything works fine. The next configuration to be 'in work' should be communicated to the integration team so it can be added to
glite_branch_3_X_0_dev. When the packages are certified and later verified, the appropriate configuration will be transferred by the integration team to
glite_3_X_cert. For more details on
glite_3_X_cert, please check
Product Team Integration Guide.
glite_3_1_cert should be used for building gLite 3.1 and
glite_3_2_cert should be used for building gLite 3.2.
When product team A needs to use a package provided by product team B which is not yet included in
glite_3_X_cert, product team A should use the available version of the package under
glite_branch_3_X_0_dev. If the version existing there is giving problems, product team A should communicate it to the EMT. Product team B should then provide a new configuration. At the same time, product team B should certify the required package as soon as possible so product team A can include it in their release when it's certified and verified. In order to do that, product team B should certify an
Internal patch.
ETICS Project properties
The following properties are defined in
glite_3_2_cert project configurations since February 2010. For the complete list of properties please check
ETICS:
- gLiteProjectVersion: version of the last locked project configuration.
- gLiteCopyrightText:
Copyright (c) Members of the EGEE Collaboration. 2004.
See http://www.eu-egee.org/partners/ for details on the copyright holders.
- gLiteLicenseText:
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License
The standard use case
The following diagram illustrates the use case of a single component being affected. It can be extended to the case of several components within the same subsystem.
| Action |
Command |
| Check out latest configuration with glite_3_2_cert |
etics-checkout --project-config glite_3_2_cert -c <config> <component> |
| Commit changes |
cvs commit |
| Tag changes |
cvs tag |
| Create new local configuration using new CVS tag |
etics-configuration prepare -c <config> <component>
vi Configuration.ini
etics-configuration add -noautocommit -i Configuration.ini -c <new config> <component> |
| Build the new configuration locally |
etics-checkout --local -c <new config> <component>
etics-build -c <new config> <component> |
| Test provisional build |
| Create release tag |
cvs tag |
| Point ETICS to release tag |
vi Configuration.ini
etics-configuration modify --noautocommit -i Configuration.ini -c <new config> <component> |
| Commit ETICS configuration |
etics-commit |
| Lock configuration |
etics-configuration lock --parent-config glite_3_2_cert -c <new config> <component> |
| Registered 'Release Candidate' build |
etics-submit build --project org.glite --project-config glite_3_2_cert --register --platforms < platform(s) name> --versioneddeps -c <new config> <component> |
| Do a deployment test in ETICS |
|
| Create and lock a subsystem configuration containing the new component(s) |
to be done by the subsystem integrator |
A deployment test will be provided in
ETICS (at the time of writing this document such a test was not yet available in
ETICS). For a given set of packages and gLite node types it first installs the production version of the node type and then updates it with the given packages. The test has to be done for every node type affected by the packages.
Note that even for the case of updating a single component a new subsystem configuration is needed as this currently is the only way to express in
glite_3_2_cert the fact that a certain component is in production.
4.2 How to handle bugs
Bugs are handled via
Savannah [R15] using the
bug tracker. A bug must not stay in status "None" for more than a week. As laid down in the
JRA1 Workplan HowTo [R23] once a bug is accepted it should be immediately assigned to a patch for future release.
When submitting a new bug to Savannah the "Category" field determines the list of persons getting notified about the bug. This list includes the coordinators of the Category as well as other developers associated with it. Ultimately the coordinator should ensure that the bug is being assigned to a developer. Alternatively the bug can be already assigned to a developer by the person submitting the bug. Taking up the bug the developer should decide if the bug can be accepted or if it has to be closed immediately. Justification for immediate closure are: Duplicate, Won't Fix, Invalid and Unreproducible.
If the bug is being accepted it should be put in status "Accepted". When development work for the bug starts the bug should go in status "In Progress". The bug may go directly into "In Progress", bypassing "Accepted", if the developer starts to work on it immediately.
After a fix for the bug has been committed the bug should go to "Integration Candidate" awaiting attachment to a patch. When a corresponding patch (with the bug attached) finally has been created the bug goes to "Ready for test". Now it's the certifier's task to assess if the bug has indeed been fixed. If the bug does not pass certification it is being set back to "None". In this case the bug may be detached from the patch or the patch might be rejected and cloned. If the bug passes certification it is being set to "Fix certified". There are bugs which cannot be checked for being fixed during certification (see
R21 describing criteria for such a situation). In this case the bug is being set to "Fix not certified". Finally the patch is being released to production and the attached bugs then get set to "Ready for review". Now it's the submitter's task to validate that the bug is fixed in production. If it turns out that the bug is still present the submitter sets it back to status "None", otherwise to "Verified" which automatically closes the bug. The time window for the submitter to check if the bug has been fixed is one month. After this period of time, if no complaints have been received, the bug will be set to "Closed" automatically.
Bug states diagram
 |
4.2.1 Bug priorities
The evaluation of the priority of a bug results in one of four
possible values. Each level implies a very specific behaviour for the
management of that bug. The four priority levels and the corresponding
behaviours are:
-
Immediate - The bug needs to be addressed as soon as possible. A release containing fixes to immediate- priority bugs can contain only fixes to immediate-priority bugs. Multiple immediate-priority bugs can be included in the same release, provided that any fix does not delay the release significantly.
-
High - The bug needs to be addressed in the next release. If the next scheduled release is too far in time, it can be anticipated.
-
Medium - The bug can be addressed in any following scheduled release.
-
Low - There is no target date for addressing the bug.
More information is available in the document
Problem Management and Change Management in gLite.
4.2.2 Bugs affecting several platforms
In case a bug is valid for more than one platform, please follow the instructions below:
- only one bug will be opened even if it's a problem that exists in several platforms.
- the bug will be attached to the patch corresponding to the platform where it was detected.
- however, if the bug exists for SL4 and SL5, the SL5 patch will be always used.
- for the other patches that contain the bug fix but where the bug is not attached, a summary of the new features and list of bug fixes will be included in the Release Notes field of the patch, even if the bug was not actually tested on those platforms.
4.2.3 Bugs from Internal Patches
Bugs from Internal Patches could be valid bugs for several services. It's therefore necessary to define the criteria to close them once they are released in Production. Bugs from Internal Patches will be moved into
Ready for Review by the Integration Team taking the following into account:
- When the first Metapackage Patch that has a dependency on an Internal Patch is moved to Production, the corresponding bugs attached to the Internal Patch will be moved into
Ready for Review putting the following comment on the bug:
Bug fix released to Production for the first time on gLite 3.X, Update NN, in service XXX.
- If a bug from an internal patch is only relevant to a certain service, the Integration Team will change the bug status to
Ready for review only when the service is released to Production containing the bug fix.
4.2.4 Bug validity
Here we collect miscellaneous guidance on how to establish if a bug is valid.
- bugs opened against "unclear error messages" as valid bugs and address them wherever possible.
4.3 How to handle patches
Patches are handled via
Savannah [R15] using the
patch tracker.
Patches are our way to apply changes to the gLite release in production. With changes we mean bug fixes as well as new services, new supported platforms and new features. When submitting a patch the different fields in the Savannah patch tracker have to be filled according to the
Product Team Integration guide [R24].
Initially the patch is in state "With Provider" meaning that the developers are editing the fields and adding bugs. The before starting certification the product team should ensure that the patch matches the following acceptance criteria:
- All fields have been filled according to the Product Team Integration guide [R14].
- An ETICS deployment test on the affected production node type has been run with the packages listed in the patch.
- The requirements on documentation as stated in section 3.5.1 "Before certification" in the document Definition and documentation of the revised software life-cycle process [R13] are fulfilled. Note that in the most frequent case of an update of an already existing node type only the updates to the documentation (if there are any updates) must be given in the patch. However for new node types documentation has to be provided accordingly.
If the acceptance criteria are fulfilled and no configuration changes are necessary the patch is set to "Ready for Certification" meaning that a yum repository is available in
ETICS for this patch. If the patch needs configuration changes it is first set to "In configuration". At this point the configuration coordinator has to provide the required configuration changes (typically updates to yaim). If they are in place the patch gets to "Configured" and subsequently to "Ready for Certification". Certification may start now, indicated by the state "In Certification". The patch might be rejected due to the following criteria:
- Some of the packages in the patch have to be changed as they break existing functionality. The functionality is being checked with the tests explained in SA3Testing [R16].
- Some important attached bugs are not fixed by the patch.
Note that if bugs are detached from the patch this does not necessarily lead to a rejection of the patch. If a patch gets rejected a clone is being produced, i.e. a new patch with the same content (including attached bugs and comments). This frees the developer from the burden of re-filling all the fields. Only new information has to be added (new packages etc.). The cloned patch starts again from status "With Provider". Patch cloning can be done easily with the
SavannahCommandLineInterface [R17]. The clone should be produced by the original submitter of the patch as the clone has as "submitter" the user who executed the cloning tool.
Patches may depend on other patches for certification. If this is necessary patch dependencies have to be added in Savannah. We note however that patch dependencies should be avoided. Dependencies between patches make the certification process more difficult and may ultimately delay the release to production.
After a patch has been certified, the integration team checks the test report meets the acceptance criteria (
https://twiki.cern.ch/twiki/bin/view/Main/HowToCertify [R26]) and if everything is correct considers the patch is verified. If a test is missing or a mandatory test has failed, the patch is assigned back "With provider". Before the final release to production a patch goes through staged rollout, a beta test where the patch is being installed at some production sites. Patches scheduled for the next staged rollout phase will be set to "Ready for Roll-out". While being in staged roll out the stage is "Rolling-out" whereas "Ready for Production" means that the patch passed the beta test and can now be released to production. The rpms are signed (only for 3.2) when they are prepared for Staged roll-out.
Patch states diagram
 |
4.3.1 Release Notes
Release notes must be prepared for every new release of a gLite product. The text should be prepared by the Product Team responsible for the release and is included in the corresponding patch in the
Release notes field.
They describe in plain English the contents of the release according to the following structure:
- What's new: Brief description of the main changes, both new features and bug fixes. This part will also be included in the summary page for a certain gLite update.
- Installation and configuration (if configuration): More details on installation and configuration, if the standard messages "Service must be reconfigured" and "Service must be restarted" are not enough.
- Know issues (if applicable): Known issues present in the release, possibly with a workaround.
- Dependencies (if applicable and only for metapackage releases): List of internal releases the metapackage release depens upon, including a reference to the patches (id, title, link) and relevant excerpts from the respective release notes.
For metapackage releases the text will be uploaded by the Integration Team to the gLite web site when the release is ready, without any significant processing apart from a conversion from text to html (using the txt2html utility).
The release notes on the gLite web site are followed by the list of bugs (id, title, link) attached to the metapackage patch.
Release notes are not meant to replace documentation.
Appendix A. Environment
This chapter describes the characteristics of the environment in which gLite daemons and services will run. This comprises the environmental variables the services have to use, guidelines for running and configuring daemons and services, and resource usage. As much as possible the standard locations for configuration files, start-up scripts etc. should be used as well as the standard methods to start/stop and configure services. Adherence to for example the unix/linux standards described in
http://www.pathname.com/fhs/ and
http://www.linuxbase.org/, simplifies the deployment of individual components and subsystems outside the gLite scope.
Much of the contents of this chapter is taken from the
EDG Developers’ Guide [R3].
A.1 Enviromental variables
It must be possible to install and configure the gLite software suite as a non-privileged user and the whole tree should be relocatable. This means that as much as possible software should be installed in the gLite root, which can be specified by setting the environment variable
$GLITE_LOCATION. The software should rely for its installation and configuration on this variable.The gLite software services shall source/parse a top level gLite configuration file and shall not rely on this being previously done (i.e. never assume that variables defined in this file are already present in the environment). The location of this file is platform dependent. For example on RedHat systems it would be
/etc/sysconfig/glite, on Debian systems
/etc/defaults/glite. This platform independent configuration for gLite services should be stored in
/etc/glite.conf , which can be overridden by
$HOME/.glite.conf.
This system dependent top level configuration file will have a key-value based syntax, containing all the needed basic prefixes with the following format:
GLITE_LOCATION=/opt/glite
GLITE_LOCATION_VAR=/var/opt/glite
GLITE_TMP=/tmp
GLITE_HOST_KEY=/etc/grid-security/hostkey.pem
GLITE_HOST_CERT=/etc/grid-security/hostcert.pem
The values should only be literal values. The typical values are given above with the exception of
$GLITE_LOCATION_VAR which has a default of
$GLITE_LOCATION/var.
For
$GLITE* values not defined in the file, the defaults may be assumed by the daemon.
The variable
$GLITE_LOCATION will point to
/opt/glite by default
The variable
$GLITE_LOCATION_VAR will point to the root of a tree for machine-specific files. Since the
$GLITE_LOCATION tree should be mountable read-only by clients, the var directory for "variable" data should not be located within this tree, but in the top level var directory. The use of this variable will allow the gLite software to have a central complete installation which is shared by many machines.
The variable
$GLITE_TMP will point to the root directory of a world-writable area for temporary files. By default, this will have the value
/tmp. The
$GLITE_TMP area is subject to periodic cleaning by cron scripts and at reboots. Consequently, daemons should not store state information in this area; such information should instead be placed in
$GLITE_LOCATION/.
Packages are free to define their own environmental variables of the form
$GLITE_<package>_<anything>
The value of package should be negotiated with those managing the release. In all cases, these variables must be derived from the standard $GLITE_LOCATION,
$GLITE_LOCATION_VAR, and
GLITE_TMP variables.
The
$GLITE_LOCATION_VAR area should be organized as to contain the following directories:
- etc : machine-specific configuration files
Note that the directory
$GLITE_LOCATION_VAR/etc is for machine specific configuration files only. Configuration files that are machine independent should go in the directory
$GLITE_LOCATION/etc.
More information on what should be contained in these directories can be found in the sections below.
A.2 Daemons and Services
Most of the grid services are intended to run continuously while the machine is up to service user requests. As such, a permanently running daemon controlled via an init.d script is appropriate rather than using inetd services.
Daemons should not run as root. If this is unavoidable, then the daemon should be written with a minimal kernel which runs as root. This daemon kernel can then either interact with another non-root daemon or spawn non-root processes to handle the bulk of the processing.
All initialization of a daemon should be done with the
init.d script which controls the daemon. The
init.d script should minimally support the start, stop, status, and restart methods. The init.d scripts are by default located in
/etc/init.d, but it must be possible during the build to configure this location to a directory writable by non-privileged user, e.g.
$GLITE_LOCATION/etc/init.d. It is advised to have a separate configuration package for each component and for each platform to maintain relocatability and flexibility.
Log files should be written to the standard area
/var/log/glite and be written in such a way that the logs can be rotated. Each log entry must comprise the time stamp and the process identifier generating it. All authentication and authorization logging should by default be diverted to a separate secure log and such a way that it will never enter the normal logs. Such a secure logging target could be a Trusted Timestamping Service (TTS) or a remote secure syslog service.
Below is an example of a log entry generated by the process 8876 on 12 July 2004 at 3:44:59 p.m.:
2004/07/12 15:44:59 [8876] ERROR>
rotator.java:14 rotate() – [APPLICATION_FAULT] RotationService - No rotation parameter
This format allows for easy concatenation of logfiles and sorting by time stamp. Please note that the length of the time stamp is fixed.
It is foreseen that services will use common gLite logging facilities, which are based on industry-standard logging tools, such as log4j, log4perl, log4Cpp and log4c. The logging tools will be customized as described in
https://edms.cern.ch/document/486630 [R10].
A.2.1 Daemon/Service credentials
The daemon should use the X509 credentials that are given by the environment variables
$GLITE_HOST_CERT and
$GLITE_HOST_KEY . If these are not set the services should respect the default SSL settings, e.g. by using
X509_get_default_cert_dir_env() and
X509_get_default_cert_file_env(). These locations can be set by
$SSL_CERT_DIR and
$SSL_CERT_FILE, respectively.
A service should check the user credentials and manage its local resources accordingly. However, if the service interacts with another service even a remote instance of the same service, then the user's (delegated) proxy must be used. This is the only way to allow the remote site to verify the credentials itself and then to correctly log transactions and quotas.
When a daemon runs as a service user, host/service credentials should be made available for that user. The current practice is to make a copy of /etc/grid-security/host*.pem in the same directory (/etc/grid-security/tomcat*.pem) or into a subdirectory (/etc/grid-security/dpmmgr/dpmmgr*.pem) and change the ownership of the credentials to the service user.
A.2.2 Daemon/Service Configuration
The
init.d scripts must contain the comments necessary to support the chkconfig command under RH Linux.
A full, annotated template configuration file must be provided with the package. This should have a “.template” suffix so that upgrades to the package do not wipe out a working configuration file. Changes in the package which require changes in the configuration file MUST be clearly stated in the package release notes and documentation.
Configuration components must be provided for each service or
init.d sysV script that needs to be reconfigured.
A.2.3 Linux Packages profile.d Scripts
The package profile.d scripts must be installed in the standard gLite tree in
etc/profile.d. These will be executed by a pair of master gLite profile.d scripts. The package profile.d script may assume that the
GLITE_LOCATION* variables have been defined.
The master gLite profile.d scripts will ONLY execute those scripts in the
$GLITE_LOCATION_VAR/etc/profile.d area. The profiles in
$GLITE_LOCATION/etc/profile.d should always take the values in the variables set by the master profile. System administrators who wish to have a separate machine-specific area must make symbolic links to the desired scripts in
$GLITE_LOCATION/etc/profile.d.
There must be no dependencies between the scripts in the
$GLITE_LOCATION/etc/profile.d area, as the order of execution of files in this area is not guaranteed.
Developers must provide both sh and csh versions of profile.d scripts The sh version must be called *.sh and must execute correctly with the sh, bash, and zsh shells.The csh-shell version must be called *.csh and must execute correctly with csh and tcsh shells. This requirement is extremely important otherwise the grid software will place restrictions on what shells the end-user can or cannot use.
On installation, the scripts should have the executable bits set.
A.2.4 Cron Jobs and Scheduled Tasks
Cron jobs are appropriate for administrative tasks which must be carried out periodically. The scripts should be installed in the
etc/cron.d area of the installation tree given by
$GLITE_LOCATION.
To increase the portability of these scripts, use the standard Bourne shell features. In addition, make sure your script sets the appropriate environment it needs to work properly (
PATH,
LD_LIBRARY_PATH, etc.) and doesn't expect it to be set by the caller.
Within, the script should be a brief description of what the script does and a recommendation on how frequently it should be run. These scripts should not require additional customization beyond the standard package configuration.
The execution of the cron jobs should be logged in
/var/log/glite.
A.3 Resource Usage
A.3.1 Temporary Space
Developers should use the
$GLITE_TMP variable to locate temporary disk space local to the machine. This location is guaranteed to be world-writable. If the location is not set,
$TMPDIR should be used. The use of the standard routines and methods to deal with temporary disk space is encouraged, e.g. tmpfile() and
mkstemp() for C/C++ and for java the tmpfile method from
java.lang
In the temporary area, create a subdirectory if your package will generate a large number of files, or there is a reasonable chance of collision with someone else's files. The name of the subdirectory should be descriptive of the package and reduce the chance of collision.
The package is responsible for cleaning up all files, sockets, etc. created by the package during its execution. Preferably this is done when the file is no longer needed but at the very least when the daemon shuts down. Individual files or directories may be left if an error has occurred and the developer feels that the temporary file may help with debugging or tracing the problem.
On worker nodes the jobs must use
$TMPDIR (when set) when looking for temporary space.
A.3.2 Ports
Ports used by a service must be clearly documented. Additionally, the service port(s) must be easily and completely configurable as well as the port ranges for temporary ports. Services should never rely on fixed port numbers. Sites should be able to organize the network as they wish, internal or external connectivity, NAT, firewall etc.
A.3.3 Network Addresses
It should be possible to set the service to advertise some other host alias as the location than the real IP address. For example if the service is running in a box with an internal IP address it should register itself into the information system with the NAT server's address. This is also needed for systems with multiple interfaces.
A.3.4 Logging Formats
A set of logging rules can be found
here [R9].
Appendix B. Copyright and License
The gLite source code is copyrighted by the
Members of the EGEE Collaboration and is licensed under the
Apache License, Version 2.0.
Each file containing source code developed within one of the EGEE projects shall contain the copyright and license attribution available at
EGEEgLiteSoftwareLicense [R11]. The specification of the year should cover all the years the code has been modified (acceptable forms are, for example, "2005, 2006", "2005-2009" and combinations of the two).
If the source code has been mainly developed outside of the EGEE projects, it should contain the following attribution:
This work has been partially funded by the EU Commission (contract INFSO-RI-222667) under the EGEE-III collaboration.
Appendix C. IPV6 Compliance
To produce IPv6 compliant code the programmers should first use standard APIs that follow RFC 3493, 3542, 4038 recommendations. There are libraries offering compatible interfaces for many languages including C, C++, Java, Python and Perl. Programmers should stay at high levels of abstraction as far as possible. For example use DNS names instead of numeric address and take IPv6 programming guide into consideration. The main problem in porting applications to use IPv6 is to deal with different operating systems and their possible configurations and to be able to offer the best connectivity, i.e. to answer both to IPv4 and IPv6 requests if possible. The gLite deployment platform is mainly Scientific Linux 4 or 5. Scientific Linux 4 and 5 are dual stack nodes where the deployment platform will be configured to enable IPv6 or IPv4. Depending on the operating system features available (separate stack, dual stack, IPv6BindOnly) and language used by the application there are two main options for a server application:
- wait for request on both IPv6 and IPv4 interface with two sockets
- wait for request only on IPv6 interface and use IPv4 mapped IPv6 address to redirect IPv4 requests to the IPv6 part of the stack
The first point to consider is the targeted operating system and what features are available on it: dual-stack, separated stack, IPv4 mapped IPv6 address, ability for IPv6 socket to listen only IPv6 request (IPV6_ONLY / bindv6only). The second point to consider is the specific socket options that are required by the application. Not all of the socket options that are available in the IPv4 API may be available in the IPv6 API. In that case IPv4 mapped IPv6 addresses might not be suitable. Some languages, like Java, may hide some complexity and take account of the different operating possibilities. Java prefers IPv4 mapped IPv6 address if possible. Unfortunately the mapped addresses need special security consideration that may need to be handled by setting up an appropriate filter on the router or firewall. Furthermore the application itself may have to take the usage of IPv4 mapped IPv6 address into account. In the “Guidelines for IP version independence in GGF specification” the authors recommend using IPv4 mapped IPv6 address in part because the code is easier to develop.
According to the specific needs of gLite and the deployment platform targeted the programmers and deployment staff have to consider this problem to define a strategy. A unique method may not handle all cases in gLite but a guideline on these topics will guide programmers in their choice. It is believed that having a coherent and seamless approach is desirable and will help, particularly during deployment when several different applications may be running on a single node.
A document with guidelines on how to produce ipv6 compliant code for the GRID can be found here:
http://www.euchinagrid.org/docs/EUChinaGRID-Del2.2v3-1.pdf
A document explaining how to test IPV6 compliance for your application can be found here:
https://edms.cern.ch/document/930868/1 [R12].
-- Maria Alandes - 16 Feb 2010
jpg 
Copyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.