DevelopersGuide < EGEE

EGEE Web>EGEEgLite>DataManagement>DevelopersGuide (2011-02-04, MariaALANDESPRADILLO) (raw view)
---+!! gLite Developer's Guide

Applicable to gLite as of version 3.1

December 10, 2009

%TOC%

---++ 1.Introduction

---+++ 1.1 Purpose
This guide is for anyone developing or modifying EGEE Middleware. Aim of the document is to provide developers with a quick reference guide to quickly become familiar with development tools, software integration process and release management. It does not contain material about coding standards and naming schemas. The interested reader is advised to consult the [[https://edms.cern.ch/document/790125/0.4][EGEE II Developer Guide]] [R2] where these topics have been treated.

---+++ 1.2 Document Organization
This guide should be considered as a live document. The guide is written in a wiki to allow faster and easier changes, contributions and document management.   [[https://twiki.cern.ch/twiki/bin/view/Atlas/TwikiToPDFAddOn][Twiki2Pdf Add-On]] will be used to provide a pdf version as required for official documents by [[http://project-egee-iii-na1-qa.web.cern.ch/project%2Degee%2Diii%2Dna1%2Dqa/EGEE-III/Procedures/DocManagmtProcedure/DocMngmt.htm][EGEE Document Management Procedure]] [R1]. The document contains two appendices with technical information about the usage of environment variables, daemons and services in gLite as well as a statements on the license used by gLite and IPV6 compliance.

The URL for this document is [[EGEE.DevelopersGuide][https://twiki.cern.ch/twiki/bin/view/EGEE/DevelopersGuide]].

---+++ 1.3 Application area
This document applies to the all software process, including  development, integration and testing activities, seen from the point of view of developers.


In several sections ETICS [R7] command line commands are used. At the time of writing this document the new ETICS client was not yet available. Thus ETICS commands given in this document are subject to change. 
---+++ 1.4 References 

| R1 | EGEE Document Management Procedure <br> [[http://project-egee-iii-na1-qa.web.cern.ch/project-EGEE-III-NA1-QA/EGEE-III/Procedures/DocManagmtProcedure/DocMngmt.htm]]|
| R2 | EGEE II Developer Guide [[https://edms.cern.ch/document/790125/0.4][https://edms.cern.ch/document/790125/0.4]] |
| R3 | EDG Developers&#8217; Guide [[https://edms.cern.ch/document/358824]] |
| R4 | Central CVS Service - HOW TO [[http://cvs.web.cern.ch/cvs/howto.php]] |
| R5 | CERN Account [[https://cernaccount.web.cern.ch/cernaccount/Default.aspx]] |
| R6 | Wiki home of ETICS: The Grid Quality Process [[https://twiki.cern.ch/twiki/bin/view/ETICS/WebHome]] |
| R7 | ETICS Web Page [[http://etics.web.cern.ch/etics/]] |
| R8 | ETICS User Guide [[https://edms.cern.ch/file/795312//ETICS-User_Manual-latest.pdf]] |
| R9 | Recommendations (and a few rules) for logging in JRA1 Code [[https://twiki.cern.ch/twiki/pub/EGEE/EGEEgLite/logging.html]] |
| R10 | A Proposal To Standardize Configuration, Logging and Error Handling [[https://edms.cern.ch/document/486630]] |
| R11 | Software License [[https://twiki.cern.ch/twiki/bin/view/EGEE/EGEEgLiteSoftwareLicense]] |
| R12 | How to test IPV6 compliance [[https://edms.cern.ch/document/930868/1]] |
| R13 | MSA3.4.1 Definition and documentation of the revised software life-cycle process [[https://edms.cern.ch/document/973115]] |
| R14 | Guidelines on how to fill a patch [[https://twiki.cern.ch/twiki/bin/view/EGEE/HowToFillAPatch]] (deprecated. See R24) |
| R15 | gLite Middleware Savannah group [[https://savannah.cern.ch/projects/jra1mdw/]] |
| R16 | Service certification checklist [[https://twiki.cern.ch/twiki/bin/view/EGEE/SA3testing]] |
| R17 | Savannah Command Line Interface [[https://twiki.cern.ch/twiki/bin/view/EGEE/SavannahCommandLineInterface]] |
| R18 | Documentation for accessing CERN CVS [[https://lcgcvs.web.cern.ch/lcgcvs/howto.php#accessing]] |
| R19 | IPV6 compliant code [[http://www.euchinagrid.org/docs/EUChinaGRID-Del2.2v3-1.pdf]] |
| R20 | [[http://egee-pre-production-service.web.cern.ch/egee-pre-production-service/index.php?dir=./rollout/][Staged Roll-Out service]] |
| R21 | Bug non verification criteria [[https://twiki.cern.ch/twiki/bin/view/EGEE/BugNonVerification][https://twiki.cern.ch/twiki/bin/view/EGEE/BugNonVerification]] |
| R22 | CERN E-Groups [[https://groups.cern.ch/Pages/default.aspx][https://groups.cern.ch/Pages/default.aspx]] |
| R23 | JRA1 Workplan !HowTo [[https://twiki.cern.ch/twiki/bin/view/EGEE/Egee3Jra1IWorkplanHowTo]] |
| R24 | Product Team Integration Guide [[https://twiki.cern.ch/twiki/bin/view/EGEE/ProdInt]] |
| R25 | EGEE-III gLite Product Team Release Process [[https://edms.cern.ch/document/1041827/1]] |
| R26 | How to certify a patch [[https://twiki.cern.ch/twiki/bin/view/Main/HowToCertify][https://twiki.cern.ch/twiki/bin/view/Main/HowToCertify]] |

---+++ 1.5 Feedback
For feedback or questions on this guide please contact Maria Alandes at CERN. Her contact details can be found in the CERN directory [[http://consult.cern.ch/xwho]].

---+++ 1.6 Mailing lists
Developers are encouraged to subscribe to the EMT mailing list =project-eu-egee-middleware-emt= to follow discussions and announcements. Subscribe using [[https://groups.cern.ch/Pages/default.aspx][CERN E-Groups]] [R22].

---+++ 1.7 Weekly meetings (telephone conference)
Two telephone conferences are held on Mondays at 14h30 and on Wednesdays at 16h30 every week. The agenda and the call for the meeting is announced in the EMT mailing list. Agendas and minutes of previous meetings can be found in Indico category [[http://indico.cern.ch/categoryDisplay.py?categId=1146][Home > Departments > IT > GT > EMT]]. Developers are encouraged to participate in this meeting. They can propose discussion topics by sending them to the EMT mailing list. The meeting is chaired by the release manager and attended by product team representatives. Some other people participating regularly in the meetings are ETICS representatives, JRA1 section leader, SA1 representatives and LCG representatives.

The EMT also performs a bug triage, making an objective assessment of a bug's *priority*. Once a week the EMT will

   * Check all bugs of severity (which is set by the submitter) major & critical submitted in the last 7 days
   * Assess their severity and consequent priority
   * Update the bug's priority, with a comment to make it clear that the priority has been set by the EMT

Typically the severity will not be changed as it records the submitter's assessment

Background and assessment criteria can be found in [[https://edms.cern.ch/document/101991][Problem Management and Change Management in gLite]]

---++ 2. Software Configuration and Version Control

---+++ 2.1 Overview 
CVS (Concurrent Versions System (CVS), also known as Concurrent Versioning System) and ETICS (eInfrastructure for Testing, Integration and Configuration of Software) are used to manage software complexity and to improve the software quality.
CVS is used to allow several developers to collaborate on a software module development in a distributed (in space and time) environment. CVS also keeps track of all work and all changes in the source code.
ETICS is used to 
   * build software modules managing complex dependencies and software integration with configuration concept
   * build software modules on different platforms
   * manage software configuration versioning
   * manage software release process
---+++ 2.2 Repository
The gLite CVS repository is hosted by the [[http://cvs.web.cern.ch/cvs/][CERN Central CVS Service]] [R4]. It can be accessed via the generic =CVSROOT=:
<verbatim>
glite.cvs.cern.ch:/cvs/glite
</verbatim>
For up to date information on how to access the CERN CVS have a look at the [[http://cvs.web.cern.ch/cvs/][CERN Central CVS Service]] [R4] webpage.

---+++ 2.3 Read Only Access gLite CVS Repository
The CVS repository can be accessed, in read only mode, two ways:
   * Direct access with [[http://glite.cvs.cern.ch/cgi-bin/glite.cgi/][http://glite.cvs.cern.ch]]
   * with anonymous connection
<verbatim>
    export CVSROOT=:pserver:anonymous@glite.cvs.cern.ch:/cvs/glite
</verbatim>
The full documentation for accessing CERN CVS can be found on the [[https://lcgcvs.web.cern.ch/lcgcvs/howto.php#accessing][CERN CVS page]] [R18]. 

---+++ 2.4 Authentication: Read/Write Access
In order to have CVS write access, it is necessary to have a CERN AFS account. Instructions to request an account are available on the [[https://cernaccount.web.cern.ch/cernaccount/Default.aspx][CERN account webpage]] [R5].

---++++ 2.4.1 Read/Write Access using [[http://cvs.web.cern.ch/cvs/howto.php#kserver-out][Kerberos V]]

<verbatim>
    export CVSROOT=':gserver:glite.cvs.cern.ch:/cvs/glite'
</verbatim>

---++++ 2.4.2 Read/Write Access using SSH

<verbatim>
    export CVSROOT=:ext:<afs-user-name>@glite.cvs.cern.ch:/cvs/glite
    export CVS_RSH=ssh
</verbatim>

To set up SSH, follow the instructions for [[http://product-support.web.cern.ch/product-support/UI/Docs/CVS/howto.html#accessing-sshlinux][Configuring SSH access from Linux/Unix]] on the CERN CVS page.

---+++ 2.5 Components, Subsystems and CVS Directories

A component is a CVS directory that contains the source code for a given functionality and a subsystem is a logical group of components. The name of CVS directory for a component should be in the form of _org.glite.subsystem.component_. Dots should not be used in the subsystem or component name. Each component should create a package where the package name is in the format _glite-subsystem-component_.  Anyone with write access to CVS should be able to create a components directory in CVS. After creating this directory you will need to obtain write access by emailing a request to [[mailto:project-eu-egee-middleware-integration@cern.ch][middleware-integration]]. The CVS directory for the component should contain the following directories. 
<verbatim>
   src: contains the source code
   doc: contains the documentation
   etc: configuration files
   tests: contains test material
   build: not need in CVS but should be use for building the code. 
</verbatim>

---+++ 2.6 CVS tags

When a component is ready to be released, the CVS componet directory should be tagged. The format of the tag should be _glite-subsystem-component_R_x_y_z_w_  where x, y, z and w are the major, minor, revision and age numbers, respectively, for the release.

---+++ 2.7 Mail notification

Developers are encouraged to subscribe to mailing lists that provide an automatic notification on every CVS commit for the component of interest. Currently mailing lists for the following components are available: build.common-cpp, testing, ce, data, datamgt (DPM, LCG-DM), dgas, lb (jp), sd (service-discovery), security, wms-utils, wms-ui, wms, yaim. For subscription to these mailing lists go to the [[https://groups.cern.ch/Pages/default.aspx][CERN E-Groups]] [R22] page and search for groups starting with the string =project-eu-egee-cvs=.

Every developement team must have resp. set up a mailing list providing an automatic notification no matter which revision control system is being used. 

---+++ 2.8. Access request

If you need to have write access to a certain CVS module, please send a mail to the EMT mailing list specifying the cvs module(s) name(s) and the CERN AFS login of the user(s) who need to have access.

---++ 3. The build system
[[http://etics.web.cern.ch/etics/][ETICS]] [R7] is the build system used by EGEE. The main goal of ETICS is to handle every aspect of the software development workflow, i.e. build, release and testing, in such a way that every operation has to be completely independent from the platforms, the languages and the tools involved in the process. The content of an ETICS site is structured as a set of hierarchies of entities, with one hierarchy for each project registered into the service; an entity can be a subsystem of the project or a component of a given subsystem. Every entity groups different configurations; a configuration contains all the information and instructions required for building, packaging and testing the software on one or more platforms. In the following it is referred to the set of information and instructions of a configuration as metadata. All gLite components and subsystems should be in the project gLite Middleware project. Interaction with ETICS can be done via the [[https://etics.cern.ch/eticsPortal/][web interface]] [R7] or command line tools. For more in-depth details and tutorials on ETICS please see the [[https://twiki.cern.ch/twiki/bin/view/ETICS/WebHome][Etics documentation]] [R6].

Note that it is difficult to build the middleware without the use of ETICS. 

---+++ 3.1 ETICS and user permissions
#UserPerm Any user must register to ETICS in order to have the correct set of permissions granted to its role. The registration can be performed at the following page:
[[https://etics.cern.ch:8443/eticsAdmin/public/registration/requestRegistration.jsp][https://etics.cern.ch:8443/eticsAdmin/public/registration/requestRegistration.jsp]]

Since the authentication mechanism of ETICS is based upon X509 public key infrastructure (PKI) before sending the request for registration the user must be sure that his credentials, the private key and the related certificate, have been correctly installed into the web browser.
ETICS grants access even to unregistered users, providing a special "guest" account in this case; the system allows those users to perform just read-only operations.

ETICS provides a fine-grained permission management; rights can be set up for the entire project or for each subsystem or component and also for a given configuration of subsystem or component. For further details about the permission management refer to:
[[http://support.eticsproject.eu/index.php?view=article&catid=2%3Atutorial&id=80%3Aundertanding-security-in-etics&option=com_content&Itemid=5][Understanding Security in ETICS]]
---+++ 3.2 Installation, configuration and set up of the ETICS command line client
#CliSetup The steps required for the installation of the ETICS client are:
   $ Download the Etics workspace setup script : =wget  "http://eticssoft.web.cern.ch/eticssoft/repository/etics-client-setup.py -O etics-client-setup=
   $ Run the Etics workspace setup script : =python etics-client-setup=
   $ Set the ETICS_HOME env variable : =export ETICS_HOME=`pwd`/etics=
   $ Add the Etics commands to your PATH: =export PATH=$PATH:$ETICS_HOME/bin=

All the configuration parameters for the ETICS client are defined into a configuration file whose default location is $HOME/.etics.conf.
Most of the parameters listed into the configuration file can be also redefined in the command line.
The basic properties for the account setting are:
   $ the path of the user certificate: =x509_user_cert=
   $ the path of the user private key: =x509_user_key=
they are required in order to establish the secure connection and to authenticate against the ETICS server and are contained into the _user_ section.
The required parameter for the connection is _server_, which is the fully qualified domain name of the service to contact and is contained into the _system_ section, the main service running is located at _etics.cern.ch_.
The ETICS client is usually able to detect the correct platform of the system where it is running on, whever this is not possible the user can force the platform definition with the _platform_ property listed into the generic _properties_ section.
Every ETICS command can work only inside a well formatted workspace, i.e. a directory of the local filesystem which contains the project metadata files. The commands that must be invoked for setting up the workspace are:
   $ for the initialization: =etics-workspace-setup=
   $ for downloading the project metadata: =etics-get-project org.glite=
---+++ 3.3 Creation of an ETICS entity
#EntityCreation The creation of a subsystem is carried out by the project administrator; the creation of a component is carried out by the project administrator or by the module administrator of the subsystem that component belongs to. The suitable command for this operation is _etics-module_:
   $ add a subsystem to the project: =etics-module add --parent org.glite --subsystem subsystem=
   $ add a component to a subsystem:  =etics-module add --parent subsystem --component component=
By default the system creates a configuration for both the subsystem and the component whose name is _org.glite.subsystem.HEAD_ or _org.glite.subsystem.component.HEAD_ respectively.

The naming convention for subsystem and components should follow the same rules defined for the CVS module name, discussed in section 2
   $ the subsystem name should be: =org.glite.subsystem_name=
   $ the component name should be: =org.glite.subsystem_name.component_name=
There's no relation between the name of the entity and the name of the CVS module whom the entity refers to.

Every operation on a subsystem or component configuration can be performed using the command _etics-configuration_:
   $ add a configuration to a component: =etics-configuration add -c component_config -i input_file component=
   $ modify metadata of a configuration: =etics-configuration modify -c component_config -i input_file component=
   $ clone a configuration : =etics-configuration clone -c target-configuration source-configuration component=

   $ list the existing configuration for a component. : =etics-list-configuration component=

Most of the commands described above requires that a set of metadata must be defined into a file. This file, called in the following _"ini-file"_, is divided into sections and must contain 

   * the main definition for a configuration:
<verbatim>
[Configuration-glite-mysubsystem-mycomponent_R_1_0_0_0]
moduleName = org.glite.mysubsystem.mycomponent
displayName = Brief description of mycomponent
description = Full description of mycomponent
projectName = org.glite
age = 0
tag = glite-mysubsystem-mycomponent_R_1_0_0_0
version = 1.0.0
path = ${projectName}/${moduleName}/${version}/noarch/
</verbatim>

   * the section that defines the CVS commands for the _default_ platform:
<verbatim>
[Platform-default:VcsCommand]
displayName = Brief description of the CVS commands of mycomponent
description = Full description of the CVS commands of mycomponent
tag = cvs -d ${vcsroot} tag -R ${tag} ${moduleName}
checkout = cvs -d ${vcsroot} co -r ${tag} -P ${moduleName}
</verbatim>

   * the section that defines the build commands for the _default_ platform:
<verbatim>
[Platform-default:BuildCommand]
postpublish = None
packaging = None
displayName = Brief description of the build commands of mycomponent
description = Full description of the build commands of mycomponent
doc = None
prepublish = None
publish = None
compile = None
init = None
install = None
clean = None
test = None
configure = None
</verbatim>
for further details concerning the syntax of an ini-file refer to the [[https://edms.cern.ch/file/795312//ETICS-User_Manual-latest.pdf][ETICS user guide]] [R8].

---+++ 3.4 Dependencies for an ETICS entity and the lock operation
#DepsAndLock One powerful feature of ETICS is the capability of handling the software dependencies, such as libraries and tools, required by a given subsystem or component with a minimum amount of effort from integrators and developers. Any dependency or resource must be registered in the system; in this way any entity can be accessed by a project, any possible combination is allowed, a subsystem can be a dependency for a project or a subsystem contained into it, a component of a project can be a dependency for another component of a different project.

The most important example concerns the project _externals_ which collects all the resources published by external projects. It is strongly recommended to use only the external resource contained into that project in order to exploit all the features of the system during the release step.

According to the way an ETICS entity uses a given dependency the system distinguishes two different criterions:
   * _static_ or _dynamic_ dependency: the former defines a fixed release version for the resource inside an entity configuration, the latter allows the entity to inherit the release version from the parent entities; they're mutually exclusive.
   * _build-time_ or _run-time_ dependency: the former specifies that the resource is required only during the build or testing step, the latter instructs the system in order to have a dependency when the artifact produced is used, typically putting such requirement into the package; the two definitions can be combined together.

When an entity defines a dynamic dependency it delegates the system to select the correct version of the resource, in general the version decided for the entire project. It is strongly recommended to make use of this feature, especially for the external dependencies, if no particular requirements must be considered.

Dynamic dependencies are not able to guarantee that a given configuration for an entity can be reproduced at any time. It is necessary to freeze, or lock, the configuration of that entity together with all the configurations of the sub-entities and dependencies. The _lock_ operation is performed with the command _etics-configuration lock_, as in the following example:
<verbatim>
etics-configuration lock --parent-config glite_3_2_cert     -c component_configuration component_name
</verbatim>

---+++ 3.5 Checking out and building a component
#CheckAndBuild The steps required for downloading and building the code of a component are:
   $ Check out a specific configuration: =etics-checkout --config component_configuration_name --project-config project_configuration_name component_name=
   $ Locally build the code: =etics-build component_name=   

   $ Integrators and developers can schedule a remote build for the configuration of the managed component with the following command: =etics-submit build --config component_configuration_name --platforms slc4_ia32_gcc346,slc4_x86_64_gcc346,sl5_ia32_gcc412,sl5_x86_64_gcc412 component_name=

   $ Also a release build against a locked component can be scheduled: =etics-submit build --config locked_component_configuration_name --platforms slc4_ia32_gcc346,slc4_x86_64_gcc346 --register component_name=

The system can publish the artifacts, such as tarballs or packages (rpm, deb), either into a temporary area, called _"volatile"_ repository, or into the permanent repository. The volatile repository is suitable only for development purposes, the permanent repository must be used for publishing the official artifacts of a release and it can be populated only if the configuration is locked. The results and every reference to artifacts produced by the remote build are contained into the [[https://etics.cern.ch/eticsPortal/#repository][ETICS reports site]] [R7]. 


All the operations above can be applied to an entire subsystem replacing the component name and the configuration with subsystem name and the corresponding configuration.
---+++ 3.6 ETICS and the software release management
#ReleaseManagement For each subsystem and component registered in ETICS it is possible to distinguish two different types of configuration:
   * the ETICS configuration for a branch
   * the ETICS configuration for a release
The former is suitable for development purposes, it contains references to CVS branches or to the HEAD main branch and cannot be locked as any definition can be changed according to the development requirements.
The naming convention for a branch configuration should be as in the following:
   $ for a subsystem: =glite-subsystem_branch_major_minor_revision=
   $ for a component: =glite-subsystem-component_branch_major_minor_revision=
The latter represents a frozen release thus it must be guaranteed it cannot be changed; it contains only references to CVS  tag and must be locked in order to freeze the entire dependency structure and the set of commands, properties and variables.
The naming convention for a frozen release configuration should be the same as the CVS tag, thus it should follow the notation:
   $ for a subsystem: =glite-subsystem_R_major_minor_revision_age=
   $ for a component: =glite-subsystem-component_R_major_minor_revision_age=
If the configuration refers to an internal release, suitable for tests only purpose for example, any notation is allowed but it is strongly recommended to put in the tag the name of the component or the subsystem together with a string summarizing the goal of the release.
The naming convention for a branch configuration states that the configuration name should contain the prefix 
   $ for a subsystem: =glite-subsystem_branch=
   $ for a component: =glite-subsystem-component_branch=
followed by a string that identifies the branch. The only exception allowed is the HEAD main branch for which the ETICS provides the default configuration name _org.glite.subsystem.HEAD_ or _org.glite.subsystem.component.HEAD_.

A release configuration can be obtained from a branch configuration with the ETICS clone operation of the web application or taking the ini-files of the branch configuration as templates for the etics-tag command of the command line client.

---+++ 3.7 How to write !ETICS configurations

This section gives guidelines on how to write !ETICS configurations. As much information as possible should be defined in the project configuration.

Which components should be grouped into subsystems?

   * A subsystem is a set of components with significant relationship at development/build time
   * Two components go in the same subsystem if and only if they need to be developed/built together
   * A single component should be buildable without the need to build the whole subsystem. One must thus avoid to define information (e.g. properties) on the subsystem level


*Component Configurations*

   * Specify only dynamic dependencies in the Default platform for direct dependencies (i.e. something of that component is used)
   * Do not override properties
   * Do not specify .DEFAULT properties
   * Should only have the default platform attached. Exceptions have to be communicated
   * About =globus= and =vdt=: Please, use =globus= package for build time only dependencies and =vdt_globus_essentials= package for run time only dependencies.

*Subsystem Configurations*

   * Are empty apart from the list of component configurations

*Project Configurations*

   * Specify .DEFAULT properties per platform

---++ 4. Release Workflow
---+++ 4.1 Understanding the release 
Since the beginning of 2010 a new release process is in place in order to prepare for the transition to EGI, the follow up project to EGEE. An overall description of the new release process can be found in the [[https://edms.cern.ch/document/1041827/1][EGEE-III gLite Product Team Release Process]] [R25] document. The process is based on product teams being entirely responsible for their products (development, testing, integration and release). ETICS is used as a build system as well as a tool to create metapackages. Please refer to the [[https://twiki.cern.ch/twiki/bin/view/EGEE/ProdInt][Product Team Integration Guide]] [R24] for a detailed explanation on how product teams should use ETICS for the release process.

Both ETICS and CVS are arranged on a subsystem basis. To work best, a subsystem should

   * Group together components with synchronised evolution, where updates are frequently correlated.
   * Contain no other unrelated components, as this imposes an artificial connection and prevents independent evolution
Unfortunately, the gLite subsystems as currently defined often break both these rules, as for historical reasons they are arranged along administrative lines. Nevertheless the scenario where updates of a component only affects the other components in the same subsystem can be regarded as the default scenario for gLite integration. 

---++++ ETICS Project configurations

The release team maintains two project configurations in ETICS for both gLite 3.1 and gLite 3.2:

   * glite_3_X_cert
   * glite_branch_3_X_0_dev 

The configuration =glite_branch_3_X_0_dev= is maintained to hold the latest tags of interest to developers, i.e. the configurations currently 'in work' and which are likely to subsequently be locked and used for a release candidate build. Any build related bugs can be considered fixed in this build. This build will not be used to create releasable software but provides an automatic build of the latest tags against new versions of dependencies, enabling anticipation of problems. Note that the tags contained in =glite_branch_3_X_0_dev= will be used to create releasable software, when locked and built against =glite_3_X_cert=. Developers can always build any configuration against =glite_3_X_cert= in the usual way.

When a release is required, the 'in work' configuration in =glite_branch_3_X_0_dev= should be built against =glite_3_X_cert= and locked when everything works fine. The next configuration to be 'in work' should be communicated to the integration team so it can be added to =glite_branch_3_X_0_dev=. When the packages are certified and later verified, the appropriate configuration will be transferred by the integration team to =glite_3_X_cert=. For more details on =glite_3_X_cert=, please check [[https://twiki.cern.ch/twiki/bin/view/EGEE/ProdInt][Product Team Integration Guide]].

=glite_3_1_cert= should be used for building gLite 3.1 and =glite_3_2_cert= should be used for building gLite 3.2.

When product team A needs to use a package provided by product team B which is not yet included in =glite_3_X_cert=, product team A should use the available version of the package under =glite_branch_3_X_0_dev=. If the version existing there is giving problems, product team A  should communicate it to the EMT. Product team B should then provide a new configuration. At the same time, product team B should certify the required package as soon as possible so product team A can include it in their release when it's certified and verified. In order to do that, product team B should certify an [[https://twiki.cern.ch/twiki/bin/view/EGEE/ProdInt#Creating_an_Internal_patch_in_Sa][Internal patch]].

---++++ ETICS Project properties

The following properties are defined in =glite_3_2_cert= project configurations since February 2010. For the complete list of properties please check ETICS:

   * gLiteProjectVersion: version of the last locked project configuration. 
   * gLiteCopyrightText: <verbatim>
Copyright (c) Members of the EGEE Collaboration. 2004. 
See http://www.eu-egee.org/partners/ for details on the copyright holders.
</verbatim>
   * gLiteLicenseText: <verbatim>
Licensed under the Apache License, Version 2.0 (the "License"); 
you may not use this file except in compliance with the License. 
You may obtain a copy of the License at 

    http://www.apache.org/licenses/LICENSE-2.0 

Unless required by applicable law or agreed to in writing, software 
distributed under the License is distributed on an "AS IS" BASIS, 
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 
See the License for the specific language governing permissions and 
limitations under the License
</verbatim>

---++++ The standard use case
The following diagram illustrates the use case of a single component being affected. It can be extended to the case of several components within the same subsystem.

|*Action*|*Command*|
|Check out latest configuration with glite_3_2_cert|==etics-checkout --project-config glite_3_2_cert -c &lt;config&gt; &lt;component&gt;==|
|Commit changes|==cvs commit==|
|Tag changes|==cvs tag==|
|Create new local configuration using new CVS tag|==etics-configuration prepare -c &lt;config&gt; &lt;component&gt;<br>vi Configuration.ini<br>etics-configuration add -noautocommit -i Configuration.ini -c &lt;new config&gt; &lt;component&gt;==|
|Build the new configuration locally|==etics-checkout --local -c &lt;new config&gt; &lt;component&gt;<br>etics-build -c &lt;new config&gt; &lt;component&gt;==|
|Test provisional build||
|Create release tag|==cvs tag==|
|Point ETICS to release tag|==vi Configuration.ini<br>etics-configuration modify --noautocommit -i Configuration.ini -c &lt;new config&gt; &lt;component&gt;==|
|Commit ETICS configuration|==etics-commit==|
|Lock configuration|==etics-configuration lock --parent-config glite_3_2_cert -c &lt;new config&gt; &lt;component&gt;==|
|Registered 'Release Candidate' build|==etics-submit build --project org.glite --project-config glite_3_2_cert --register --platforms &lt; platform(s) name&gt; --versioneddeps -c &lt;new config&gt; &lt;component&gt;==|
|Do a deployment test in ETICS| |
|Create and lock a subsystem configuration containing the new component(s)| to be done by the subsystem integrator |

A deployment test will be provided in ETICS (at the time of writing this document such a test was not yet available in ETICS). For a given set of packages and gLite node types it first installs the production version of the node type and then updates it with the given packages. The test has to be done for every node type affected by the packages.

Note that even for the case of updating a single component a new subsystem configuration is needed as this currently is the only way to express in =glite_3_2_cert= the fact that a certain component is in production. 

---+++ 4.2 How to handle release tasks

The purpose of using tasks is to provide a unique place, the [[http://bit.ly/22we3i][Release Schedule]], where all the information on the most significant activity performed by gLite developers is collected. A release task is typically justified by a one or more requirements coming from the users, especially WLCG. A release task can also be opened independently by a product team, for example when the corresponding development concerns an improvement anticipating issues in scalability, reliability, performance, etc.

A task should be created only for metapackage releases. Internal patches do not need to be tracked.

The [[http://bit.ly/22we3i][Release Schedule]] contains a set of release tasks describing the upcoming metapackage releases. All releases, scheduled and unscheduled (emergency), major, minor and revisions, should appear in the Release Schedule. This allows to have a complete history of the releases.

When opening a release task:

   * The summary should have the form _[name of the component] release x.y.z-w_. 

   * The _Should be finished on_ date is an estimate of when the corresponding release will enter certification. A month resolution is sufficient. If necessary, the date can be changed.

   * A corresponding Savannah patch should be opened at the same time and linked to the task with a dependency _task -> patch_. The summary of the patch should have the same value of the task summary, possibly extended with platform information. Multiple patches, e.g. one per platform, can be attached to a release task. The existence of the patch very early in the process allows to attach a bug to a patch as soon as the bug is accepted.

   * The comment in _Original Submission_ should provide a short description of the main changes in the release. 

*A patch entering certification should always be associated to a release task*. For scheduled releases, the task should be opened well in advance. For unscheduled (emergency) releases, the task can be opened at the last minute.

When a patch is =Certified=, the _Status_ of the corresponding task is set to =Done= by the product team, but the _Open/Close_ field is left =Open=. The Integration Team is responsible for closing the task after moving the associated patch =In production=.

If the patch is rejected or sent back to the developers (in _Status_ =With Provider=) the release task is set back to =Assigned=.

---++++ 4.2.1 Versioning

Every release should be versioned according to the usual _major.minor.revision-age_ schema:

   * a *major* release of a component is characterized by a well-defined interface and behaviour;

   * within a major release, a *minor* release of a component can introduce changes in interface or behaviour in a strictly backwards-compatible way;

   * within a minor release, a *revision* release of a component introduces only bug fixes;

   * a change in *age* means a simple re-build of the component, without any changes, neither in code nor in configuration.


---+++ 4.3 How to handle bugs

Bugs are handled via [[https://savannah.cern.ch/projects/jra1mdw/][Savannah]] [R15] using the [[https://savannah.cern.ch/bugs/index.php?group=jra1mdw][bug tracker]]. A bug must not stay in status "None" for more than a week. As laid down in the [[https://twiki.cern.ch/twiki/bin/view/EGEE/Egee3Jra1IWorkplanHowTo][JRA1 Workplan HowTo]] [R23] once a bug is accepted it should be immediately assigned to a patch for future release.

When submitting a new bug to Savannah the "Category" field determines the list of persons getting notified about the bug. This list includes the coordinators of the Category as well as other developers associated with it. Ultimately the coordinator should ensure that the bug is being assigned to a developer. Alternatively the bug can be already assigned to a developer by the person submitting the bug. Taking up the bug the developer should decide if the bug can be accepted or if it has to be closed immediately. Justification for immediate closure are: Duplicate, Won't Fix, Invalid and Unreproducible.

If the bug is being accepted it should be put in status "Accepted". When development work for the bug starts the bug should go in status "In Progress". The bug may go directly into "In Progress", bypassing "Accepted", if the developer starts to work on it immediately.

After a fix for the bug has been committed the bug should go to "Integration Candidate" awaiting attachment to a patch. When a corresponding patch (with the bug attached) finally has been created the bug goes to "Ready for test". Now it's the certifier's task to assess if the bug has indeed been fixed. If the bug does not pass certification it is being set back to "None". In this case the bug may be detached from the patch or the patch might be rejected and cloned. If the bug passes certification it is being set to "Fix certified". There are bugs which cannot be checked for being fixed during certification (see [[https://twiki.cern.ch/twiki/bin/view/EGEE/BugNonVerification][R21]] describing criteria for such a situation). In this case the bug is being set to "Fix not certified". Finally the patch is being released to production and the attached bugs then get set to "Ready for review". Now it's the submitter's task to validate that the bug is fixed in production. If it turns out that the bug is still present the submitter sets it back to status "None", otherwise to "Verified" which automatically closes the bug. The time window for the submitter to check if the bug has been fixed is one month. After this period of time, if no complaints have been received, the bug will be set to "Closed" automatically.

  
<table align="center">
<caption align="bottom">Bug states diagram</caption>
<tr><td>[[https://twiki.cern.ch/twiki/pub/EGEE/DevelopersGuide/BugStates_twiki.jpg]]</td></tr>
</table>

---++++ 4.3.1 Bug priorities

The evaluation of the priority of a bug results in one of four
possible values. Each level implies a very specific behaviour for the
management of that bug. The four priority levels and the corresponding
behaviours are:

   $ ==Immediate==: The bug needs to be addressed as soon as possible. A release
     containing fixes to immediate- priority bugs can contain only fixes
     to immediate-priority bugs. Multiple immediate-priority bugs can be
     included in the same release, provided that any fix does not delay
     the release significantly.

   $ ==High==: The bug needs to be addressed in the next release. If the next
      scheduled release is too far in time, it can be anticipated.

   $ ==Medium==: The bug can be addressed in any following scheduled release.

   $ ==Low==: There is no target date for addressing the bug.

More information is available in the document [[https://edms.cern.ch/document/1019911][Problem Management and Change Management in gLite]].

---++++ 4.3.2 Bugs affecting several platforms

In case a bug is valid for more than one platform, please follow the instructions below:

   * only one bug will be opened even if it's a problem that exists in several platforms.
   * the bug will be attached to the patch corresponding to the platform where it was detected.
   * however, if the bug exists for SL4 and SL5, the SL5 patch will be always used.
   * for the other patches that contain the bug fix but where the bug is not attached, a summary of the new features and list of bug fixes will be included in the Release Notes field of the patch, even if the bug was not actually tested on those platforms.

---++++ 4.3.3 Bugs from Internal Patches

Bugs from Internal Patches could be valid bugs for several services. It's therefore necessary to define the criteria to close them once they are released in Production. Bugs from Internal Patches will be moved into =Ready for Review= by the Integration Team taking the following into account:

   * When the first Metapackage Patch that has a dependency on an Internal Patch is moved to Production, the corresponding bugs attached to the Internal Patch will be moved into =Ready for Review= putting the following comment on the bug:

<verbatim>
         Bug fix released to Production for the first time on gLite 3.X, Update NN, in service XXX.
</verbatim>

   * If a bug from an internal patch is only relevant to a certain service, the Integration Team will change the bug status to =Ready for review= only when the service is released to Production containing the bug fix.

---++++ 4.3.4 Bug validity

Here we collect miscellaneous guidance on how to establish if a bug is valid. 

   * bugs opened against "unclear error messages" as valid bugs and address them wherever possible.

---+++ 4.4 How to handle patches
Patches are handled via [[https://savannah.cern.ch/projects/jra1mdw/][Savannah]]  [R15] using the [[https://savannah.cern.ch/patch/?group=jra1mdw][patch tracker]].

Patches are our way to apply changes to the gLite release in production. With changes we mean bug fixes as well as new services, new supported platforms and new features. When submitting a patch the different fields in the Savannah patch tracker have to be filled according to the [[https://twiki.cern.ch/twiki/bin/view/EGEE/ProdInt][Product Team Integration guide]] [R24].

Product teams have to make sure that patches are attached to the corresponding task according to the guidelines explained in the section [[https://twiki.cern.ch/twiki/bin/view/EGEE/DevelopersGuide#4_2_How_to_handle_release_tasks][How to handle release tasks]]. A product team can open a patch as soon as a release is planned and follow the discussions at the EMT to keep track of other internal or metapackage patches relevant to the release they are preparing. In this way, they can attach those patches as soon as they are certified.

Initially the patch is in state "With Provider" meaning that the developers are editing the fields and adding bugs. The before starting certification the product team should ensure that the patch matches the following acceptance criteria:

   * All fields have been filled according to the  [[https://twiki.cern.ch/twiki/bin/view/EGEE/ProdInt][Product Team Integration guide]] [R14].
   * An ETICS deployment test on the affected production node type has been run with the packages listed in the patch.
   * The requirements on documentation as stated in section 3.5.1 "Before certification" in the document [[https://edms.cern.ch/document/973115][ Definition and documentation of the revised software life-cycle process]] [R13] are fulfilled. Note that in the most frequent case of an update of an already existing node type only the updates to the documentation (if there are any updates) must be given in the patch. However for new node types documentation has to be provided accordingly. 

If the acceptance criteria are fulfilled and no configuration changes are necessary the patch is set to "Ready for Certification" meaning that a yum repository is available in ETICS for this patch. If the patch needs configuration changes it is first set to "In configuration". At this point the configuration coordinator has to provide the required configuration changes (typically updates to yaim). If they are in place the patch gets to "Configured" and subsequently to "Ready for Certification". Certification may start now, indicated by the state "In Certification". The patch might be rejected due to the following criteria:

   * Some of the packages in the patch have to be changed as they break existing functionality. The functionality is being checked with the tests explained in [[SA3Testing][SA3Testing]] [R16].
   * Some important attached bugs are not fixed by the patch.

Note that if bugs are detached from the patch this does not necessarily lead to a rejection of the patch. If a patch gets rejected a clone is being produced, i.e. a new patch with the same content (including attached bugs and comments). This frees the developer from the burden of re-filling all the fields. Only new information has to be added (new packages etc.). The cloned patch starts again from status "With Provider". Patch cloning can be done easily with the [[SavannahCommandLineInterface][SavannahCommandLineInterface]] [R17]. The clone should be produced by the original submitter of the patch as the clone has as "submitter" the user who executed the cloning tool.

Patches may depend on other patches for certification. If this is necessary patch dependencies have to be added in Savannah. We note however that patch dependencies should be avoided. Dependencies between patches make the certification process more difficult and may ultimately delay the release to production.

After a patch has been certified, the integration team checks the test report meets the acceptance criteria ([[https://twiki.cern.ch/twiki/bin/view/Main/HowToCertify][https://twiki.cern.ch/twiki/bin/view/Main/HowToCertify]] [R26]) and if everything is correct considers the patch is verified. If a test is missing or a mandatory test has failed, the patch is assigned back "With provider". Before the final release to production a patch goes through staged rollout, a beta test where the patch is being installed at some production sites. Patches scheduled for the next staged rollout phase will be set to "Ready for Roll-out". While being in staged roll out the stage is "Rolling-out" whereas "Ready for Production" means that the patch passed the beta test and can now be released to production. The rpms are signed (only for 3.2) when they are prepared for Staged roll-out.


<table align="center">
<caption align="bottom">Patch states diagram</caption>
<tr><td>[[https://twiki.cern.ch/twiki/pub/EGEE/DevelopersGuide/PatchStates_twiki.jpg]]</td></tr>
</table>

---++++ 4.4.1 Release Notes

Release notes must be prepared for every new release of a gLite product. The text should be prepared by the Product Team responsible for the release and is included in the corresponding patch in the =Release notes= field. 
They describe in plain English the contents of the release according to the following structure:

   * What's new:   Brief description of the main changes, both new features and bug fixes. This part will also be included in the summary page for a certain gLite update.
   * Installation and configuration (if configuration): More details on installation and configuration, if the standard messages "Service must be reconfigured" and "Service must be restarted" are not enough.
   * Know issues (if applicable): Known issues present in the release, possibly with a workaround.
   * Dependencies (if applicable and only for metapackage releases): List of internal releases the metapackage release depens upon, including a reference to the patches (id, title, link) and relevant excerpts from the respective release notes.
  
For metapackage releases the text will be uploaded by the Integration Team to the gLite web site when the release is ready, without any significant processing apart from a conversion from text to html (using the txt2html utility).

The release notes on the gLite web site are followed by the list of bugs (id, title, link) attached to the metapackage patch.

Release notes are not meant to replace documentation.

---++ Appendix A. Environment
This chapter describes the characteristics of the environment in which gLite daemons and services will run. This comprises the environmental variables the services have to use, guidelines for running and configuring daemons and services, and resource usage. As much as possible the standard locations for configuration files, start-up scripts etc. should be used as well as the standard methods to start/stop and configure services. Adherence to for example the unix/linux standards described in [[http://www.pathname.com/fhs/][http://www.pathname.com/fhs/]] and [[http://www.linuxbase.org/][http://www.linuxbase.org/]], simplifies the deployment of individual components and subsystems outside the gLite scope.
Much of the contents of this chapter is taken from the [[https://edms.cern.ch/document/358824][EDG Developers&#8217; Guide]] [R3].

---+++ A.1 Enviromental variables
It must be possible to install and configure the gLite software suite as a non-privileged user and the whole tree should be relocatable. This means that as much as possible software should be installed in the gLite root, which can be specified by setting the environment variable =$GLITE_LOCATION=. The software should rely for its installation and configuration on this variable.The gLite software services shall source/parse a top level gLite configuration file and shall not rely on this being previously done (i.e. never assume that variables defined in this file are already present in the environment). The location of this file is platform dependent. For example on !RedHat systems it would be =/etc/sysconfig/glite=, on Debian systems =/etc/defaults/glite=. This platform independent configuration for gLite services should be stored in =/etc/glite.conf= , which can be overridden by =$HOME/.glite.conf=.

This system dependent top level configuration file will have a key-value based syntax, containing all the needed basic prefixes with the following format: 

<verbatim>
GLITE_LOCATION=/opt/glite
GLITE_LOCATION_VAR=/var/opt/glite
GLITE_TMP=/tmp
GLITE_HOST_KEY=/etc/grid-security/hostkey.pem
GLITE_HOST_CERT=/etc/grid-security/hostcert.pem
</verbatim>

The values should only be literal values.  The typical values are given above with the exception of =$GLITE_LOCATION_VAR= which has a default of =$GLITE_LOCATION/var=.

For =$GLITE*= values not defined in the file, the defaults may be assumed by the daemon.

The variable =$GLITE_LOCATION= will point to =/opt/glite= by default

The variable =$GLITE_LOCATION_VAR= will point to the root of a tree for machine-specific files.  Since the =$GLITE_LOCATION= tree should be mountable read-only by clients, the var directory for "variable" data should not be located within this tree, but in the top level var directory. The use of this variable will allow the gLite software to have a central complete installation which is shared by many machines.

The variable =$GLITE_TMP= will point to the root directory of a world-writable area for temporary files.  By default, this will have the value =/tmp=. The =$GLITE_TMP= area is subject to periodic cleaning by cron scripts and at reboots.  Consequently, daemons should not store state information in this area; such information should instead be placed in =$GLITE_LOCATION/<my_dir>=.

Packages are free to define their own environmental variables of the form 
<verbatim>
$GLITE_<package>_<anything>
</verbatim>
The value of package should be negotiated with those managing the release.  In all cases, these variables must be derived from the standard $GLITE_LOCATION, =$GLITE_LOCATION_VAR=, and =GLITE_TMP= variables.

The =$GLITE_LOCATION_VAR= area should be organized as to contain the following directories:
   * etc : machine-specific configuration files

Note that the directory =$GLITE_LOCATION_VAR/etc= is for machine specific configuration files only. Configuration files that are machine independent should go in the directory =$GLITE_LOCATION/etc=. 

More information on what should be contained in these directories can be found in the sections below.

---+++ A.2 Daemons and Services

Most of the grid services are intended to run continuously while the machine is up to service user requests. As such, a permanently running daemon controlled via an init.d script is appropriate rather than using inetd services.

Daemons should not run as root.  If this is unavoidable, then the daemon should be written with a minimal kernel which runs as root.  This daemon kernel can then either interact with another non-root daemon or spawn non-root processes to handle the bulk of the processing. 

All initialization of a daemon should be done with the =init.d= script which controls the daemon.  The =init.d= script should minimally support the start, stop, status, and restart methods. The init.d scripts are by default located in =/etc/init.d=, but it must be possible during the build to configure this location to a directory writable by non-privileged user, e.g. =$GLITE_LOCATION/etc/init.d=. It is advised to have a separate configuration package for each component and for each platform to maintain relocatability and flexibility.

Log files should be written to the standard area =/var/log/glite= and be written in such a way that the logs can be rotated. Each log entry must comprise the time stamp and the process identifier generating it. All authentication and authorization logging should by default be diverted to a separate secure log and such a way that it will never enter the normal logs. Such a secure logging target could be a Trusted Timestamping Service (TTS) or a remote secure syslog service.

Below is an example of a log entry generated by the process 8876 on 12 July 2004 at 3:44:59 p.m.:
<verbatim>
2004/07/12 15:44:59 [8876] ERROR>
</verbatim>
<verbatim>
rotator.java:14 rotate() &#8211; [APPLICATION_FAULT] RotationService - No rotation parameter
</verbatim>
This format allows for easy concatenation of logfiles and sorting by time stamp. Please note that the length of the time stamp is fixed. 

It is foreseen that services will use common gLite logging facilities, which are based on industry-standard logging tools, such as log4j, log4perl, log4Cpp and log4c. The logging tools will be customized as described in [[https://edms.cern.ch/document/486630][https://edms.cern.ch/document/486630]] [R10].

---++++ A.2.1 Daemon/Service credentials

The daemon should use the X509 credentials that are given by the environment variables =$GLITE_HOST_CERT= and =$GLITE_HOST_KEY= . If these are not set the services should respect the default SSL settings, e.g. by using =X509_get_default_cert_dir_env()= and =X509_get_default_cert_file_env()=. These locations can be set by =$SSL_CERT_DIR= and =$SSL_CERT_FILE=, respectively.

A service should check the user credentials and manage its local resources accordingly. However, if the service interacts with another service even a remote instance of the same service, then the user's (delegated) proxy must be used.  This is the only way to allow the remote site to verify the credentials itself and then to correctly log transactions and quotas.

When a daemon runs as a service user, host/service credentials should be made available for that user. The current practice is to make a copy of /etc/grid-security/host*.pem in the same directory (/etc/grid-security/tomcat*.pem) or into a subdirectory (/etc/grid-security/dpmmgr/dpmmgr*.pem) and change the ownership of the credentials to the service user.

---++++ A.2.2 Daemon/Service Configuration

The =init.d= scripts must contain the comments necessary to support the chkconfig command under RH Linux.

A full, annotated template configuration file must be provided with the package.  This should have a &#8220;.template&#8221; suffix so that upgrades to the package do not wipe out a working configuration file.  Changes in the package which require changes in the configuration file MUST be clearly stated in the package release notes and documentation. 

Configuration components must be provided for each service or =init.d= sysV script that needs to be reconfigured.


---++++ A.2.3 Linux Packages profile.d Scripts

The package profile.d scripts must be installed in the standard gLite tree in =etc/profile.d=.  These will be executed by a pair of master gLite profile.d scripts.  The package profile.d script may assume that the =GLITE_LOCATION*= variables have been defined.

The master gLite profile.d scripts will ONLY execute those scripts in the =$GLITE_LOCATION_VAR/etc/profile.d= area.  The profiles in =$GLITE_LOCATION/etc/profile.d= should always take the values in the variables set by the master profile.  System administrators who wish to have a separate machine-specific area must make symbolic links to the desired scripts in =$GLITE_LOCATION/etc/profile.d=.  

There must be no dependencies between the scripts in the =$GLITE_LOCATION/etc/profile.d= area, as the order of execution of files in this area is not guaranteed.  

Developers must provide both sh and csh versions of profile.d scripts The sh version must be called *.sh and must execute correctly with the sh, bash, and zsh shells.The csh-shell version must be called *.csh and must execute correctly with csh and tcsh shells.  This requirement is extremely important otherwise the grid software will place restrictions on what shells the end-user can or cannot use. 

On installation, the scripts should have the executable bits set.

---++++ A.2.4 Cron Jobs and Scheduled Tasks

Cron jobs are appropriate for administrative tasks which must be carried out periodically.  The scripts should be installed in the =etc/cron.d= area of the installation tree given by =$GLITE_LOCATION=.

To increase the portability of these scripts, use the standard Bourne shell features. In addition, make sure your script sets the appropriate environment it needs to work properly (=PATH=, =LD_LIBRARY_PATH=, etc.) and doesn't expect it to be set by the caller.

Within, the script should be a brief description of what the script does and a recommendation on how frequently it should be run. These scripts should not require additional customization beyond the standard package configuration. 

The execution of the cron jobs should be logged in =/var/log/glite=.

---++++ A.3 Resource Usage

---++++ A.3.1 Temporary Space

Developers should use the =$GLITE_TMP= variable to locate temporary disk space local to the machine.  This location is guaranteed to be world-writable. If the location is not set, =$TMPDIR= should be used. The use of the standard routines and methods to deal with temporary disk space is encouraged, e.g. tmpfile() and =mkstemp()= for C/C++ and for java the tmpfile method from =java.lang=

In the temporary area, create a subdirectory if your package will generate a large number of files, or there is a reasonable chance of collision with someone else's files. The name of the subdirectory should be descriptive of the package and reduce the chance of collision.

The package is responsible for cleaning up all files, sockets, etc. created by the package during its execution. Preferably this is done when the file is no longer needed but at the very least when the daemon shuts down. Individual files or directories may be left if an error has occurred and the developer feels that the temporary file may help with debugging or tracing the problem.

On worker nodes the jobs must use =$TMPDIR= (when set) when looking for temporary space.

---++++ A.3.2 Ports

Ports used by a service must be clearly documented.  Additionally, the service port(s) must be easily and completely configurable as well as the port ranges for temporary ports. Services should never rely on fixed port numbers. Sites should be able to organize the network as they wish, internal or external connectivity, NAT, firewall etc.

---++++ A.3.3 Network Addresses

It should be possible to set the service to advertise some other host alias as the location than the real IP address. For example if the service is running in a box with an internal IP address it should register itself into the information system with the NAT server's address. This is also needed for systems with multiple interfaces.


---++++ A.3.4 Logging Formats
A set of logging rules can be found [[https://twiki.cern.ch/twiki/pub/EGEE/EGEEgLite/logging.html][here]] [R9].


---++ Appendix B. Copyright and License

The gLite source code is copyrighted by the [[http://www.eu-egee.org/partners/][Members of the EGEE Collaboration]] and is licensed under the [[http://www.apache.org/licenses/LICENSE-2.0][Apache License, Version 2.0]].

Each file containing source code developed within one of the EGEE projects shall contain the copyright and license attribution available at EGEEgLiteSoftwareLicense [R11]. The specification of the year should cover all the years the code has been modified (acceptable forms are, for example, "2005, 2006", "2005-2009" and combinations of the two).

If the source code has been mainly developed outside of the EGEE projects, it should contain the following attribution:

=This work has been partially funded by the EU Commission (contract INFSO-RI-222667) under the EGEE-III collaboration.=


---++ Appendix C. IPV6 Compliance

To produce IPv6 compliant code the programmers should first use standard APIs that follow RFC 3493, 3542, 4038 recommendations. There are libraries offering compatible interfaces for many languages including C, C++, Java, Python and Perl. Programmers should stay at high levels of abstraction as far as possible. For example use DNS names instead of numeric address and take IPv6 programming guide  into consideration. The main problem in porting applications to use IPv6 is to deal with different operating systems and their possible configurations and to be able to offer the best connectivity, i.e. to answer both to IPv4 and IPv6 requests if possible. The gLite deployment platform is mainly Scientific Linux 4 or 5. Scientific Linux 4 and 5 are dual stack nodes  where the deployment platform will be configured to enable IPv6 or IPv4. Depending on the operating system features available (separate stack, dual stack, !IPv6BindOnly) and language used by the application there are two main options for a server application: 
   1. wait for request on both IPv6 and IPv4 interface with two sockets 
   2. wait for request only on IPv6 interface and use IPv4 mapped IPv6 address to redirect IPv4 requests to the IPv6 part of the stack 
The first point to consider is the targeted operating system and what features are available on it: dual-stack, separated stack, IPv4 mapped IPv6 address, ability for IPv6 socket to listen only IPv6 request (IPV6_ONLY / bindv6only). The second point to consider is the specific socket options that are required by the application. Not all of the socket options that are available in the IPv4 API may be available in the IPv6 API. In that case IPv4 mapped IPv6 addresses might not be suitable. Some languages, like Java, may hide some complexity and take account of the different operating possibilities. Java  prefers IPv4 mapped IPv6 address if possible. Unfortunately the mapped addresses need special security consideration  that may need to be handled by setting up an appropriate filter on the router or firewall. Furthermore the application itself may have to take the usage of IPv4 mapped IPv6 address into account. In the &#8220;Guidelines for IP version independence in GGF specification&#8221;  the authors recommend using IPv4 mapped IPv6 address in part because the code is easier to develop. 
According to the specific needs of gLite and the deployment platform targeted the programmers and deployment staff have to consider this problem to define a strategy. A unique method may not handle all cases in gLite but a guideline on these topics will guide programmers in their choice. It is believed that having a coherent and seamless approach is desirable and will help, particularly during deployment when several different applications may be running on a single node.

A document with guidelines on how to produce ipv6 compliant code for the GRID can be found here:  http://www.euchinagrid.org/docs/EUChinaGRID-Del2.2v3-1.pdf 

A document explaining how to test IPV6 compliance for your application can be found here: https://edms.cern.ch/document/930868/1 [R12].


-- Maria Alandes - 16 Feb 2010
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
jpg	BugStates.jpg	r1	manage	220.9 K	2009-08-14 - 16:05	AndreasUnterkircher	BugStates
jpg	BugStates_twiki.jpg	r1	manage	220.9 K	2010-02-16 - 11:59	MariaALANDESPRADILLO	New "Fix not certified" state included.
jpg	PatchStates.jpg	r4 r3 r2 r1	manage	70.2 K	2010-01-21 - 14:45	UnknownUser	PatchStates
jpg	PatchStates_twiki.jpg	r1	manage	79.6 K	2010-02-16 - 11:59	MariaALANDESPRADILLO	New Verified status included in the Patch States.
Topic revision: r64 - 2011-02-04 - MariaALANDESPRADILLO
EGEE
Main.WebList
Welcome Guest
- Cern Search
- TWiki Search
- Google Search
EGEE All webs
Copyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback