LCG Developers Guide

This guide is for anyone developing or modifying code for LCG. The main objective is to define the procedures used by LCG for software development and to ensure that the software produced meets quality required for production.

The guide covers the following areas.

• CVS Structure And Source Code Management.
• Required Documentation.
• Software Configuration.
• Software Packaging.
• Software Testing.
• The Autobuild System.
• Submission For Addition To LCG.

Procedure

The software development procedure can be broken down into a few simple steps.

• Create a new module in CVS.
• Write the code and documentation.
• Thoroughly test the code.
• Tag the CVS tree for the module.
• Contact the building system manager to add your module in the list of modules to build.
• Ensure that Autobuild has successfully created the package.
• Thoroughly test the package.
• Submit the autobuilt package to LCG.
• Fix any bugs found by the integration and certification process.

Quality Control And Feedback

Initial level of quality control lies with the developer. The developer should ensure that the software has the required functionality and that this is working. Unit test can be very usful in this respect but others tests should be considered as well. The methods and results of these additional tests should be documented.

The second level of control is the Autobuild machine. Packages will only be published in the Autobuild repository if the build is successful. The feedback to the developer is via email and the Autobuild results page.

The last level of quality control is done the LCG integration and certification process. If any problems are found, these should be fed back to to developer by using the Savannah bug tracking tool.

CVS And Source Code Management

All source code is held in the CERN central CVS repository, in the project lcgware.

Read Only Access

export CVSROOT=:pserver:anonymous@isscvs.cern.ch:/local/reps/lcgware

Read/Write Access

     export CVSROOT=:ext:isscvs.cern.ch:/local/reps/lcgware
     export CVS_RSH=ssh

Developpers have to have an afs account at cern. Contact the CVS librarian to open an account.

For more information, see CERN CVS documentation page.

Modules, Meta-Modules And CVS Directories

A module is a CVS directory that contains the source code for a given functionality. The name of the module should be in the format of project-area-name. Each module should create a package with the same name of the module eg. the module lcg-example-program should create the package lcg-example-program. As one module should create one package, the granularity of packaging should be considered when creating a module, see section on packaging.

A meta-module is a CVS directory that contains modules. If a number of similar modules exist then they should be grouped into a meta-module. eg. The meta-module lcg-example could contain the modules lcg-example-client and lcg-example-server. CVS structure

The CVS directory content for modules and meta-modules should be as follows.

• The top-level CVS directory should contain only modules and meta-modules.
• A meta-module should contain only modules, a README file and a Makefile.
• A module should also contain such files plus the directories, src, etc, doc, build and test.

The files and directories mentioned above should be used in the following way.

* The README file, see the documentation section for details. * Build files are required for Autobuild. * The src directory contains all the source code for the module. * The etc directory contains all the static configuration files, templates etc. for the module. * The doc directory should contain all the documentation for the module. This should include any example configuration files. * The test directory should contain all testing material, including unit tests and descriptions of one-off tests that have been conducted. See the testing section for more details. * The build area is used for compiling, testing and packaging should not be checked into CVS.

Version Numbering

Version numbers, CVS tags should be in the format version_revision_patch e.g. lcg1_0_2. The version number should change only when there are non-backward compatible changes or when there are major changes with features. The revision number change is for additional backward compatible changes. The patch number is for bugfixes. When a tag is made, the Autobuild machine will build the code and do some automatic testing.

Install Locations

The table below shows where installed files should be located. If new directories need to be created they should not be created at the top level. To avoid conflicts with other packages and to uniquely identify the documentation for a package, the documentation should be placed in the directory $LCG_LOCATION/share/doc/name with name being the package name. Below $LCG_LOCATION/share/man, the usual Unix conventions should be followed. $LCG_LOCATION is typically /opt/lcg, although this can not be assumed. Care must be taken to make the package entirely relocatable.

Software Tree Layout

Location	Purpose
$LCG_LOCATION/bin	End user executables
$LCG_LOCATION/etc	Configuration files (use *.conf extension)
$LCG_LOCATION/etc/profile.d	Profile scripts
$LCG_LOCATION/include	Header files
$LCG_LOCATION/lib	Sharable and static libraries (excl. java)
$LCG_LOCATION/libexec	Support executables invoked by bin or sbin commands
$LCG_LOCATION/sbin	Non-end user, system, or configuration executables
$LCG_LOCATION/share	Shared architecture-independent files
$LCG_LOCATION/share/java	Java libraries (*.jar, *.war, etc.)
$LCG_LOCATION/share/doc/name-version	LICENSE, README, and documentation
$LCG_LOCATION/share/man	man pages
$LCG_LOCATION/share/man/man1	User programs
$LCG_LOCATION/share/man/man2	Unix system calls
$LCG_LOCATION/share/man/man3	Programming libraries
$LCG_LOCATION/share/man/man4	Device node files
$LCG_LOCATION/share/man/man5	System configuration file formats
$LCG_LOCATION/share/man/man6	Games
$LCG_LOCATION/share/man/man7	Macro file formats
$LCG_LOCATION/share/man/man8	Daemons
$LCG_LOCATION/share/man/man9	Local system calls
$LCG_LOCATION/share/man/mann	Miscellaneous
$LCG_LOCATION/var/log	log files

Documentation

Every package should contain a README file. Other types of documentation will depend on what is in the package.

The README file

This file should be in plain text and should contain the following information or a link towhere the information can be found.

• The function of the module.
• Change log.
• The main download/information page for the software.
• The license of the software.
• Build and runtime dependencies.
• How to build and install.
• A configuration guide.
• How to get more informations, especially about usage.
• Known bugs and workarounds.
• Planned evolution (future of the software).
• Contact: who made the software and how to contact them.

Overview Documentation

All large packages should contain an overview of what the package does and how it works.

API Documentation

All APIs should have API documentation. This can easily be provided with Doxygen for C/C++ code, Javadoc for Java code and POD for Perl.

End User Documentation

Every program that is executed by either a user system administrator requires documentation, especially a man page. This is anything that goes in bin or sbin.

Configuration Documentation

If a package requires configuration, documentation on how to do the configuration is required. A

Test documentation

All testing that has been conducted on the module should be documented. This does not need to be distributed but should reside in the modules test directory in CVS. See testing section for more details

Configuration

The software should be configured using one configuration script and one configuration file. There is no reason why the configuration scripts can not create multiple configuration files and call helper scripts.

An example configuration file should be provided in the documentation.

For manual installation all that should be required is to copy the example configuration file to the correct location. Edit it with the required values and run the configuration script. If possible, a configuration script that will work out of the box should be provided.

The LCG release is installed an configured using Yaim. If you wish to create Yaim configuration, please read the Yaim Deveopers Guide.

Testing

Everything to do with testing of a module should be contained in the test directory for that module.

Every module should have a unit testing suite that can be run from the compiled code. This test suite could be automatically run by the autobuild machine after it has compiled the code. The coverage of the unit tests and limitations should be described in a text file within the test directory.

Any one-off test such as stress tests and performance tests should be documented with results, in such a way that someone else could repeat the test if so desired.

Another testing suite should test the module once installed. This could be packaged but is not necessary. Also instructions of how someone can test the package once installed should also be contained in a text file within the test directory.

Software Packaging

This section describes how to ensure packages are consistent with the installation tools and do not conflict with other packages.

Granularity Of Packaging

A package is a single file that contains the software product or some subset of it for distribution. If the guide in the section on CVS management has been followed then the granularity of packaging should already have been decided. The following should be considered when deciding the granularity of packaging.

• The packages must be large enough that no cyclic dependencies exist between them.
• The packages should be split to separate code likely to be deployed on separate machines, e.g. client and server code.
• There must be no conflicts between the packages, e.g. a file in the same location in two packages.
• Packages should be small to allow individual components to be independently upgraded.
• Public interfaces should be in separate packages and versioned independently.
• Code intended only for developers should be split into a separate package.

Package Dependencies

If the packaging method contains dependency management, this should be used. The build and runtime dependencies should already be listed in either the README file and or in the documentation.

Specific Package Formats

Names of packages should correspond to the CVS module name. Also the version of x.y.z should correspond exactly to a CVS module tag of lcgx_y_z. If packages are released for different compiler versions, an extension should be added to the package name in order to retain this information in the name. The current acknowledged extension is the string _gcc3_2_2 for gcc3.2.2. To help giving the appropriate name to a package, the Autobuild system defines the variable LCG_RPM_RELEASE_VERSION to either the empty string or _gcc3_2_2. Developers are encouraged to rely on this variable in order to set the appropriate name.

RPM Packages

For more information about RPMs see, http://www.rpm.org

The RPM spec file should be located in the etc CVS directory for the module. The following points should be considered when building RPMS.

• RPM building should be called using rpmbuild commad not rpm -b.
• All of the fields in the spec file should be completely and correctly provided.
• RPM packages allow a release number in addition to the version. As the RPM spec file is in CVS, each change in the package will involve at least a patch level revision. Consequently, the release number will always be 1.

• Any RPM packages built from the LCG fork of the EDG code will require the release number to be lcg1. This will ensure that there are no conflicts with version numbers and make the source repository easy to locate.

• The RPM format allows a hierarchical categorization of packages. For LCG software set group name in spec file to grid/lcg.
• RPM is continually evolving and adds convenient macros. Unfortunately using them makes the package incompatible with earlier versions of RPM. Therefore, avoid fancy features of the latest and greatest RPM including macros such as %makeinstall (distribution dependent), and features of rpmlib (version dependent).
• Configuration varies greatly from platform to platform and should be clearly separated from the package itself. Consequently, RPM installation scripts like %post, %pre, for configuration should be avoided. However, it is ok to replace path names resulting from the build procedure.

• The Autobuild machine builds and installs the module using the Makefile. The only line required for the build section in the RPM spec file should be as follows.

\%build
make install install-doc prefix=\%{buildroot}\%{prefix}

• To enhance the portability of the package, do not disable RPM's internal provide mechanism. This means for example that the following should not be in the spec file:

   AutoReqProv:    no

• Correctly specifying the dependencies simplifies installation. The distinction between build and run-time dependencies should be made. The BuildRequires tag specifies build dependencies and the Requires tag specifies run-time dependencies. Every required package not automatically found should be manually added.

   Requires:    lcg-example-client

• All RPMs should be buildable by a non-root user. The BuildRoot feature should be used and following lines should be in the specfile.

   %files
   %defattr(-,root,root)

Automatic Build System

The Autobuild system is used to automatically build code that is in the LCG CVS repository. The Autobuild system will build the latest tag of the format lcgx_y_z for each modules configure on it to be build. To add your module contact the Building system manager (Louis Poncet) If the build is successful, and any tests on the packages have passed, the generated packages will be published in the autobuild repository. If the package fails to build or does not pass the tests, the package will not be published and an email will be sent to the package owner.

The result are shown on the Autobuild Result Page

Autotools

Modules managed by Autotools must be able to create the configure script from a CVS checkout with the following command: ./autogen.sh. The new name is more compliant with usages in the open source community.

The autogen.sh file resides at the root of the module and contains the necessary first steps for Autotools to run, for example:

#!/bin/sh set -x aclocal -I config autoheader libtoolize --automake autoconf

The generated configure script is then run to generate the Makefile.

Makefile

The top-level Makefile must contain the required targets and perform the required tasks.

all

Default target, build the package (create executable and libraries).

install

Install the software (default location must be $EDG_LOCATION).

dist

Create a source tar.gz archive using the naming convention described in Section [*].

binary-dist

(optional) Create a binary tar.gz archive using the naming convention described in Section [*].

rpm

Generate RPM packages. Built RPMs must be found under a RedHat directory tree analogous to the one in /usr/src/redhat. The _topdir should either be left undefined (managed by the autobuild program), set to the build directory or set under rpm in this directory. Autobuild will look in the following places for generated rpms:

• If rpm directory0.1 exists, then rpm/RPMS// and rpm/SRPMS/.
• Else, if RPMS directory exists, then RPMS// and SRPMS/.
• Otherwise, the actual rpm _topdir location will be used as base path.

apidoc

Generate developer documentation (using Doxygen for C/C++ or JavaDoc for Java). This documentation should be found in doc/apidoc directory.

userdoc

Generate user documentation. It should be found in doc/userdoc directory.

check

Some automatic checking method will be implemented, but several levels of checking may be needed that will not be addressed by a single make target. This item has yet to be developed further.

All targets which succeed must return zero to the caller, all targets which fail must return a non-zero value.

Invoking any of the required targets must never result in an interactive prompt. The build procedure is an automatic process which has no ability to respond to the prompt and hangs as a result.

Ant

Some Java modules use ant as a building tool. Ant targets should be the same as those described for Makefile, except that the all target is named compile with this tool.

All targets which succeed must return zero to the caller, all targets which fail must return a non-zero value.

Invoking any of the required targets must never result in an interactive prompt. The build procedure is an automatic process which has no ability to respond to the prompt and hangs as a result.

Build On Demand

The build on demand system allows a build to be performed immediately on request. To do this you need to telnet to the build on demand machine

telnet machine 6666

and issue the command

lcg-example lcgx_y_z

where lcg-example is the module to build and lcgx_y_z is the CVS tag.

The build results and logs are published on a dedicated web page, follow the links from the autobuild web page (under Build On Demand page).

LCG Submission

The [[https://savannah.cern.ch/patch/?func=additem&group=lcgoperation][LCG package submission page] is used to submit packages to LCG for consideration for inclusion in a release. This system is for submission of all new LCG software and external rpms. It is useful for tracking the evolution of the LCG source code and middleware. All packages should be submitted to LCG via this page. This includes LCG packages, EDG packages and external packages.

Note: You have to create an account and log in on the system before using it.

The submission page comprises of a web form with the following fields.

• Name (package/component/archive):
• Revision Tag:
• Names of binary packages:
• Names of source packages:
• Priority:
• Assigned to:
• Configuration:
• Place to find packages:
• Dependencies:
• Summary:
• Original Submission:

When this form is submitted, it is automatically checked to ensure that all the required fields have been filled. Each new package is assigned a status to show where it is in the test phase. The management system will automatically inform the people concerned. Any problems found with the integration of this new package will be entered into the relevant section in Savannah.