Integration Procedure


Packages are built using the ETICS build system, for more details on how to use the ETICS build system see the ETICS user guide. The aim of integration is to take the packages from the ETICS repository and maintain yum repositories for certification, pre-production and production. One aspect of integration is to control what versions of packages are used and to ensure that everything is installable. Various integration scripts have been created to help automate many of the tasks involved. The integration dashboard shows the status integration. The node lxn1190 has been set up to act as a integration machine and has glite-release checkout out in the directory /home/glbuild.


The ETICS build system is seen as a "Build Factory" which creates packages and places them in the ETICS repository. Integration is done at the package level and ensures than any problems with building or the build system will not affect integration. The ETICS repository should be seen as the interface between building and integration. All packages must be taken from the ETICS repository which can be found in AFS.

  /afs/           for gLite software
  /afs/         for external software

The rpms in there are stored in subdirectories of the format:


e.g. in

For 3.1, the build is defined using the configuration glite_branch_3_1_0 in the project org.glite. When a developer updates a component and creates a new subsystem, this will need to be communicated to the maintainer of the glite configuration. Usually this is done via the patch submission in Savannah. The builds occur automatically four times a day and currently takes about eight hours. The builds are done in parallel on all the supported platforms.

In addition, RPMs that are not yet in the ETICS system are stored in

in the same structure as they would have within the ETICS repository for the certification and release process.

YUM Repositories

Note on using protect directive

YUM repositories which haven been 'protected' are not fully protected! Let me show you this on an example package.

A metapackage m_A has a dependency a_DEP. This is provided by a package B in a protected repository but also provided by package C in another repository (it doesn't matter if it is protected as well or not). YUM might then not take package B to satisfy a_DEP. The directive protect only prevents updates from other repos on already installed packages.

gLite Repositories

The YUM repositories for certification and PPS can be found in AFS, /afs/, which corresponds to

In 3.1, symbolic links per node should be made to the directory generic so that unique repositories can be used in the case of conflicts between node types. All the node types repositories are symbolic links to generic except glite-WN, dcache node types, which contain a symbolic link to generic-dcache, and glite-HYDRA_mysql.

In 3.2, all the packages are under glite-GENERIC and each node type contains also a repository with only repository metadata.

3.1 CERT /afs/
3.2 CERT /afs/
PPS (prepare) /afs/
PPS /afs/
BETA (prepare) /afs/
PROD (prepare) /afs/
PROD /afs/
Integration /afs/ None
ETICS /afs/

Integration Repository

On AFS: /afs/ This directory contains packages which are not built in ETICS nor available from ETICS external repository. An example here are the dCache rpms.

VDT Packages

The gLite middleware uses some VDT packages which -in case of an update- may have to be downloaded into the integration repository or into etics (requesting external package form). You can find the VDT packages here:

These repositories include the following packages :

  • uberftp-client
  • vdt_globus_essentials
  • vdt_globus_rm_essentials
  • vdt_globus_jobmanager_common

IDEA! Alternatively the vdt packages can be found in ETICS here


On afs: /afs/

The repository structure is as follows for each patch:


ALERT! Note: In order to install the patch, the production repository for the affected metapackage has to be enabled too.

Pre-production Service (PPS)

On afs: /afs/

The repository structure is as follows for 3.1 (where repo is generic, glite-WN, generic-dcache or glite-HYDRA_mysql; and arch is i386 or x86_64):


The repository structure is as follows for 3.2:



On afs: On afs: /afs/

This location is where we work, but the actual production repository is in /afs/ or

ATTENTION WITH CHANGES : Changes into /afs/ are effective every 30min past the full hour (1.30,2.30, 3.30, etc..)!

The structure of the repository is as follows for 3.1:

  ./$release/generic/$os/$arch/RPMS.release       for originaly released gLite rpms
  ./$release/generic/$os/$arch/RPMS.updates      for any rpm released after the initial release
  ./$release/generic/$os/$arch/RPMS.externals    for external rpms released originally

The repository structure is as follows for 3.2:


Package Life Cycle

A set of packages are requested to be added to the release. A patch represents this set of packages. A patch is usually identified by a number, however, for an initial release the patch can be identified by a node type name.

A patch must first be added to the certification repository so that it can be tested.


Once a patch has been certified, it can be considered for pre-production or production. In the pps and production repository, the packages required should be put into one of the following directories.

New Packages
Updated Packages
External Packages

In 3.2, packages are signed once they are moved into the intermediate certified repo, which is just a location under where the relevant packages are signed and stored. This happens just before moving to pps. The sequence would be cert -> sign rpms -> certified -> pps -> prod.

In 3.1, we don't sign packages but we also use the certified intermediate repository.

Information Management

As integration is done at the package level, the name and version is required to identify a package. To be able to locate the package in the ETICS repository, the corresponding component name that is used in the ETICS system is also required. The format used to store this information in a file is


The star at the end is only required to identify the first level dependencies which are used to create a meta-package. All the entries without the star are lower level dependencies. Putting only the first level dependencies in the meta-package is required for testing, however, for production all the lower level dependencies will also need to be include in the meta-package. This is required as we need to check the dependencies during testing but for the release we need to include all packages so that the version of the meta-package identifies what is installed.

Lists of packages for each node type are created for the release. These lists can be found in the CVS directory glilte-release/prod-lists or glilte-release/pps-lists directory. The list should checked back into CVS after changes have been made and a tag should be made when the list is released.

Repository Management

The repositories are created when creating a release to production or pps. The script to be used in make-release and it uses the lists to build the repositories. It writes by default in the prepare area and later the repositories can be moved to the official pps and production locations.

The Application Area

For the 3.1 releases we are introducing an extra distribution route for a subset of the clients. We will maintain a section of the Application Area repository in order to give the experiments early access to updates and to enable them to use these clients in building their own distributions.

The client list is held in the file glite-AA and this should be maintained just like any other rpm list. The Application Area repository should be populated with any client updates which enter certification, and the list defining what has been released should be updated when a release is made. The full story is described here.

Patch Handling

To make it easier to manage the transition of patches between the different repositories, the move-patch command can be used to move the patches between the different state and hence repository locations.

In order to better understand the different patch and bug states, please check these two links:


All build and integration problems must be tracked in Savannah. The field Category should be set to Integration.

Problems found in a release should be described in the Integration Diagnosis page.

All communication of integration issues should be done through the integration email list:


Handling patches within product teams

Product teams should create a patch whenever they want to release a new version of their products to production. Follow the Product Team Integration guide to know the details of the process.

Handling patches with scripts for CERT, PPS, Production

All scripts are located here: here . They are configured to use a common config file containing all relevant locations/control flags, etc. This can be found in glite-release/

Each script contains a help/documentation.

Checking out the release script(s)

klog.krb $user;
kinit $user;
export CVSROOT=:gserver:$;
export CVS_RSH=ssh;
cvs co glite-release;
yum install createrepo; 
yum install repoview; 

Checking out the savannah client

more Information here : SavannahCommandLineInterface
cvs co org.glite.integration.savannah;

Releasing to Certification

Follow the 'Move to certification' check list that includes the steps to be followed when moving a patch into Readyfor certification.

Releasing to PPS

Follow the preproduction release check list that includes the steps to be followed when doing a preproduction release.

Releasing to Staged rollout

Follow the Staged rollout release check list that includes the steps to be followed when doing a staged rollout release.

Releasing to PROD

Follow the production release check list that includes the steps to be followed when doing a production release.

Signing RPMs

RPMS are signed in gLite 3.2. Check the certification signing procedure to know all the details about the signing procedure.

In order to sign rpms, you need to create the following files in the machine where you want to sign the rpms:

/root/.gnupg -> This contains the key and the necessary files to sign with the GPG key.
/root/.rpmmacros -> This file should contain:

%_signature gpg
%_gpg_name Gd Integration <>

RPMs are signed only in 3.2 and this is done automatically within the glite-release scripts. The integrator has to pay attention when moving patches into certified since the password of the key will be asked.

How and when to use the Integration repository

The integration repository is located under: /afs/

And it contains two directories called:


WHEN to use this repository: whenever we need a package that is not built in ETICS, and itís not available in DAG, jpackage or the OS repositories either.

HOW to use this repository:

  • If the package comes from an external project like maui, torque, dcache or vdt, use externals.
  • If the package is built by us or anybody else in the experiments not using ETICS, then use org.glite.

Just check first if thereís a subdirectory under externals or org.glite with the structure package_name/x.y.z/platform, otherwise create it and copy the package there. Please, put a comment in the Changelog file and always keep the Integration team informed. Itís important to know what we put there.

Why is this repository important? Itís important in particular for 3.2 releases, becase in gLite 3.2 external packages are not signed and only the packages coming from / are regarded as externals and wonít be signed. The packages under org.glite are not considered external packages and they will be signed.

Rolling back the latest release of PROD

  • Restore CVS prod lists
    • For each affected metapackage of update NN
      • cvs update -j HEAD -j glite-release_glite-XXXX_R_x_y_z-1_0 glite-release/prod-lists/slc4_[a32|x86_64]_gcc346/glite-XXXX
      • cvs ci -m "Restoring prod list for glite-release_glite-XXXX_R_x_y_z-1_0" glite-release/prod-lists/slc4_[a32|x86_64]_gcc346/glite-XXXX
      • cvs tag glite-release_glite-XXXX_R_x_y_z_0 glite-release/prod-lists/slc4_[a32|x86_64]_gcc346/glite-XXXX
  • Prepare a backup of the release pages
    • To mount the glite_repository in the machine:
      modprobe cifs
      mount -t cifs -o username=CERN\\glbuild // /mnt/glite_repository
    • Copy the directories that you want to backup
  • Prepare the release pages for the next update
    • For each affected metapackage of update NN
      • cd /mnt/glite-release/package/R3.X/(sl5_)(x86_64)/deployment/glite-XXXX/
      • cp latest-old/glite-XXXX latest/glite-XXXX
  • Make the release as usual (check with SA1 SA3 the value of the release field in the patches)

The rollback script generates the list of CVS tags to be applied into the lists/pps-lists or lists/prod-lists to rollback to a previous pps or production update. It also deletes from the repo the packages introduced in the current version.

rollback -s [pps|prod] -p [slc4_ia32_gcc346|slc4_x86_64_gcc346|sl5_ia32_gcc412|sl5_x86_64_gcc412] -r update_number

When the script is executed, find in the TMP_DIR the list of rpms to keep, rpms to remove, set of concerned patches and the script to apply the CVS tags into the lists.

Create a new metapackage in PPS or PROD

ALERT! Note: For this, the Savannah patch must have all the packages which belong to this new metapackage 'to be added' in Metapackage changes field.

Example (introduction of a new metapackage) :

Savannah Metapackage changes field

Execute Command (example for PPS, same is for PROD)

#move to PPS 
./scripts/move-patch -s pps -m PPS-glite-SCAS 2511

The Result will be a new file in pps-lists/<DEFAULT_PLATFORM>/PPS-glite-SCAS with this this content:

#> cat PPS-glite-SCAS

The same procedure has to be done for production lists.

Example (introduction of a new metapackage with changes to other lists) :

The mechanism allows not only to introduce a new metapackage with a patch, but also to do changes at the same time to other metapackage lists. This is archived by adding metapackage changes for those lists.

The Example here shows the introduction of glite-SCAS to PPS and at the same time adding new packages to glite-WN.

Savannah Metapackage changes field


Other HowTos

Creating a Metapackage

 Usage : create-metapackage -l list -u update_nr -p platform -v version [-m metapackage name] [-r]
scripts/create-metapackage -l prod-lists/slc4_ia32_gcc346/glite-BDII -u 24 -p slc4_ia32_gcc346 -v 3.1.12-0 -m my-BDII-metapackage
Wrote: /root/glite-release/create-metapackage.tmp/SRPMS/my-BDII-metapackage-3.1.12-0.src.rpm
Wrote: /root/glite-release/create-metapackage.tmp/RPMS/i386/my-BDII-metapackage-3.1.12-0.i386.rpm

Creating a version marker (glite-WN x86_64 case)
scripts/create-metapackage -l prod-lists/slc4_x86_64_gcc346/glite-WN -u 24 -p slc4_x86_64_gcc346 -v 3.1.12-0 -r
Wrote: /root/glite-release/create-metapackage.tmp/SRPMS/glite-WN-version-3.1.12-0.src.rpm
Wrote: /root/glite-release/create-metapackage.tmp/RPMS/x86_64/glite-WN-version-3.1.12-0.x86_64.rpm

Creating a rpm group description
 Usage: create-rpm-group-description -p platform -v version -l list -o outfile
scripts/create-rpm-group-description -p slc4_x86_64_gcc346 -v 3.1.0-0 -l prod-lists/slc4_x86_64_gcc346/glite-WN -o /tmp/comps-del.xml 
Generating a group repository
 scripts/create-rpm-group-description -p slc4_x86_64_gcc346 -v 3.1.0-0 -l prod-lists/slc4_x86_64_gcc346/glite-WN -o /tmp/comps-del.xml
# Generate the repository, rpms must be there already (e.g. RPMS.release)
scripts/generate-yum-repository -d /tmp/del2 -g /tmp/comps-del.xml 

Creating a Metapackage for the first time with 32bit and 64bit dependencies

In order to create new metapackages with 32bit and 64bit dependencies, the first time you move the corresponding patch into Ready for Certification, you need to do the following manual steps:

  • Create the dependency list in the glite-release module in CVS under glite-release/prod-lists/architecture/glite-metapackage. The dependency list should contain the #META-add-package-platform entries for the 32bit rpms. See the next section Packages for Groupinstall must be available from both platforms for more details. Commit the changes.
  • In the Savannah patch, specify in the RPM name field only the 64bit rpms. Don't forget to specify the Metapackage changes for each of these rpms.
  • Create the patch directory /afs/
  • Copy the 32bit rpms in the patch directory you've just created.
  • Move the patch into Ready for Certification as usual. Remember to install the metapackage with yum groupinstall to get 32bit and 64bit versions.

Moving a patch into certification with 32bit and 64bit dependencies

In order to create a patch repository with 32bit and 64bit dependencies, you need to do the following manual steps:

  • Create the patch directory /afs/ or /afs/ if the package is an external one.
  • Copy the 32bit rpms in the patch directory you've just created.
  • If the package is a new dependency, add in the corresponding prod-list the #META-add-package-platform|package_name|sl5_ia32_gcc412 to be able to properly build the groupinstall taking the new dependencies.
  • Move the patch into Ready for Certification as usual. Remember to run the following commands in certification to get 32bit and 64bit versions:
    • yum groupinstall cert-glite-UI/WN if there are new depencies.
    • yum groupupdate if there are only new versions.

Backup copy of the gLite web pages

Select the path \\\glite and right click on the folder or file you would like to retrieve. You can then select properties and in the properties window the tab previous version.

You can still make the request through Helpdesk giving the name of the file or folder to recover and the date to get it back from.


To perform automatic tests after moving a patch to certification, or after creating a new preview or production release, a set of scripts have been developed. These scripts allow to perform the needed tests inside a chroot environment using mock, and can be the base for more automatic or manual tests.

Prepare a new machine for testing

Before performing the tests, we need to prepare the test machine with the following steps.

Get a copy of glite-release from CVS

First, we need a copy of glite-release where we can find the scripts needed under glite-release/scripts/chroot_test_scripts.

Prepare the environment

The prepare_env script will install and configure mock and will create specific config files for the different platforms. Run it without parameters.

Prepare the chroot tar files

Afterwards, we need to generate the clean chroots for each platform that will be the base for the test, that can be done executing prepare_tar for each needed platform:

./prepare_tar sl-4-i386.cfg 
./prepare_tar sl-4-x86_64.cfg 
./prepare_tar sl-5-x86_64.cfg 

Create the test

A test consist in a directory containing all the needed information to perform the automatic test. This directory structure can be generated automatically using the script create_test.

The options accepted are:

Usage: create_test -r release -p platform -s state [-a] -n [patch_number|update_number]

  -r   release
  -p   platform
  -s   [cert|preview|prod]
  -n   patch_number if -s cert, update_number if -s [preview|prod]
  -a   don't reinstall the Savannah DB

Create a test for a patch

./create_test -r 3.2 -p slc5_x86_64_gcc412 -s cert -n 3225

Will create a test directory named 'patch_3225'.

Create a test for a preview

./create_test -r 3.1 -p slc4_ia32_gcc346 -s preview -n 05

Will create a test directory named '3.1.0_Bundle_05-i386'.

Create a test for a release

./create_test -r 3.1 -p slc4_x86_64_gcc346 -s prod -n 55

Will create a test directory named '3.1.0_Update_55-x86_64'.

Test directory structure for verification and manual modification

In this section the test directory structure is explained to allow manual verification before running the test or to introduce modifications in the default tests created with create_test.

Directory estructure

Inside a test directory there is a directory for each metapackage to test. Inside the metapackage directory there are two files and another directory as follows.

In this example, we have the metapackage 'glite-CREAM' the fake metapackage 'glite-XXXXX' to show a test for multiple metapackages. The final directory structure is:


Install command

Inside each metapackage directory, there is a file named 'install_cmd' containing the installation command needed for this metapackage, for example:

/usr/bin/yum -y install glite-CREAM

Rpm lists

There is another file named 'rpms.list' containing a list of rpm names that should have been installed after the test, for example:


Repo files

Finally, there is a directory named 'repos' containig the special repo files needed to install the meta package, in this example:


Run a test

Running a test

After creating and/or modifying a test we have to run it. For that, there is a script named 'run_test'. The options accepted are:

Usage: run_test -r release -a arch -t test_directory -d

  -r   release
  -a   arch
  -t   test_directory
  -d   don't delete the chroot

To continue with the previous example we could run:

./run_test -r 3.1 -a i386 -t 3.1.0_Update_56-i386

It will give some output and if the test is successful it will print: "The installation test was successful!"

Playing with the chroot if the test fails

If the installation test was not successful (or if we provided the -d option) we can go to the chroot and see what is the state after the test.

The chroots are in /var/lib/mock/, if the test fails, we can go there, enter in the chroot directory, and we will see a root directory containing the whole chroot. The name of the chroot depends on the platform, the default names are sl-4-i386, sl-4-x86_64, sl-5-x86_64.

If we provide the options -d, we could end up with more than one chroot (one for each metapackage to test), therefore, the metapackage name is append to the chroot name.

If we want to enter in the chroot as the script does we can use the mock commands. First, we need to log as mockuser:

su mockuser

And then, we can run a mock shell with:

mock shell -r sl-4-i386.cfg
changing the config file depending on the architecture as before.

Note that if you want to do this after using the -d option, first you have to rename the chroot of the metapackage you want to it's original name. For example:

mv /var/lib/mock/sl-4-i386_glite-CREAM /var/lib/mock/sl-4-i386

If the installation fails, it's needed to clean the chroot before running a new test. See the next section to learn how to do it.

Cleaning the chroot


To clean a chroot NEVER use an rm command. Can be some mounted filesystems and that could remove important files (for example in AFS).

Instead, use the 'clean_chroot' script (always as root, not as mockuser):

./clean_chroot sl-4-i386

If you have used the -d option, you should move the chroot to the original name before using 'clean_chroot'. It should be implemented an option in 'clean_chroot' to do this automatically.

Known Issues

Packages for Groupinstall must be available from both platforms

It is currently only possible to have 32 AND 64 Bit versions of a certain package on a group installation. This means, that a groupinstall will always contain the 64 Bit packages, but not a 32Bit version of which 64 Bit version is not stated in the list.

3 Examples for a metapackage list :

OK . This will copy both versions of classads in the repo and state the package in the comps.xml

OK . This will copy the 64Bit versionof classads in the repo and state the package in the comps.xml

WRONG! . This will copy the 32 Bit version of classads but it will NOT be stated in the comps.xml


Node Tracker : Glite31NodeTracker
Integration Dashboard : integration dashboard
AA RSS Feed :
gLite RSS Feed :
CA update :
Edit | Attach | Watch | Print version | History: r73 < r72 < r71 < r70 < r69 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r73 - 2009-11-11 - MariaALANDESPRADILLO
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EGEE All webs login

This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback