Production Release Check List

Introduction

The following list must be used in combination with the Integration Procedure twiki that describes all the technical details involved in the release process. This twiki aims to list the necessary steps to create a production release helping to avoid mistakes and decrease human errors.

NOTE The following steps must be followed in both 32bit and 64bit machines. Check in the patches to be released which platforms are affected.

Check List

1. Make sure the previous production release was properly tagged in CVS. You can do this by checking that the last version of each metapackage in the production list is tagged in CVS. If not, tag it by increasing the version. The version should be the same one as the last version published in the gLite web pages for each metapackage.

2. Read carefully the mail sent by PPS announcing a new production release. In the mail, you'll have the information about the update number and the list of patches that should be moved to production.

3. Before working with glite-release in CVS in your machine, run cvs update to make sure you have the last version.

4. For each patch in the mail:

• This should happen when moving patches to PPS, but until the process becomes more stable, doublecheck in the list of associated bugs, if any, that the bug status is meaningful. Remove Invalid/Obsolete bugs. Make sure that in case there are unfixed bugs there's a Known issues section mentioning it in the Release Notes. Add it yourself otherwise. At this point the Release notes should be in good shape but double check again and ask for an improved test to the patch submitter if the quality is poor or the explanations are not very useful.

• Move the patch into production:

  ./scripts/move-patch -s prod PATCHNUMBER

• Check the metapackages affected by the patch have included the updated versions of the relevant rpms. Only the affected metapackages listed in the patch should change:

  cvs diff prod-lists/*

• Commit the changes to the production list:

  cvs commit -m "Applied patch #PATCHNUMBER prod-lists/<platform>*

• Mark the Release field in the corresponding Savannah patch with

  3.1.0 Update <b>NN</b>

5. At this point search in Savannah for all the patches whose Release field is 3.1.0 Update NN. Double check with the original PPS mail that you don't forget any patch!

6. Create the updateNN.prod file:

scripts/list-updated-components -r prod -u NN -p <platform>

7. Check the updateNN.prod file:

scripts/print-release updateNN.prod <platform> check

Only the metapackages that may have dependencies on rpms that change in this release should appear. This is difficult to track when the release contains several patches but it's good to do some extra checking at this point. Get familiar with the patches and with the affected metapackages and try to detect some inconsistencies.

8. Make the Release. If you don't want to tag the release at this point, please run the command with the option -s.

 scripts/make-release -l updateNN.prod -p <platform>

It's important to check that the relevant rpms have been properly copied in each affected metapackage directory in /afs/cern.ch/project/gd/www/glite/prepare/prod/3.1/.

9. Copy the production release from the prepare area:

cd /afs/cern.ch/project/egee/gLite
rm -Rf R3.1_old
cp -pR  /afs/cern.ch/project/gd/www/glite/prepare/prod/3.1 R3.1_new 

10. Generate the Release Notes:

• You need a clean machine with org.glite.integration.release module from CVS. It contains the necessary scripts to create the release notes. If the machine is a clean one, also manually install yum install openldap-servers.
• Export the patches in Savannah by querying for 3.1.0 Update NN in Savannah Patches Export. This takes a while. Note that this is a funny tool. To make sure you get the proper xml file, select the preproduction query form and only fill in the Release field to search for 3.1.0 Update NN.
• Download the Savannah export file in the machine and reformat it if needed. Normally you need to manually add special headers like:

     <custom_text_area_9>
     &lt;h5&gt;List of affected metapackages separated by comas&lt;/h5&gt;
     </custom_text_area_9> 

• Generate the release notes (this takes a while in 32bit due to the high number of node types) by using:

     ./createReleasePages.sh --date=dd.mm.yyyy --gliteRelease=3.1 
     --update=NN --patchFile=exportedfile; --priority=Normal

11. Upload the Release Notes in the web server by running:

 
   ./scripts/uploadReleaseNotes.sh --gliteVersion=3.1 --user=CERN\\glbuild

This will not overwrite the existing pages but will copy the new ones with different names (e.g. glite-BDII_new.asp)

12. Apply manual modificiations to present the information in a coherent away. In particular:

• Main page for 32Bit and main page for 64Bit
  • Check that a summary for each patch is present.
  • Check the summary for each patch reflects the highlights of the patch and that contains a list of affected metapackages. Make sure the text is well presented and it's coherent within the context of the web page. Text in the Release Notes field of the Savannah patch is copied and pasted here and sometimes it needs some changes to make it fit within the overall documentation.
  • Check the list of patches is complete according to the original mail sent by PPS.
  • Check if the new service packages reflect the output from print-release updateNN.prod.
  • Manually add an introduction comment like:

   This update contains multiple bug fixes in various areas. Please see below for a 
   summary of the released patches, a list of the  patches and the service update 
   table linking to each individual servide update containing specific information.

• Service specific update page relevant to the current release.
  • Check the release notes make sense for the service and that only the patches affecting the service are actually described.
  • Add any useful information like: known issues reported by PPS in its release reports, configuration changes (new YAIM variables), etc.
  • Check the list of bugs is correct.
  • Check the list of rpms is correct by comparing it with the rpm list in the relevant patches. Add/remove any rpm if necessary.

12. Finally check the rpms in the repository:

• All the rpms for the released patches are present.
• No other new rpms are included by mistake.

13. Send mail to sa1-release-test. They'll check that everything is OK and they'll give the green light to publish the release. The mail should be something like:

Dear all,

please find a preview on the production release notes here:
http://glite.web.cern.ch/glite/packages/R3.1/updates_new.asp
http://glite.web.cern.ch/glite/packages/R3.1/x86_64/updates_new.asp

To get the details of the update, please use "Details" in the service node listing. 
You will not see any update on the service node main page.

The corresponding RPM packages can be found here:
http://linuxsoft.cern.ch/EGEE/gLite/R3.1_new/
(this location will be renamed when the update is officially released)

Any input/feedback provided as a reply to this mail would be valuable and is therefore welcome. 

Cheers,
The integrator

14. Publish the new repository (every 30min past full hour glitesoft is synchronized):

cd /afs/cern.ch/project/egee/gLite
mv R3.1 R3.1_old
mv R3.1_new R3.1

15. Publish the release notes by executing /mnt/glite-repository/packages/3.1/publishRelease.sh. If there's some delay between the moment of publication and the moment of creation of the pages, please make sure the date associated with the update corresponds to the date when you are publishing it and not earlier.

17. For each release patch change the Savannah status to "In production".

18. Announce the Release using the CIC portal.

• Go to the CIC portal and select Send a EGEE Broadcast.
• Then select Broadcast Information from ROC section (although the broadcast form is the same for all).
• Select News publication in all CIC portal views YES
• Select CIC-on-Duty, OSG, ROC managers, VO managers, Production site admins.
• Add in the CC: support-eis@cernNOSPAMPLEASE.ch,glite-announce@cern.ch.

SUBJECT:

 Release of UPDATE nn to gLite 3. x. Priority: [Normal/HIGH/**URGENT**] 

MESSAGE:

Dear members of the EGEE Grid Production Service,

UPDATE nn for gLite 3.x has been released to the EGEE Grid production service.
The priority of the updates is: [Normal/HIGH/**URGENT**]

All details of the update can be found in:
  http://glite.web.cern.ch/glite/packages/R3.x/updates.asp
(services supported on 32b architectures)
and
http://glite.web.cern.ch/glite/packages/R3.x/x86_64/updates.asp
(services supported on 64b architectures)

The updated middleware services with new version numbers are given on the Updates web page.


For more detailed information including fixed bugs, updated RPMs, configuration changes and
how to deploy, please go to the 'Details' link next to each service on the 'Updates' web page.


** PLEASE ALSO READ THE NOTES AND 'KNOWN ISSUES' SECTION FOR EACH SERVICE. **


Note that the UI/WN tarball is available from the individual service pages
(click on the service name on the main page). 

Remember to report any issues with this set of updates using GGUS:  www.ggus.org



Best regards,
    The gLite Middleware Teams

-- MariaALANDESPRADILLO - 15 Jan 2009