Production Release Check List

Introduction

The following list must be used in combination with the Integration Procedure twiki that describes all the technical details involved in the release process. This twiki aims to list the necessary steps to create a production release helping to avoid mistakes and decrease human errors.

Check List template

Please, print the following check list templates to go through the production release process:

• Prod check list template

Pre requirements

The following machines with mysql-server and mysql client installed are needed in order to do a production release:

Repo machines:

• glite 3.1, 32bit: SLC4/32bit repo machine
• glite 3.1, 64bit: SLC4/64bit repo machine
• glite 3.2, 64bit: SL5/64bit repo machine. Iinstall your GPG key to sign rpms.

Repo machines should all have installed:

• glite-release CVS module
• mysql-server
• mysql client

Release notes machines:

• glite 3.1, 32bit: SL4/32bit release-notes machine
• glite 3.1, 64bit: SL4/64bit release-notes machine
• glite 3.2, 64bit: SL5/64bit release-notes machine (PyXML is needed if it's not installed by default).

Release notes machines should all have installed:

• org.glite.integration.release CVS module
• mysql-server
• mysql client installed
• linkchecker (Only SL5).
• java 1.5.0 (only SL4)
  • Check that you have the CERN-only repofile in /etc/yum.repos.d
  • yum -y install jpackage-utils
  • yum -y --enablerepo CERN-only install java-1.5.0-sun java-1.5.0-sun-devel

Check List

1. cd glite-release in your repo machine and run cvs update to make sure you have the last version. Make sure that in config.sh, GLITE_RELEASE contains the right value of the release you are doing

2. Check the prepare area is equal to the production one. Use the check_prod_all script for this. Otherwise delete the current prepare area and copy from production:

cp -pR /afs/cern.ch/project/egee/gLite/R3.x /afs/cern.ch/project/gd/www/glite/prepare/prod/3.x

7. For each patch in the mail:

• Check in the list of associated bugs, if any, that the bug status is Fix certified or Fix not certified. Remove Invalid/Obsolete bugs. Contact certifiers if there are bugs in Ready for test since they should update this status after verifying the bug.

• Check the release notes again and ask for an improved text to the patch submitter if the quality is poor or the explanations are not very useful.

• Check if any of the following packages are released in the patch and are affecting WN and UI in SL5/64bit, or WN in SL4/64bit: CGSI_gSOAP_2.7-voms CGSI_gSOAP_2.7 DPM-client DPM-interfaces2 DPM-interfaces GFAL-client glite-security-voms-api-cpp glite-security-voms-api-c lcg-dm-common lcg_util LFC-client LFC-interfaces2 LFC-interfaces vdt_globus_essentials libdcap libdcap-devel libdcap-tunnel-gsi. In this case, check if the i386 packages are already there, and if not, follow the steps described in this How to, to deal with group install patches. ( UNTIL BUG:56188 IS FIXED)

8. At this point search in Savannah for all the patches whose Release field is 3.1.0 Update NN*. Double check with the original Operations mail that *you don't forget any patch!

11. Populate the org.glite.integration.release/noarch/lists/metapackages.R3X.list file in the release-notes machine. The file should just contain a plain list of affected metapackages (without versions). Make sure you do a cvs update before in the machine to make sure you use the last version of org.glite.integration.release.

12. Make the Release

Retrieve the scripts with:

svn co https://svn.cern.ch/reps/grelscripts

Move the patch to the production repo:

preview_to_production.py  -r <release> -o <os> -a <arch> <patch [... patch]>

13. In the release-notes machine, run the following script to create the xml file ( R3.X.0-NN.xml) that will contain the savannah information for every patch in the release:

./scripts/release_notes_xml.sh -r 3.X.0 -u NN

Before executing the script the following packages need to be installed: tidy and txt2html. That can be achieved with:

yum install tidy                # SL4/5 repository needed
cpan install HTML::TextToHTML        # Anwser "no", to the manual configuration, the default option for all the other questions should work

You can check the spelling using:

aspell -c R3.1.0-04_i386.xml

It's a good idea to save the common words and put links to ~/.aspell.en.prepl and ~/.aspell.en.pws to somewhere in AFS to keep them between releases.

14. In the release-notes machine, launch the following script under org.glite.integration.release/noarch. It takes a long time to be executed and it will save you some time to run it at this point.

3.1 32bits: ./scripts/RepositoryRpmList.sh --gliteVersion=3.1 --afsRepo=/afs/cern.ch/project/gd/www/glite/prepare/prod/3.1/ 
                                                             --newReleaseDir=temp/newRepository_RpmList --metaPackFile=lists/metapackages.R3.1.list --os=sl4 --arch=i386 --reltype=prod

3.1 64bits: ./scripts/RepositoryRpmList.sh --gliteVersion=3.1 --afsRepo=/afs/cern.ch/project/gd/www/glite/prepare/prod/3.1/ 
                                                             --newReleaseDir=temp/newRepository_RpmList --metaPackFile=lists/metapackages.R3.1.list --os=sl4 --arch=x86_64 --reltype=prod

3.2 64bits: ./scripts/RepositoryRpmList.sh --gliteVersion=3.2 --afsRepo=/afs/cern.ch/project/gd/www/glite/prepare/prod/3.2/ 
                                                             --newReleaseDir=temp/newRepository_RpmList --metaPackFile=lists/metapackages.R3.2.list --os=sl5 --arch=x86_64 --reltype=prod

15. Copy the production release from the prepare area:

cd /afs/cern.ch/project/egee/gLite
rm -Rf R3.X_old
cp -pR  /afs/cern.ch/project/gd/www/glite/prepare/prod/3.1 R3.X_new 

16. Run installation tests.

17. Create the tarballs if the UI and/or the WN are released in this bundle.Bear in mind that you can reuse the tarballs from staged rollout if they haven't changed.

In the machine lxbrf2713, you should run the following scripts (under glite-release/scripts/chroot_test_scripts):

./create_tarball -r 3.X -a arch -p glite-UI
./create_tarball -r 3.X -a arch -p glite-WN

The scripts create two tarball files under glite-release/scripts/chroot_test_scripts/tarballs:

glite-UI-3.X.Y-0.tar.gz 
glite-UI-3.X.Y-0-external.tar.gz 

glite-WN-3.X.Y-0.tar.gz 
glite-WN-3.X.Y-0-external.tar.gz 

These tarball files have to be copied in:

/afs/cern.ch/project/gd/www/download/relocatable/glite-UI/(SL4_i686 or SL5_x86_64) 
/afs/cern.ch/project/gd/www/download/relocatable/glite-WN/(SL4_i686,SL4_x86_64 or SL5_x86_64) 

18. Generate the Release Notes:

• Format the file R3.X.0-NN.xml to improve the layout and the contents of the release notes.
• Generate the release notes. Check with Operations whether the priority should be high in case any patch is marked with High in the priority field, otherwise it should be Normal:

  3.1 32bits : ./createReleasePages.sh --date=dd.mm.yyyy --gliteRelease=3.1 --update=NN --patchFile=exportedfile 
                                                       --priority=Normal --arch=i386 --os=sl4 --reltype=prod

   3.1 64bits: ./createReleasePages.sh --date=dd.mm.yyyy --gliteRelease=3.1 --update=NN --patchFile=exportedfile 
                                                       --priority=Normal --arch=x86_64 --os=sl4 --reltype=prod

   3.2 64bits: ./createReleasePages.sh --date=dd.mm.yyyy --gliteRelease=3.2 --update=NN --patchFile=exportedfile 
                                                       --priority=Normal --arch=x86_64 --os=sl5 --reltype=prod

19. Upload the Release Notes in the web server by running:

 
  3.1 32bits :   ./scripts/uploadReleaseNotes.sh --gliteVersion=3.1 --user=CERN\\glbuild --arch=i386 --os=sl4 --reltype=prod

  3.1 64bits :   ./scripts/uploadReleaseNotes.sh --gliteVersion=3.1 --user=CERN\\glbuild --arch=x86_64 --os=sl4 --reltype=prod

  3.2 64bits :   ./scripts/uploadReleaseNotes.sh --gliteVersion=3.2 --user=CERN\\glbuild --arch=x86_64 --os=sl5 --reltype=prod

This will not overwrite the existing pages but will copy the new ones with different names (e.g. glite-BDII_new.asp)

20. Apply any modifications in R3.X.0-NN.xml file until you are satisfied with the release notes. Then regenerate the release pages with the --skipRPMgen option (since the part of the script that creates the list of released packages doesn't need to be repeated again). This is what you need to execute in each iteration until the contents of the release notes are OK:

  3.1 32bits : ./createReleasePages.sh --date=dd.mm.yyyy --gliteRelease=3.1 --update=NN --patchFile=exportedfile 
                                                       --priority=Normal --arch=i386 --os=sl4 --reltype=prod --skipRPMgen 
                   ./scripts/uploadReleaseNotes.sh --gliteVersion=3.1 --user=CERN\\glbuild --arch=i386 --os=sl4 --reltype=prod


   3.1 64bits: ./createReleasePages.sh --date=dd.mm.yyyy --gliteRelease=3.1 --update=NN --patchFile=exportedfile 
                                                       --priority=Normal --arch=x86_64 --os=sl4 --reltype=prod --skipRPMgen
                   ./scripts/uploadReleaseNotes.sh --gliteVersion=3.1 --user=CERN\\glbuild --arch=x86_64 --os=sl4 --reltype=prod


   3.2 64bits: ./createReleasePages.sh --date=dd.mm.yyyy --gliteRelease=3.2 --update=NN --patchFile=exportedfile 
                                                       --priority=Normal --arch=x86_64 --os=sl5 --reltype=prod --skipRPMgen
                   ./scripts/uploadReleaseNotes.sh --gliteVersion=3.2 --user=CERN\\glbuild --arch=x86_64 --os=sl5 --reltype=prod

21. Add any known issues by editing manually the relevant metapackage pages. In order to do that you need to mount the gLite web page in your release-notes machine:

mount -t smbfs -o username=CERN\\glbuild //glite.web.cern.ch/glite /mnt/glite_repository

3.1 32bits: vi /mnt/glite_repository/packages/R3.1/deployment/glite-metapackage/glite-metapackage-known-issues.asp

3.1 64bits: vi /mnt/glite_repository/packages/R3.1/x86_64/deployment/glite-metapackage/glite-metapackage-known-issues.asp

3.2 64bits: vi /mnt/glite_repository/packages/R3.2/sl5_x86_64/deployment/glite-metapackage/glite-metapackage-known-issues.asp

.

22. Do a final check of the contents of the release pages:

• Main page:
  • gLite 3.1 32bits
  • gLite 3.1 64bits
  • gLite 3.2 64bits
    • Check that a summary for each patch is present.
    • Check the summary for each patch reflects the highlights of the patch and that contains a list of affected metapackages. Make sure the text is well presented and it's coherent within the context of the web page. Text in the Release Notes field of the Savannah patch is copied and pasted here and sometimes it needs some changes to make it fit within the overall documentation.
    • Check the list of patches is complete according to the original mail sent by Operations.
    • Check the priority of patches is correct. If a patch is priority High the corresponding affected services should be in red, stating High.
    • Check the new versions of the affected metapackages correspond with the output of print-release updateNN.prod.

• Service specific update pages:
  • Check the release notes make sense for the service and that only the patches affecting the service are actually described.
  • Check the list of rpms is correct by comparing it with the rpm list in the relevant patches.
  • Check the list of bugs is correct by comparing it with the bug list in the relevant patches.
  • Check the URLs of the rpms are correct, in the update page and in the rpm list page (both HTML and txt).

• Some known issues to be checked manually:
  • In the rpm.list.txt and html pages of the cream CE there is sometimes a typo on some rpm versions. Check there's no 1:. Ocurrences of 1: should be deleted.

23. Send a preview mail to grid-deployment-managers, project-eu-egee-middleware-emt and gd-release-team. In the subject write preview production update XX. They'll check that everything is OK and they'll give the green light to publish the release. The mail should be something like:

Dear all,

please find a preview of the production release notes here:
http://glite.web.cern.ch/glite/packages/R3.1/updates_new.asp and/or
http://glite.web.cern.ch/glite/packages/R3.1/x86_64/updates_new.asp or
http://glite.web.cern.ch/glite/packages/R3.2/sl5_x86_64/updates_new.asp

Any input/feedback provided as a reply to this mail would be valuable and is therefore welcome. 

Cheers,
The integrator

24. In case any of the patches provide a fix for a security issue identified by GSVS, contact Andreas to get the URL for the Security Advice. Modify the web pages to include this URL as soon as it's available.

25. Publish the new repository (every 30min past full hour glitesoft is synchronized):

cd /afs/cern.ch/project/egee/gLite
mv R3.X R3.X_old
mv R3.X_new R3.X

26. Publish the release notes by executing /mnt/glite_repository/packages/R3.X/((sl5)_x86_64)/publishRelease.sh. If there's some delay between the moment of publication and the moment of creation of the pages, please make sure the date associated with the update corresponds to the date when you are publishing it and not earlier.

27. Make sure there are no broken links (this has to be executed always in an SL5 machine):

• Modify the corresponding noarch/scripts/check_links/R3.X_iarch_preview.list where arch is either i386 or x86_64.
• Execute ./check_links.sh -r 3.X -a arch and check the results.
• Fix any broken links.

28. For each release patch change the Savannah status to In production. Change the attached bugs status to Ready for Review. This can be done using the SCLI. Please, download the script in the location where you normally have org.glite.savannah.viewer/cli/:

svn co http://svnweb.cern.ch/guest/aelwell/savannah_cli/trunk
cd trunk/
 ./ReadyForReview.sh -p PATCH_NUMBER

29. If there are new node types released, remember to:

• Create the repo files under http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.X/.
• Add in the Generic Install Guide any relevant information about the new node type where needed.
• Add in the http://glite.web.cern.ch/glite/packages/R3.X/ a new entry for the new node type.

30. Announce the Release using the CIC portal.

• Go to the CIC portal and select Send a EGEE Broadcast.
• Then select Broadcast Information from ROC section (although the broadcast form is the same for all).
• Select News publication in all CIC portal views YES
• Select CIC-on-Duty, OSG, ROC managers, VO managers, Production site admins.
• Add in the CC: support-eis@cern.ch, glite-announce@cern.ch, glite-discuss@cern.ch, gd-release-team.

SUBJECT:

 Release of UPDATE nn to gLite 3. x. Priority: [Normal/HIGH/**URGENT**] 

MESSAGE:

Dear members of the EGEE Grid Production Service,

UPDATE nn for gLite 3.x has been released to the EGEE Grid production service.
The priority of the updates is: [Normal/HIGH/**URGENT**]

The highlights of the update are:

-
-

All details of the update can be found in:
[COPY AND PASTE updates URL here]

Note that the UI and WN tarballs are available from the UI and WN pages.

Remember to report any issues with this set of updates using GGUS: www.ggus.org

Best regards,
The gLite Middleware Team

31. Remove from /afs/cern.ch/project/gd/www/glite/certified/3.X/patches/slX/arch all the RPMS.XXXX files corresponding to any released patch. Remove the patch also from /afs/cern.ch/project/gd/www/glite/cert/3.X.

32. Recreate the preview repository if not all the patches in the preview have been moved to production. Follow the Beta check list to do this.

33. If all the patches in the preview have been moved to production:

• update the preview web pages to announce there are no preview updates available. This can be done automatically by running in the release notes machine:

  /mnt/glite_repository/preview/R3.X/((sl5)_x86_64)/closeRelease.sh

• Update the metapackages repos to point to the empty repo by running:
  • In prepare/beta/3.X, remove all the existing directories and leave only empty.
  • Manually modify the script in scripts/create-empty-repo (the for loop at the end) to define those metapackages that are not affected by the update and that should point to empty.
  • scripts/create-empty-repo -r 3.X -a iarch -o os
  • copy the prepare/beta/3.X to the real beta location

-- MariaALANDESPRADILLO - 15 Jan 2009