Middleware repository update procedure

Introduction

We operate apt-get and yum repositories for the grid middleware and the CA rpms. The apt-get repository is used by yaim. The yum repositories exist in two forms, yum 2.2 and yum 2.4. The repositories all exist under the same AFS directory and are generated from the rpm lists in the middleware CVS. Updates are added manually.

Meta rpms

The middleware required for each node type is captured in the dependencies of a 'meta rpm', which itself contains no software. This meta rpm, for example lcg-CE.rpm, is generated from the rpm list in the CE-rpm file in CVS. There is also a CA-rpm file which holds the current list of certification authority rpms.

Required software

In order to manage the repositories, you need to have a shell on a machine which can access AFS and write to the relevant directories. Furthermore, you will need to have the appropriate utilities available for the generation of each repository; genbasedir, yum-arch and createrepo. These are available from the SL3 packages 'yum' and 'apt', and the SL4 package createrepo which can safely be installed on SL3. These utils are not available on lxplus! lxb1712 has all required tools.

Generating a repository from scratch

Update of rpm list for apt generation

The following rpm lists are used

• certification; lcgdeploy.cvs.cern.ch:/cvs/lcgdeploy/lcg-release/rpmlist_sl3
• release; lcgdeploy.cvs.cern.ch:/cvs/lcgdeploy/lcg2/rpmlist_sl3
• CA; lcgdeploy.cvs.cern.ch:/cvs/lcgdeploy/lcg2/rpmlist

The *-rpm files in these folders define what goes into the meta-rpms. These files will have to be updated in order to build a repository.

Generating the repositories

There is a script you can use to generate the repositories. Check out lcgdeploy.cvs.cern.ch:/cvs/lcgdeploy/lcg-scripts/yaim/server and use repgen.pl. This script checks out a particular tag from CVS, builds the metarpms from the files there, puts the rpms in the repository, puts all the middleware in the repository and then generates the apt and yum repositories in the directory.

The CA and release repository is held under

/afs/cern.ch/project/gd/www/gis/apt

The cert testbed one is under

/afs/cern.ch/project/gd/www/apt-cert

To build a repository based on the LCG-2_7_0 tag, producing meta rpms of version 2.7.0 for sl3, issue the following command

./repgen -t LCG-2_7_0 -v 2.7.0 -m mw -o sl3

This will produce the repository under /afs/cern.ch/project/gd/www/gis/apt/LCG-2_7_0/en/i386 with the files necessary for it to be accessible via both apt and yum.

Updating the main repository

This should not normally be done - use the updates folder. If this is necessary though, rebuild replacing the lcg_sl3.updates with lcg_sl3;

RELEASE=LCG-2_7_0

genbasedir --flat --bloat --bz2only --partial /afs/cern.ch/project/gd/www/gis/apt/$RELEASE/sl3/en/i386 lcg_sl3

yum-arch -l /afs/cern.ch/project/gd/www/gis/apt/$RELEASE/sl3/en/i386/RPMS.lcg_sl3

createrepo /afs/cern.ch/project/gd/www/gis/apt/$RELEASE/sl3/en/i386/RPMS.lcg_sl3

CAs

To create the CA repository based on the tag LCG_CA-0_28 and with the lcg-CA rpm at version 0.28, do

./repgen -t LCG_CA-0_28 -v 0.28 -m CA

Make sure that CA-rpm and security-rpm.h have the appropriate tag (in this case LCG_CA-0_28).

The repository will be built under /afs/cern.ch/project/gd/www/gis/apt/LCG_CA/en/i386/.

Procedure for CA update

The general procedure can be found here : https://wiki.egi.eu/wiki/EGI_IGTF_Release_Process

A script which executes the steps below can be found here : http://glite.cvs.cern.ch:8180/cgi-bin/glite.cgi/org.glite.integration.release/

For the integration team part these are the steps:

• Install EUgridPMA public key: wget https://dist.eugridpma.info/distribution/igtf/current/GPG-KEY-EUGridPMA-RPM-3 && rpm --import GPG-KEY-EUGridPMA-RPM-3
• create directory in /afs/cern.ch/project/egee/gLite/APT/LCG-CAs (usually in the format x.yy-z
• create RPMS.production in /afs/cern.ch/project/egee/gLite/APT/LCG-CAs/x.yy-z/
• wget rpms from new CA repo (announced via GGUS ticket) into RPMS.production.
• Copy the dummy-ca-certs-20090630-1.noarch.rpm from a previous CA update into RPMS.production as well.
• Sign the dummy-ca-certs-20090630-1.noarch.rpm with our key
• run createrepo /afs/cern.ch/project/egee/gLite/APT/LCG-CAs/x.yy-z/
• run genbasedir --flat --bloat --partial /afs/cern.ch/project/egee/gLite/APT/LCG-CAs/x.yy-z/
• run yum-arch -l /afs/cern.ch/project/egee/gLite/APT/LCG-CAs/x.yy-z/
• run

  cd /afs/cern.ch/project/egee/gLite/APT/LCG-CAs/x.yy-z/
  echo "<HTML><BODY>" >index.html
  for i in `ls RPMS.production/`; do echo "<a href="RPMS.production/$i">$i</a><br/>" ; done >> index.html
  echo "</BODY></HTML>" >> index.html 

• create /afs/cern.ch/project/gd/www/lcg-CAs-vx.yy-1
• wget lcg-CA-x.yy-z.txt, pro_software_meta_lcg_CA.list, pro_software_meta_lcg_CA.tpl and cas.tpl from the URL specified in the GGUS ticket and copy them into /afs/cern.ch/project/gd/www/lcg-CAs-vx.yy-1
• create /afs/cern.ch/project/gd/www/lcg2CAlist_new.html (use the previous as template) containing the right versions and dates
• announce email to david@lipNOSPAMPLEASE.pt and sam-support@cernNOSPAMPLEASE.ch (template below here ) and put a comment in the GGUS ticket: Mail sent to Mario David with a preview of the release so they can test it. Waiting for green light from him to proceed.
• WAIT for Mario David to confirm the tests were OK.
• When Mario David confirm tests are OK, assign the GGUS ticket to SAM with the following comment: Tests are OK as confirmed by PPS. Assigning the ticket to SAM to wait for their confirmation before releasing to Production.
• WAIT for SAM to confirm they are OK and ready to release. The GGUS ticket is again assigned to us.
• update current symbolic link in /afs/cern.ch/project/egee/gLite/APT/LCG-CAs/current pointing to the new location /afs/cern.ch/project/egee/gLite/APT/LCG-CAs/x.yy-z
• Rename lcg2CAlist.html in /afs/cern.ch/project/gd/www to lcg2CAlist.html.yyyymmdd (the date where the file was created).
• Copy lcg2CAlist_new.html to lcg2CAlist.html.
• Send the EGEE broadcast (select "To ROC Managers", "To Production Site Admin" and "To Pre-Production Site Admin"), and ccing the glite-announce mailing list. Use this template.
• After this, assign the GGUS ticket to Security Management SU.

PPS Email Template

Dear all,

Please find a preview for the new CA’s version x.y-z available here. YUM repo file should contain:

[CA]
name=CAs
baseurl=http://linuxsoft.cern.ch/LCG-CAs/x.y-z

The release notes can be found under:
http://grid-deployment.web.cern.ch/grid-deployment/lcg2CAlist_new.html

As soon as it has been approved by PPS and SAM team the repo will be enabled under:
http://glitesoft.cern.ch/LCG-CAs/current/RPMS.production/

and the release notes under:
http://grid-deployment.web.cern.ch/grid-deployment/lcg2CAlist.html

Publishing updates

Updates should be added manually to the updates folder.

If any file is changed or added to an apt/yum repository, the repository must be rebuilt otherwise there will be an inconsistency between the indexed information and the directory contents.

If you have added a file to the updates folder, you can rebuild a modified repository like this.

RELEASE=LCG-2_7_0

genbasedir --flat --bloat --bz2only --partial /afs/cern.ch/project/gd/www/gis/apt/$RELEASE/sl3/en/i386 lcg_sl3.updates

yum-arch -l /afs/cern.ch/project/gd/www/gis/apt/$RELEASE/sl3/en/i386/RPMS.lcg_sl3.updates/

createrepo /afs/cern.ch/project/gd/www/gis/apt/$RELEASE/sl3/en/i386/RPMS.lcg_sl3.updates/

DO NOT put anything in the updates folder which break sites which upgrade automatically. Putting a fix here which is enabled by later admin intervention is fine.

All updates must be associated with an announcement to the ROLLOUT and the ROCs, and a news entry on the CIC portal. You can use their broadcast tool to do all this at once. Take a look at earlier news entries about updates to see the information required. Note that the CIC site is protected by certificate.

Repgen options

The full list of repgen options is as follows

-t TAG_name : the tag to check out from CVS
-v rpm_version : the version number of the rpms to be generated
-m [mwCA] : operate in middleware (mw) or CA (ca) mode
-o [rh73sl3] : operating system (mw mode only)
-r : recheck RpmDir on AFS for new middleware
-b : build rpms only, do not create the repositories
-d repository top-level directory : create repository with this name, rather than the default which is the TAG supplied
-n node_type : generate the rpm for this node type only, not the full set. This option will not remove existing meta rpms from the repository
-f apt_folder : generate the whole thing under this directory, useful for testing
-c CVS path : path under lcgdeploy.cvs.cern.ch:/cvs/lcgdeploy from which to checkout
-i [i386|ia64] : architecture - default is i386
-s obsolete this tag in the meta-rpms (test this before using on a live repository, obsoletes can have interesting effects)