Release Process

Below is an outline of how to build a CASTOR release.

STEP1: Preparation

1.1 - Increment the version number

Update the debian/changelog file adding a new entry for the release. For example:

[lxadm06] ~/v2_1_9Version > nano debian/changelog
castor (2.1.9-6) unstable; urgency=low

  * Sixth 2.1.9 production release

-- Dennis Waldron <dennis.waldron@cern.ch> Mon, 17 May 2010 08:00:00 +0100

1.2 - Update the ReleaseNotes

The ReleaseNotes should contain:

• A list of all bug fixes in the release (UPDATED and FIXED)
• A list of all package modifications. If files have changed packages or names it must be documented.
• Recommendations of additional software upgrades e.g. LSF, rsyslog, memcache which CASTOR may utilize and benefit from being upgraded.
• Changes to any configuration files.
• Upgrade instructions detailing the steps to be followed. If some part of the process is likely to take time or is critical to the success of the upgrade then this must be clearly highlighted.

1.3 - Create the database upgrade scripts

One can obtain empty upgrade scripts and a valid DB creation script using :

./makesql.sh
cd upgrades
../tools/makeUpgradeSqlScripts.py 

Then you have to manually fill the upgrade scripts, in particular with schema changes, but also with revalidation of the SQL code.

In order to check that the upgrade scripts are correct, the easiest is to use SQL Developer's database diff tool :

• wipe the scratch_dev01 DB by running drop_oracle_schema.sql on it
• recreate it with the creation script of the previous release
• update it with the fresh upgrade script you want to test
• wipe the scratch_dev02 DB by running drop_oracle_schema.sql on it
• recreate it with the creation script of the release you are building
• open SQL developer and use the diff tool to diff the 2 DBs
  • take care to do it twice, once in each direction : scratch_dev01 to scratch_dev02 and scratch_dev02 to scratch_dev01
  • some differences are normal, e.g. the different names of internal types created by the DB

Finally commit your upgrade scripts to git.

WARNING

• If a schema change needs to be performed (e.g the structure of a table has been modified) verify whether the change is backwards compatible. If not, check that the database schemaVersion has been updated accordingly. If you need to change it, don't forget to change it in the code!

STEP2: Building the software

2.1 - Tag the software in git:

    git tag v2_1_15_1 -m "New tag for release 2.1.15-1"
    git push origin v2_1_15_1

2.2 - Launch the Jenkins build

Make sure that Jenkins has built the commit that you have tagged. This can be checked by looking at the package name : it has to have the correct version, not a big number as the last digit.

In case it did bot build the right commit, drop your tag with :

  git tag -d v2_1_15_1
  git push origin :refs/tags/v2_1_15_1

Then go back to step 2.1

2.3 - Tag the software in Jenkins

TODO use promotion to tag releases...

STEP3: Upgrading the certification instance

3.1 - Identify the right certification instance

There are 2 certification instances right now :

Cert Name	OS	ARCH	CASTOR Version	State
lxc2cert1	CC7	x86_64	2.1.16-0	running under puppet
lxc2cert2	SLC6	x86_64	2.1.16-0	running under puppet

One is dedicated to the last but one major release, for long term support, the other to the most recent major release.

3.2 - Update the certification instance using puppet

The upgrade of the RPMs is automatic if puppet is pointing to the right repo.

In order to update an instance, follow closely your release notes - this exercise is also there to test them. Take care to reenable the tape servers after the upgrade, using something like :

mco shell -T castor -F hostgroup_2=c2cert2 'cat /etc/castor/TPCONFIG | cut -d " " -f 1 | xargs -i tpconfig {} up'

You may have to change parameters of the config files or force the version in the puppet profiles. Check the release notes for this.

In such a case, checkout the it-puppet-hostgroup-castor module and checkout the dev branch :

git clone https://:@gitlab.cern.ch:8443/ai/it-puppet-hostgroup-castor.git
cd it-puppet-hostgroup-castor
git checkout dev

Edit the file in data/hostgroup/castor/c2cert/c2cert.yaml or the more specific files in the subtree data/hostgroup/castor/c2cert/c2cert

When you need to update the RPMs, edit the same yaml file for the os_repository_date entry. Also take care that teamcity_branch is correct.

Once you're done with editing the it-puppet-hostgroup-castor module, commit and push back to origin dev :

git push origin dev

And force the puppet run + yum update:

mco puppet runonce -T castor -F hostgroup_2=c2cert<n>
mco shell -T castor -F hostgroup_2=c2cert<n> 'yum clean all; yum -y update'

You only have to wait a bit now (ok, a lot...)

3.3 - Run the test suite

After the upgrade the test suite should be executed :

  cd /tmp; DIR=`mktemp -d CastorTestSuite.XXXX`; cd $DIR; git clone --depth 1 https://:@gitlab.cern.ch:8443/castor/CASTOR.git; cd CASTOR; git checkout master; cd test/testsuite
  vim castortests/CastorTestSuite.options
  kinit
  grid-proxy-init
  py.test -v --all -k '-fileOverwritten' --tb=short
  cd ../../..; rm -rf $DIR

Note that on CC7 you need a -k= in the py.test line :

  cd /tmp; DIR=`mktemp -d CastorTestSuite.XXXX`; cd $DIR; git clone --depth 1 https://:@gitlab.cern.ch:8443/castor/CASTOR.git; cd CASTOR; git checkout master; cd test/testsuite
  vim castortests/CastorTestSuite.options
  kinit
  grid-proxy-init
  py.test -v --all -k='-fileOverwritten' --tb=short
  cd ../../..; rm -rf $DIR

Everything should be ok OK i.e. all tests have passed but the ones known to fail according to the Release Notes.

STEP4: Making the final release

4.1 - Push the release to AFS and savannah (official release area)

• Run the following commands for which you will need to have access to AFS. The output/artifacts will be written to /afs/cern.ch/project/castor/www/DIST/intReleases/

[lxadm04] ~/CASTOR-tools > cat .git/config | grep url
   url = https://:@gitlab.cern.ch:8443/castor/CASTOR-tools.git
[lxadm04] ~/CASTOR-tools > cd devtools
[lxadm04] ~/CASTOR-tools/devtools > python publishRelease.py -t v2.1.11-1

• If this is a full production release and not an internal release then move the build output from the internal release area to production:

[lxadm04] ~ > mv /afs/cern.ch/project/castor/www/DIST/intReleases/2.1.15* /afs/cern.ch/project/castor/www/DIST/CERN/savannah/CASTOR.pkg/2.1.15-\*/
[lxadm04] ~ > cd /afs/cern.ch/project/castor/www/DIST/CERN/savannah/CASTOR.pkg/
[lxadm04] /afs/cern.ch/project/castor/www/DIST/CERN/savannah/CASTOR.pkg > unlink LATEST-2.1.15
[lxadm04] /afs/cern.ch/project/castor/www/DIST/CERN/savannah/CASTOR.pkg > ln -s 2.1.15-\*/2.1.15-2 LATEST-2.1.15

4.2 - Publish the release in Koji

• publish the release in koji for puppet managed nodes and clients:

For a full production release:

[lxadm04] ~ > koji build castor5 /afs/cern.ch/project/cndoc/wwwds/HSM/CASTOR/DIST/CERN/savannah/CASTOR.pkg/LATEST/SL5/x86_64/castor-*.slc5.src.rpm
[lxadm04] ~ > koji build castor6 /afs/cern.ch/project/cndoc/wwwds/HSM/CASTOR/DIST/CERN/savannah/CASTOR.pkg/LATEST/SL6/x86_64/castor-*.slc6.src.rpm
[lxadm04] ~ > koji build castor7 /afs/cern.ch/project/cndoc/wwwds/HSM/CASTOR/DIST/CERN/savannah/CASTOR.pkg/LATEST/CC7/x86_64/castor-*.el7.cern.src.rpm

For an internal release:

[lxadm04] ~ > koji build castor5 /afs/cern.ch/project/castor/www/DIST/intReleases/2.1.15-12/SL5/x86_64/castor-*.slc5.src.rpm
[lxadm04] ~ > koji build castor6 /afs/cern.ch/project/castor/www/DIST/intReleases/2.1.15-12/SL6/x86_64/castor-*.slc6.src.rpm
[lxadm04] ~ > koji build castor7 /afs/cern.ch/project/castor/www/DIST/intReleases/2.1.15-12/CC7/x86_64/castor-*.el7.cern.src.rpm

Note that this builds both the i386 and the x86_64 versions of the packages. In case of RPM dependency problems with the build, run the following command or ask Linux support to run it for you:

[aiadm] ~ > koji regen-repo castor6-build

If you accidentally built the wrong version of the CASTOR rpms in Koji, then you can remove them from Koji using the following command, of course replacing the version strings castor6 and castor-2.1.15-5391.slc6 with the version you want removed:

koji untag-build castor6-testing castor-2.1.15-5391.slc6

Please note that the untag-build command will delete the rpms from Koji but it will NOT delete the fact that the build number, in this case 5391, has been used and cannot be used again. In order to be able to reuse the build number, please ask the Koji administrators to run the Koji reset command.

Packages built this way end up in the castorN-testing tag, N = {5, 6, 7}. In case you need to tag the packages in a different tag, use the following commands:

[aiadm] ~ > koji add-pkg <tag> castor-2.1.15-3.<dist> --owner=castor-team  
[aiadm} ~ > koji tag-pkg <tag> castor-2.1.15-3.<dist>

In case you have problems using Koji, check your configuration in ~/.koji/config. Here is what you should have :

[koji]
    server = http://koji.cern.ch/kojihub
    weburl = http://koji.cern.ch/koji
    topurl = http://koji.cern.ch/kojifiles
    topdir = /mnt/koji

More details about Koji: BuildingRPMswithKoji.

4.3 - update CASTOR Web site

• open http://castor.cern.ch
• make sure you're signed in (as the proper user)
• click on the "version info" link at the top left of the page
• fill the form
  • version number and version number Title identical : 2.1.15-3
  • Release notes title is "2.1.15-3 Release Notes"
  • fill the date as well as it is used to for the sorting
  • the version number URL is http://castorwww.web.cern.ch/castorwww/DIST/CERN/savannah/CASTOR.pkg/2.1.15-*/2.1.15-3/
  • the Release notes' URL is http://castorwww.web.cern.ch/castorwww/DIST/CERN/savannah/CASTOR.pkg/2.1.15-*/2.1.15-3/ReleaseNotes
  • do not tick "Provide a menu link"

4.4 - Official announcement

• Close all fixed bugs in Jira which are part of the release. This can be done in bulk :
  • go to jira and view only the items to be closed using this selection : "project = CASTOR AND issuetype = Bug AND status = Resolved"
  • click Tools at the top right and choose Bulk Change all Issues
  • select all of them by clicking the box right to the title line, then click "Next"
  • choose "Transition Issues" and click "Next"
  • choose "Close Issues" and click "Next"
  • add a comment "part of release 2.1.15-5" and click "Next" again
  • click "Confirm"

• Announce the release to the official mailing lists: castor-dev, castor-operations, castor-operation-external. Look at previous release annoucements if you're not sure what to write. Typical mail is entitled "CASTOR x.x Release" and has following content :

Dear All,

A new release of the 2.1.14 series of CASTOR (2.1.14-15) is now officially available in the release area
(http://cern.ch/castorwww/DIST/CERN/savannah/CASTOR.pkg/LATEST), including full release notes and upgrade instructions.

Regards,

Giuseppe
for the CASTOR dev team

<include the first part of the release notes here>

Congratulations you have made a release!

Last update: GiuseppeLoPresti 2016-08-12