Procedures for staged roll-out

This page details technical procedures to be applied by the Operations Team in order to fulfill some of the steps referenced in SA1/SA3workflow

Some of them make use of a tool called ASTAS (Automatic Savannah TAsk Submission). Please refer to the ASTAS Tutorial for further developments and customisation.

Create the Roll-out tasks and assign them to the EA Sites

• Connect to an afs-enabled node
  

  Prerequisite

  access to node with afs client and MySql client libraries

• Change to the ASTAS directory
  > cd /afs/cern.ch/project/gd/egee/www/preproduction/ActivityManagement/astas

• Load environment variables
  > source set_env.csh

• Delete the Savannah cookie if it exists (otherwise, you could send the reports with the username of the last user who executed the script). NOTE: the removal can be done only by the owner of the cookie
  > rm sav_cookie

• Create the bundle configuration file
  • A python script is used to perform this step. It will
    • query savannah database for patches included in an update (PPS release or staged roll-out selectable)
    • determine which services are affected by an update
    • query PPS Registry to determine which sites are involved in testing activity (pre-deployment or early adopters)
    • create config XML file where it will write those information in systematic way.
    • NOTE: writing access to the afs directory tree '../../astas' is needed for that This config file is to be used as input for astas_prepare_xmls.py script in order to produce XML files for bulk submision to savannah
  • Help Command
    > python astas_conf_creator.py  --help
  • Syntax
    

    >  python astas_conf_creator.py -r 3.Y.0 -u NN -f SRO-update-config/glite3.Y.0BundleNN.conf
    DB username: saview 

  • Example
    

    >  python astas_conf_creator.py -r 3.1.0 -u 01 -f SRO-update-config/glite3.1.0Bundle01.conf
    [INFO]  File 'SRO-update-config/glite3.1.0Bundle01.conf' is openned for writing.
    DB username: saview
    DB password:
    Configuration file successfully saved to SRO-update-config/glite3.1.0Bundle01.
        

• Inspect the bundle configuration file (to spot possible anomalies)

  >  less SRO-update-config/glite3.Y.0BundleNN.conf 

• Produce source xml files for the tasks
  • A python script is used to perform this step. It will use the configuration file produced in the previous step and a predefined template file in order to produce a numbers of xml files, ready for bulk submission submission with savannah_my script
  • Help Command
    > python astas_prepare_xmls.py --help

NOTE : at this stage you have to modify the duration to one week (the default is one day). Check the option in astas-prepare

• • Syntax
    

    > python astas_prepare_xmls.py -d update -c SRO-update-config/glite3.Y.0BundleNN.conf -t template-staged-roll-out.xml

  • Example
    

    > python astas_prepare_xmls.py -d update -c SRO-update-config/glite3.1.0Bundle01.conf -t template-staged-roll-out.xml

    
    

NOTE

the template does not need to be changed every time. If you need to modify or to create a new one please have a look at the example.

• Inspect the XML files in the 'update' directory (to spot possible anomalies)

• Submit the tasks * The savannah_my script is used to interface to savannah tracking system and perform this step. It will use the XML files produced in the previous step.
  • Help Command
    > python savannah_my.py --help * Syntax
    

    > python savannah_my.py -o submit_all -g sa1dep -t task -d update/3.Y.0_Bundle_NN/ 
    Savannah username: use yours

  • Example
    

    > python savannah_my.py -o submit_all -g sa1dep -t task -d update/3.1.0_Bundle_01/  
    Savannah username: aretico
    Savannah password:

Publish the Roll-out reports

To Publish the reports

• Connect to an afs-enabled node
  

  Prerequisite

  access to node with afs client and MySql client libraries

• Change to the ASTAS directory
  > cd /afs/cern.ch/project/gd/egee/www/preproduction/ActivityManagement/astas

• Load environment variables
  > source set_env.csh

• Create the site reports
  > python ./astas_report_creator.py -d update/3.1.0_Bundle_02 -r /afs/cern.ch/project/gd/egee/www/preproduction/ActivityManagement/REPORTS/

Example of task template creation

Simple templates for tasks can be created using an interactive utility of the savanna_my.py script. The following example shows the creation of a template for deployment test in PPS

• Connect to an afs-enabled node
  

  Prerequisite

  access to node with afs client and MySql client libraries

• Change to the ASTAS directory
  > cd /afs/cern.ch/project/gd/egee/www/preproduction/ActivityManagement/astas

• Load environment variables
  > source set_env.csh

• Create the template (example)
  

  > python savannah_my.py -o xml_template -g sa1dep -t task -f template-dep_test.xml -I 

• Possible paragraphs/Questions to answer
  

  day -->skip
  month -->skip
  year -->skip
  Category: (MQS): 103
  Group_ID: 103
  Assigned: 105
  Privacy: (Public): 1
  Assign to: -->skip
  Field hours: -->skip (4 hours by default)
  Summary: -->skip
  Details: -->skip
  Custom_ta1: -->skip
  input_file(attachments): -->skip
  add-cc: -->skip
  cc-comment:: -->skip 

Advanced configuration is possible using template variables (this was applied e.g. to the template for staged roll-out ). Please check the ASTAS Tutorial for the details.

Roll-back the tasks of a bundle

In the previous cases releases were rolled back directly with a comment on the existing staged roll-out tasks adding a comment as in the example below. If there are too man ytasks to manage we can consider the option to open new tasks in the same way as we did to create the original tasks using a special template for roll-back

Hi,

during the staged roll-out of this bundle an issue was found affecting the UI and the LB.
As this bundle is a security update unfortunately we are obliged to roll-back the entire bundle.

Therefore you are kindly requested now to roll-back the installation at your site to the previous version.

The rpms have already been removed from the beta repository, so a possible procedure to roll-back is the following

> manually remove all the packages updated during the upgrade
   (this operation is not supported by yum)
> re-run yum install [your-node-type]

Then please close this task setting the status to "done" and the outcome to "not applicable"

Please contact gd-release-team@cern.ch for any issues with the procedure above

We apologise for the inconvenience and thank you for your understanding

The gLite release Team