Ganga Plugin Migration

This page contains proposal for the plugin migration framework. It gives examples for Ganga developers what they may need to do if the old plugin objects stored within jobs in repository have to be converted into plugin objects with a newer class schema.

Schemas compatibility

Every Ganga (plugin) class has a schema. Schemas are distinguished by their versions, which in turn have major and minor numbers, e.g. (m,n), where m is the major and n is the minor number. Two schemas are considered to be incompatible if they have different major version numbers. In all other cases they are deemed to be compatible.

Current implementation

The framework attempts automatic migration of a plugin object stored in the repository providing the schemas of old and new plugin classes are compatible. The automatic migration works only if the intersection of two sets of attributes described in the old and new class schemas stays unchanged. In this case the Ganga framework either ignores old attributes omitted in the new schema or fills new attributes missing in the old schema with default values. Therefore the plugin developer has to ensure that the major version number of the class schema is increased whenever any attribute changes its meaning (in practice the schema category or the sequence flag). For example, if schema of old plugin class describes attributes a and b, and has version number (1,0), it is sufficient to increase just the minor version number of the schema of any new plugin class with attributes a, or a, b, c i.e. (1,0) --> (1,1), providing a and b are the same in both classes. However if one of attributes, say a changes its meaning e.g., from being a File to a sequence of Files than the major version number has to be increased: (1,0) --> (2,0). In this case the Ganga framework will not try to transform the old plugin object, and the new one is created in incomplete state.

The Proposal

In order to support migration of Ganga (plugin) objects in case of incompatible class schemas the developer of the new plugin class will need to implement two additional class methods described below and to define a migration class. For convenience the migration class may be derived from the new plugin class. The only requirement is that the schema of the migration class is compatible with the schema of the old plugin class. The Ganga framework will ask user permission to perform the migration. If Ganga session is not interactive the migration will not take place. Once objects are migrated they can not be reverted to the old version. The migration in case of compatible schemas will stay as in current implementation. The additional class methods to be implemented are the following:
  • getMigrationClass(cls, version). This class method returns the migration class with the schema . Alternatively, it may return a migration class with a schema more recent than schema , but in this case the returned class must have "getMigrationClass" and "getMigrationObject" methods implemented, so that a chain of convertions can be applied.
  • getMigrationObject(cls, obj). This method takes as input an object of the class returned by the "getMigrationClass" method, performs object transformation and returns migrated object of the new plugin class (cls).
In order to provide support for all old plugin objects, the developer of the new plugin class can create a chain of migration classes, each migration class being relevant to the change of the major version number. For example, if the repository contains plugins with schema versions (1,0), (1,1), (2,0), (2,1), and the new plugin has schema version (3,0), the two migration classes have to be created relevant to transformations: (1,1) --> (2,0), and (2,1) --> (3,0). This migration classes have to have the two class methods getMigrationClass and getMigrationObject described above.

Example

In the example a simple (no chain) migration of the Athena class is shown. The old schema has version (1,0) and the new one has version (2,0). The schemas differ in that the old one describes option_file as a single item, whereas the new schema declares it as a sequence.

################## new class ########################################################
class Athena(IApplication):
    """The main Athena Job Handler"""

    _schema = Schema(Version(2,0), {
                 'atlas_release'   : SimpleItem(defvalue='',doc='ATLAS Software Release'),
                 'user_area'       : FileItem(doc='A tar file of the user area'),
                 'max_events'      : SimpleItem(defvalue='',doc='Maximum number of events'),
                 'option_file'     : FileItem(defvalue=[], sequence=1, strict_sequence=0, doc="list of job options files" ),
                 'options'         : SimpleItem(defvalue='',doc='Additional Athena options'),
                 'user_setupfile'  : FileItem(doc='User setup script for special setup'),
                 'exclude_from_user_area' : SimpleItem(defvalue=[],sequence=1,doc='Pattern of files to exclude from user area'),
                 'exclude_package' : SimpleItem(defvalue=[],sequence=1,doc='Packages to exclude from user area requirements file')
              })
                     
    _category = 'applications'
    _name = 'Athena'

## additional class methods to support migration ################################
    def getMigrationClass(cls, version):
        """This class method returns a (stub) class compatible with the schema <version>.
        Alternatively, it may return a (stub) class with a schema more recent than schema <version>,
        but in this case the returned class must have "getMigrationClass" and "getMigrationObject"
        methods implemented, so that a chain of convertions can be applied."""
        return AthenaMigration12
    getMigrationClass = classmethod(getMigrationClass)

    def getMigrationObject(cls, obj):
        """This method takes as input an object of the class returned by the "getMigrationClass" method,
        performs object transformation and returns migrated object of this class (cls)."""
        converted_obj = cls()
        for attr, item in converted_obj._schema.allItems():
            # specific convertion stuff
            if attr == 'option_file':
                setattr(converted_obj, attr, [getattr(obj, attr)]) # correction: []
            else:
                setattr(converted_obj, attr, getattr(obj, attr))
            return converted_obj
    getMigrationObject = classmethod(getMigrationObject)
## end of additional class methods to support migration ################################
## <Athena class implementation is not relevant for the example>########################

###### migration class #############################################################
class AthenaMigration12(Athena):
    """This is a migration class for Athena Job Handler with schema version 1.2.
    There is no need to implement any methods in this class, because they will not be used.
    However, the class may have "getMigrationClass" and "getMigrationObject" class 
    methods, so that a chain of convertions can be applied."""

    _schema = Schema(Version(1,0), {
                 'atlas_release'   : SimpleItem(defvalue='',doc='ATLAS Software Release'),
                 'user_area'       : FileItem(doc='A tar file of the user area'),
                 'max_events'      : SimpleItem(defvalue='',doc='Maximum number of events'),
                 'option_file'     : FileItem(defvalue=[], sequence=0, strict_sequence=0, doc="list of job options files" ),
                 'options'         : SimpleItem(defvalue='',doc='Additional Athena options'),
                 'user_setupfile'  : FileItem(doc='User setup script for special setup'),
                 'exclude_from_user_area' : SimpleItem(defvalue=[],sequence=0,doc='Pattern of files to exclude from user area'),
                 'exclude_package' : SimpleItem(defvalue=[],sequence=1,doc='Packages to exclude from user area requirements file')
              })

    _category = 'application_converters' # put this class in different category
    _name = 'AthenaMigration12'
###### end of migration class #######################################################

Migration Control

Migration of Ganga plugins is controlled by the migration control object, which is an attribute of the module Ganga.GPIDev.Streamers.MigrationControl, and also by the configuration section [MigrationControl] of Ganga configuration.

[MigrationControl]
migration = allow #allows migration for all plugins
# migration = deny #denies migration for all plugins
# any other value or no value will require interactive user input

More fine grained control over migration can be achieved either via MigrationControl.migration migration control object or as a result of interactive user input. If an old plugin of category category, with name name, and version version is met the user has the following choices to for migration:

'1' : 'Yes',
'2' : 'No',
'3' : 'Yes for All',
'4' : 'No for All',
'5' : 'Yes for All with this _version_',
'6' : 'No for All with this _version_',
'7' : 'Yes for All with this _name_',
'8' : 'No for All with this _name_',
'9' : 'Yes for All within this _category_',
'10': 'No for All within this _category_'

The same options can be set in the migration control object

migration.allow() --> allow all
migration.allow(category) --> allow all from category
migration.allow(category, name) --> allow for all versions of name from category
migration.allow(category, name, version) --> allow for version and name from category
migration.deny() --> deny all
migration.deny(category) --> deny all from category
migration.deny(category, name) --> deny for all versions of name from category
migration.deny(category, name, version)  --> deny for version and name from category.

In order to migrate old jobs within a script one can use the function migrateGangaObjects from the module Ganga.Core.JobRepository.GangaObjectMigration. This function transforms older versions of GangaObjects to the current versions. The migration is made in place over repository, i.e. it does not affect the job registry. Therefore the job registry has to be rescanned after the migration. One may use jobs._impl._scan_repository() to do so. Only completely converted jobs will be migrated. No jobs in "incomplete" state will be saved in the repository. Once migrated the plugins may not be backward compatible. Arguments:

  • repository - current job repository (can be accessed as jobs._impl._repository)
  • migration_control - migration control object that allows/denies migration of particular plugins (an instance of MigrationControl class. If this argument is ommitted the migration will require interactive input

Example of usage within a script:

from Ganga.GPIDev.Streamers.MigrationControl import MigrationControl
migration_control = MigrationControl()
migration_control.allow() #allows migration of all possible plugins
migrateGangaObjects(jobs._impl.repository, migration_control)
jobs._impl._scan_repository()

Important notice: migrated jobs stay in the registry. They are not automatically flushed back to the repository. In order to facilitate this process one can access list of all migrated jobs as Ganga.GPIDev.Streamers.MigrationControl.migrated_jobs. This list may also contain templates if not emptied before migration of templates.

-- Main.asaroka - 16 May 2007 -- Main.asaroka - 31 May 2007

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2007-05-31 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ArdaGrid All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback