Post processors for Ganga 6

Idea

The idea behind the new postprocessor objects is to:

  • Simplify the merger code (i.e. remove the 'mergetool' objects)

  • Add the ability to fail jobs on certain criteria.

The user tutorial for the new postprocessing objects can be found here.

Schema Migration

Ganga 5 jobs will have the old merger objects with the 'mergers' category. To load these jobs, we can do two things:

  • Keep the j.merger job attribute, but hide it.

  • Add the old merger objects into the list of plugins, with the old category.

This means that Ganga 5 jobs can be loaded in Ganga 6 and the old merger objects are accessible inside Ganga. All old/new merger objects have the category postprocessor, as desired.

If you were to do the following in Ganga:

for j in jobs:
    if j._impl.merger and not len(j.postprocessors):
        j._impl.postprocessors = MultiProcessor(j._impl.mergers)

The old jobs would work just as they did before under Ganga 6 and be fully compatible.

After this the old merger objects can be removed from the code.

MultiProcessor

The postprocessor objects are called inside the updateStatus method of the job class:


 if newstatus == 'completed' or newstatus == 'failed' or newstatus == 'killed':
      if self.postprocessors:
             passed = self.postprocessors.execute(self,newstatus)
                    if passed is not True:
                        newstatus = 'failed'

j.postprocessors is a MultiProcessor object, defined in IPostProcessor.py. The execute method basically executes each postprocessor, in the right order. The ordering is: Mergers, Checkers, Notifier. The MultiProcessor object is supposed to behave like a list of postprocessors to the user.

Plugging into the new merger code

Ganga/GPIDev/Adapters/IMerger.py contains the interface for all merger objects. It essentially, after lots of checking, finds the list of files to merge. All merger objects should inherit from IMerger.

The most trivial example is the TextMerger:


class TextMerger(IMerger):
    """Merger class for text
    """
    _category = 'postprocessor'
    _name = 'TextMerger'
    _schema = IMerger._schema.inherit_copy()
    _schema.datadict['compress'] = SimpleItem(defvalue = False, doc='Output should be compressed with gzip.')
        
    def mergefiles(self, file_list, output_file):

        do some merging ...

Checkers

The main addition with the postprocessor change is the introduction of the checker. A checker will only run on jobs about to be completed and has the ability to fail jobs. Currently there are three Checkers:

  • FileChecker - checks output files and fails job if a particular string is found (or not found).

  • MetaDataChecker - simple checker to compare with the meta data of a job, this class is hidden has to be overridden in the experiment plugin.

  • CustomChecker - probably the most useful checker, allows to user to write some python to fail their job.

To implement the experiment specific MetaDataChecker, you can do something like:

class LHCbMetaDataChecker(MetaDataChecker):
    """
    Checks the meta data of a job is within some range, defined by minVal and maxVal
    Currently accepts 'lumi', 'inputevents', 'outputevents'.
    """
    _schema = MetaDataChecker._schema.inherit_copy()
    _category = 'postprocessor'
    _name = 'LHCbMetaDataChecker'
    _exportmethods = ['check']    


    def convertLHCbMetadata(self,job):
        try:
            if self.attribute == 'inputevents':
                return job.metadata['events']['input']
            if self.attribute == 'outputevents':
                return job.metadata['events']['output']
            if self.attribute == 'lumi':
                return float(job.metadata['lumi'][1:job.metadata['lumi'].find(' ')])
            return job.metadata[self.attribute]
        except:
            return None

    def check(self,job):
        """
        Checks metadata of job is within a certain range.
        """
        self.convert_metadata = self.convertLHCbMetadata(job)
        super(LHCbMetaDataChecker,self).check(job)

Where you define the convert_metadata attribute which pulls out a float depending on what string the user gives.

Notifier

Finally, the ability to email a user about the status of a job. If a notifier is attached, the default behaviour is to email the user when the master job has finished (whatever the status) and also email if any subjobs fail. If the auto-resubmit feature is enabled and a job failed, no email is sent. Users can up the verbosity if they want to get emails about every subjob.

This emailing system will not be useful for everyone and should be carefully advertised.

The SMTP host is defaulted to 'localhost' and can be overridden. In the LHCb.ini file it is set to 'cernmx.cern.ch'.

The email template is currently:

Dear User,

Job(j.fqid) has gone into completed state.

Regards, Ganga

PS: This is an automated notification from Ganga, if you would like these messages to stop please remove the notifier object from future jobs.

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2012-11-20 - PatrickOwen
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ArdaGrid All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback