Timestamps

This proposal refers to recording the ganga job state transition times (so not the timestamps recorded in the submission backends or on the worker node). NO TRUE ANYMORE.

There are the following use-cases:

  • interactive usage:
    • select jobs based on the creation time
    • compare the creation time of two jobs
  • robots:
    • gather statistics
    • implement timeouts on job monitoring
  • internal optimizations:
    • refresh internal cache (e.g. file workspace) if job resubmitted (?)

To avoid cluttering of Job object at the top level all timing information is grouped:

  • j.time.new creation time of the Job object
  • j.time.submitted initially 0, after submit() or resubmit() the time when the job entered 'submitted' state
  • j.time.running idem
  • j.time.completed
  • j.time.failed
  • j.time.killed

INFO object: resubmission counter, monitoring cookies, trash ...

updateStatus() should accept the time (specified by the backend) to give better estimate of the time most implementations would retrieve this information in a special file in output sandbox, the name of this file should be standard provide a utility to write and read this special file

If a job is resubmitted, then all the fields except new are cleared and the j.time.submitted reflects the resubmission time. If resubmission is unsuccessful (e.g. not input dir) then j.time remains untouched.

Time is stored as Epoch seconds so comparison and sorting is easy:

  • j1.time.new < j2.time.new

This means that you get Epoch seconds when printing the attribute directly:

  • print j.time.new -> 568487483833 (epoch time)

However when time is formatted to a human readable string when printing the entire object:

  • print j.time -> JobTimestamp { new =  '30 Oct 2006, 11:15', .... }

Updates from the Developer Days in Oslo 11.02-13.02 2009

It was decided to keep more or less the schema above. The new and submit time is to be set by the client, while the other timestamps must be set by the backends and mirror the real time for the state transition, independent of whether the user is running ganga or not. In addition to these main timestamps there may be a more application specific timestamps.

The information should be stored using the python datetime object
http://docs.python.org/library/datetime.html

The timestamps should be kept in a dictionary.

A history function should be implemented show the progress of the job j.history()

Search and sort methods would be helpful. The timing information is meant to be fed into the Plotter in order to provide graphical presentation of the timing data.

-- JakubMoscicki - 15 Jun 2007

In Development 01.07.2009 - Present (04.08.2009)

A class called JobTime which stores a dictionary of datetime object timestamps has been created. Their corresponding keys are the states into which the Job object enters. The six default states that are stamped are as above:

  • new
  • submitted
  • running
  • completed
  • failed
  • killed

Syntax reads as j.time..

The timestamps dictionary itself can be accessed as a dictionary would normally be in python, e.g. j.time.timestamps['new'].

Subjobs of a master Job have their individual timestamps created as a job with no subjobs would. For the master Job, the 'new' stamp is created when it itself is created. All other stamps are taken as the earliest occurance among the corresponding stamps in its subjobs, except for the completed state, where the latest is used.

Methods

Access to the timestamps can be gained using:

  • j.time.new()
  • j.time.submitted()
  • j.time.running()
  • j.time.completed()
  • j.time.failed()
  • j.time.killed()

Access to other stamps, e.g. user created stamps, can be obtained using j.time.statetime()

For a more readable display of all timestamps available, the method j.time.display() can be used to print a simple table containing timing and state information. With this method and the methods listed above a format argument can be supplied, using the strftime codes detailed at http://docs.python.org/library/datetime.html?#strftime-behavior.

Specific durations between certain states can be returned as a timedelta object:

  • j.time.waittime() returns duration between 'submitted' and 'running' states
  • j.time.runtime() return duration between 'running' and 'completed'/'failed'/'killed' states (whichever occurs first if any coexist within the dictionary)

Some backends may already record timing information, in more or less detail than is required by the user. For example, the Dirac backend often timestamps multiple stages within the same state, usually producing a dictionary of 10-20 (or sometimes more!) lists containing a timestamp and corresponding information about the state of the job at that time.

It is currently assumed the user is not interested in all that information, and the JobTime object decides to extract one stamp per state, based on which occured first.

If the user does however want this information, j.time.details() can be used to return a dictionary containg all the information the backend has produced. This has currently only been implemented for Dirac and the Local backend.

Details can also be printed directly to the screen using the method j.time.printdetails().

For jobs without subjobs, no argument is requierd for details or printdetails. For jobs consistin of a number of subjobs, supplying the subjob id as the argument will ersult in details being obtained for that job. To get the details for all the subjobs suppluy 'all' as the argument. This means that a user won't accidentally ask Ganga to print the details for a large number of subjobs.

Backends

For implementation in other backends it is required that two functions getStateTime() and timedetails() are added to the backend. For more details see the code for the IBackend at python/Ganga/GPIDev/Adapters/IBackend.py. An example implementation (for the Local backend) can be found in python/Ganga/GPIDev/Lib/Localhost/Localhost.py.

Known issues

  1. Timestamps are only persisted in the XML repository.
  2. There is no timezone support. All timestamps are calculated as UTC times. It is up to the user to make conversions.

Testing

This version of ganga can be tested at cern from the file path: /afs/cern.ch/user/u/uegede/public/timestamps/install/ganga/bin/ganga

Email questions, comments and suggestions to: justin.whitehouse07@icNOSPAMPLEASE.ac.uk

-- Justin Whitehouse -- UlrikEgede - 04 Aug 2009

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2009-08-04 - UlrikEgede
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ArdaGrid All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback