Proposal machine / job features data structure and an interface to retrieve it

This page contains a description of a data structure to be proposed to the VOs for retrieving machine / job features and a tool / interface to retrieve the information. Note: This is currently under discussion and not to be regarded as final.

Data Structure

Data

The data shall be represented in json format, see below an example. If the values represented for the different features are of type integer or float they shall be converted to those formats instead of printing strings.

?? Shall we have a validation within the tool??

{
  "machinefeatures": {
    "hs06": 171.03999999999999, 
    "jobslots": 18, 
    "log_cores": 12, 
    "phys_cores": 12
  }, 
  "jobfeatures": {
    "disk_limit_GB": 1331.0385343999999, 
    "jobstart_secs": 1379938695, 
    "wall_limit_secs": 1182.03928905519, 
    "cpu_limit_secs": 262.67539756782003, 
    "cpu_limit_secs_lrms": 480, 
    "allocated_CPU": 1, 
    "mem_limit_MB": 4000000
  }
}

Other messages

If the data is produced by calling a tool from the command line, the same data structure shall be used to provide WARNING or ERROR messages to the caller. Every message shall contain three dash (" - ") delimited parts

  1. a time stamp in iso format, the time shall be in UTC
  2. the type of message, i.e. INFO, WARNING, ERROR
  3. a text providing more information on the message

{
  "messages": [
    "2013-09-23T15:31:47.371497 UTC - ERROR - Cannot find job / machine features information on this node"
  ]
}

Interface

Two ways are provided to retrieve the information, by calling the tool from the command line or importing it into your python module

Command line

Calling the tool from the command line shall return the above mentioned data structure for both the data part (if any) and messages (if any).

Example:

macites01:src roiser$ ./mjf.py 
{"machinefeatures": {"hs06": 171.03999999999999, "jobslots": 18, "log_cores": 12, "phys_cores": 12}, "jobfeatures": {"disk_limit_GB": 1331.0385343999999, "jobstart_secs": 1379938695, "wall_limit_secs": 1182.03928905519, "cpu_limit_secs": 262.67539756782003, "cpu_limit_secs_lrms": 480, "allocated_CPU": 1, "mem_limit_MB": 4000000}}

Python interface

help of the python class

    class mjf
     |  Methods defined here:
     |  
     |  __init__(self, ext=False)
     |      initialise the class instance and collect machine / job features found on the node
     |      and store it in the internal data structure.
     |      
     |      ext: is the script called from the command line (True) or imported (False). This is
     |           e.g. used when to decide whether to throw exceptions (import) or to return
     |           messages within the internal data structure (command line).
     |  
     |  clean(self)
     |      clean the data structure, i.e. reset it to an empty dictionary
     |  
     |  collect(self)
     |      collect the machine / job features information provided
     |      on this node and store it in the internal data structure
     |      Note, this function will not previously clean the data
     |  
     |  feature(self, key, feat='')
     |      return a value for a given feature-key, optionally a feature namne
     |      (e.g. MACHINEFEATURES, JOBFEATURES) can be given as a search tip.
     |      if no optinal feature is given all features will be searched for the key
     |      and the first occurance will be returned (not deterministic). Otherwise
     |      the key will only be searched within the feature space given.
     |      If no key has been found an emtpy string will be returned.
     |      
     |      key: the name of the feature key for which the value shall be retrieved (case sensitive)
     |      feat: the name of the feature (e.g. MACHINEFEATURES) given as optional search hint
     |      
     |      return: value of the feature key or empty string if not found
     |  
     |  featureKeys(self)
     |      return the individual feature names that were collected so far
     |      
     |      return: a dictionary of {'<feature-name>':[<feature-key>,...]}
     |              can be empty if no data was collected / found so far
     |  
     |  features(self)
     |      return the machine / job features data collected so far as
     |      json data structure
     |      
     |      return: json data structure of the features found on the node
     |              can be empty dictionary if no data was collected / found

Source code

The source code of the tool (both command line / python interface) can be (temporarily) found at github

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2013-09-24 - StefanRoiser
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback