The PanDA Monitor Web Platform

Introduction

The new Panda Web Platform was developed to back the Panda Monitor Web server that is more maintainable, supports the JSON/JQuery architecture, is easily extensible, and integrates well with other ADC monitoring tools and components

Main components:

The new PanDA Monitor allows to encapsulate the PandaDB schema evolution within its Data Access Layer ( TaskBuffer ) and hide it from the Web applications.

diagram

JSONP JSON wget curl ajax WSGI Web service

One can separate here:

  • Back-end - to access the data source, generate and publish both the Web content as well as the user interface
  • Front-end - to provide the custom representation of the published content.

One can see that the framework publishes the content generated by any custom application in json format "by default"

The design allows for the 3d party front-ends to be used to process the back-end published content. The front-end is free to choose the technology to treat the content. It can be CLI curl / wget / python-script based , ajax or the regular JavaScript applications (see the Developer Guide for details )

Panda Monitor Home Page

The New Web Platform-based Panda Monitor web site provides the Home page that shows the list of the new top-level functions

diagram
as well as the links to all classic Panda Pages
diagram
The top menu items of the "classic" Page are available via the new interface as well. These items were moved from the top menu bar in the dedicated "Operations" accordion position onto the left pane of the new page
diagram

API

http[s]://pandamon.cern.ch/[~version/][context/][application][?param1=value1[&param2=value2[& . . . . [&paramN=valueN]

where

   http: | https:     - the network protocol 
   //pandamon.cern.ch - the platform Web Server host name
   /~dev              - The framework version (optional)
   /context           - the arbitrary application context name (optional)
   /application       - the application name (optional) followed by the list of the  non default parameter values

The new platform-based Panda Monitor Web Application is backward compatible with the classical Panda Monitor. It does support the legacy "classic" Panda Monitor Interface.

For example, the 3 different URLs,

redirect the user query to one and the same Web Server and generate the same "classic" Web page.

The Platform Application is a custom python program invoked by WSGI server. It is to create and publish the python object. The framework is to encode the published object into the JSON format and ship it to the remote Web client over http/https protocol. Optionally, 'on demand' only, the application can publish the rendering JavaScript JQuery-based function. The function can be used by the Web Browser to visualize the content the application has published (see the Developer Guide for details )

The module "alist - List Active Modules" lists all available top-level modules. In addition some methods from the set of about 120 methods of the Panda Server TaskBuffer package can be used via the monitor "taskBuffer" module API:
         http://pandamon.cern.ch/taskBuffer?method=some_method

API parameter value format

Each 'pmModule' is free to define the format of the value it accepts. However, one can distinguish thee different formats, .

single value
This is default format The single value is the string defining the value that has no "," comma or '*' symbols. This is default format
For example,
         http://pandamon.cern.ch/listusers?topsize=20&days=3
to get the 20 names long PanDA user "top list" for the last 3 days.
regexp pattern
Slightly modified *regexp pattern* can be used to define the range of the possible parameter values.
For example,
         http://pandamon.cern.ch/tasks/listtasks1?name=mc12_8TeV.(12689)|(16196)*&hours=5000
to get the list of the Monte-Carlo production tasks from 'mc12_8TeV' project with the 'dsn' the fist 5 digits are either 12689 or 16196 for the last 5000 hours
Check the example above. It demonstrates how the standard regexp was modified.
The symbol dot - "." is always just the ordinary character "." rather the special "pattern" for "any symbol".
The symbol '*' is treated as the regexp expression '.*' dot+asteric, i.e. it is a "wildcard-like' pattern for "any number of any symbol"*.
comma-separated
The list of the "comma-separated" values. Each value can be either "single value" or "regexp pattern".
For example,
         http://pandamon.cern.ch/tasks/listtasks1?tidm=1134632,1134629
to query the parameters of two concrete tasks.
If any element of the 'comma-separated' values is "regexp" then all comma-separated values are treated as "regexp" rather as "single value" .
Compare the query:
         http://pandamon.cern.ch/tasks/listtasks1?name=mc12_8TeV.(12689)|(16196)*&hours=5000&formats=NTUP_TOP
to list all tasks with the 'formats = NTUP_TOP' sharp vs.
         http://pandamon.cern.ch/tasks/listtasks1?name=mc12_8TeV.(12689)|(16196)*&hours=5000&formats=TOP,N*
where the task format should match either 'TOP' or 'N*' regexp pattern.

The "single value' is the default format. Check the module API documentation to see whether some parameter accepts the comma-separated or "regexp" formats as well.

Browser API

The module "alist - List Active Modules" lists all available top-level modules. In addition some methods from about 120 methods of Panda Server TaskBuffer package can be used via the monitor "taskBuffer" module API:

http://pandamon.cern.ch/taskBuffer?method=some_method

Upon invocation of some module the framework publishes the module generated content as well as the automatically generated module API documentation. To access the documentation the user should click the "'?" help button.

diagram
Since the help is generated automatically by extracting the python doc string from the module code it is supposed to be always up-to-date

'Drill Down" Web GUI

Many Panda Monitor pages share the common "drill down" Web GUI to narrow the items selection like this,

DrillDownPage

The diagram below explains the purpose of the different GUI controls and indicators:

DrillDownControl

The current selection parameters are checked and displayed by the "Selection Bar" that can be found over the "item summary. For example, the screenshot below shows the "Selection Bar" and "Summary" screen for user who wanted to know how many her jobs the Panda is going retried yet or is retrying using the site ANALY_AGLT2_SL6

SelectionBar

'Drill Up" Web GUI

Many Panda Monitor pages share the common "drill up" Web GUI that allows removing some item selectors to get the wider selection of the items. For example, un-checking the "ComputingSite" check box from the example above

SelectionBarDrillUp.PNG

one can get the list of the jobs to be retried world-wide as follows:

SelectionBarDrillUp2.PNG

CLI API

Since the framework always publishes the content in the jSON format no special trick is needed to use the Panda Monitor as Web service for the CLI / Ajax based front-end clients. The automatically generated API document always contains the up-to-date reference to the correct "curl" command to fetch the information.

One should take in account that the Apache Web server compresses (’gzips’) the content. Therefore one is recommended to apply the "--compressed" curl option to conserve the network traffic and get the result faster.

Custom Monitoring Client Code

Using CLI API is trivial for anybody to create his our custom monitoring code. Try to copy /paste the code below from any your local computer word-wide and execute. ( You may want to browse the example repository too. Do not hesitate contributing your own examples. It is welcome and appreciated! )

CLI example: Get the Panda Job Info:

#!/usr/bin/python
import commands,pprint
cmd ="curl -s --compressed  'http://pandamon.cern.ch/jobinfo?jobparam=jobStatus,cpuConsumptionTime,modificationHost,computingSite,prodUserName,prodSourceLabel&jobtype=production&hours=2&dump=yes'"
out = commands.getoutput(cmd)
# find the json response
null  = None  # map JSON 'null'  to the python "None" (see: http://docs.python.org/2/library/json.html#json-to-py-table for details )
false = False # map JSON 'false' to the python "False"
true  = True  # map JSON 'true'  to the python "True"
jtxt = eval(out)
jdict =jtxt['pm']
jobs1 = jdict[0]
jobs = jobs1['json']['info']
header = jobs1['json']['header']
pprint.pprint(jobs,indent=1, width=240)
for j in jobs:
   rowdict =  dict(zip(header,j))
   print  '*******'
   pprint.pprint(rowdict,indent=2, width=300)

CLI example: Get the Panda Production Task Info:

#!/usr/bin/python
""" Custom Monitoring Client Code """
# $Id: PandaPlatform.txt,v 1.27 2013/12/18 20:09:41 fine_40bnl_2egov Exp $
# see: https://twiki.cern.ch/twiki/bin/viewauth/Atlas/PandaPlatform#Custom_Monitoring_Client_Code
import commands,pprint
cmd ="curl -s --compressed  'http://pandamon.cern.ch/tasks/listtasks1'"
out = commands.getoutput(cmd)
# find json responce
null  = None  # map JSON 'null'  to the python "None" (see: http://docs.python.org/2/library/json.html#json-to-py-table for details )
false = False # map JSON 'false' to the python "False"
true  = True  # map JSON 'true'  to the python "True"
jtxt = eval(out)
jdict =jtxt['pm']
jobs1 = jdict[0]
tasks = jobs1['json']['tasks']['rows']
header = jobs1['json']['tasks']['header']
for t in tasks:
   rowdict =  dict(zip(header,t))
   print  '*******'
   pprint.pprint(rowdict,indent=2, width=300)

References


Major updates:
-- ValeriFine - 13-Feb-2013



Responsible: ValeriFine

Never reviewed

Edit | Attach | Watch | Print version | History: r27 < r26 < r25 < r24 < r23 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r27 - 2013-12-18 - ValeriFine
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback