The PanDA Monitor Web Platform

Introduction

new Panda Web Platform

Main components:

• Panda Server
• TaskBuffer
• PandaDB
• Panda Web Platfrom
• The library of the custom end-user back-end applications - modules
• The library of the custom end-user front-end applications - JavaScript function

The new PanDA Monitor allows to encapsulate the PandaDB schema evolution within its Data Access Layer ( TaskBuffer ) and hide it from the Web applications.

JSONP JSON wget curl ajax WSGI Web service

One can separate here:

• Back-end - to access the data source, generate and publish the content
• Front-end - to provide the custom representation of the published content.

One can see that the framework publishes the content generated by any custom application in json format "by default"

The design allows for the 3d party front-ends to be used to process the back-end published content. The front-end is free to choose the technology to treat the content. It can be CLI curl / wget / python-script based , ajax or the regular JavaScript applications (see the Developer Guide for details )

Panda Monitor Home Page

The New Web Platform-based Panda Monitor web site provides the Home page that shows the list of the new top-level functions

classic Panda

API

http[s]://pandamon.cern.ch/[~version/][context/][application][?param1=value1[&param2=value2[& . . . . [&paramN=valueN]

where

   http: | https:     - the network protocol 
   //pandamon.cern.ch - the platform Web Server host name
   /~dev              - The framework version (optional)
   /context           - the arbitrary application context name (optional)
   /application       - the application name (optional) followed by the list of the  non default parameter values

The new platform-based Panda Monitor Web Application is backward compatible with the classical Panda Monitor. It does support the legacy "classic" Panda Monitor Interface.

For example, the 3 different URLs,

• the "classical" one : http://panda.cern.ch?dash=prod
• the new http://pandamon.cern.ch?dash=prod
• and the http://pandamon-eu.atlascloud.org?dash=prod

The Platform Application is a custom python program invoked by WSGI server. It is to create and publish the python object. The framework is to encode the published object into the JSON format and ship it to the remote Web client over http/https protocol. Optionally, 'on demand' only, the application can publish the rendering JavaScript JQuery-based function. The function can be used by the Web Browser to visualize the content the application has published (see the Developer Guide for details )

The module "alist - List Active Modules" lists all available top-level modules. In addition some methods from the set of about 120 methods of the Panda Server TaskBuffer package can be used via the monitor "taskBuffer" module API:
         http://pandamon.cern.ch/taskBuffer?method=some_method

API parameter value format

Each 'pmModule' is free to define the format of the value it accepts. However, one can distinguish thee different formats, .

single value

This is default format The single value is the string defining the value that has no "," comma or '*' symbols. This is default format
For example,
         http://pandamon.cern.ch/listusers?topsize=20&days=3
to get the 20 names long PanDA user "top list" for the last 3 days.

regexp pattern

Slightly modified *regexp pattern* can be used to define the range of the possible parameter values.
For example,
         http://pandamon.cern.ch/tasks/listtasks1?name=mc12_8TeV.(12689)|(16196)*&hours=5000
to get the list of the Monte-Carlo production tasks from 'mc12_8TeV' project with the 'dsn' the fist 5 digits are either 12689 or 16196 for the last 5000 hours
Check the example above. It demonstrates how the standard regexp was modified.
The symbol dot - "." is always just the ordinary character "." rather the special "pattern" for "any symbol".
The symbol '*' is treated as the regexp expression '.*' dot+asteric, i.e. it is a "wildcard-like' pattern for "any number of any symbol"*.

comma-separated

The list of the "comma-separated" values. Each value can be either "single value" or "regexp pattern".
For example,
         http://pandamon.cern.ch/tasks/listtasks1?tidm=1134632,1134629
to query the parameters of two concrete tasks.
If any element of the 'comma-separated' values is "regexp" then all comma-separated values are treated as "regexp" rather as "single value" .
Compare the query:
         http://pandamon.cern.ch/tasks/listtasks1?name=mc12_8TeV.(12689)|(16196)*&hours=5000&formats=NTUP_TOP
to list all tasks with the 'formats = NTUP_TOP' sharp vs.
         http://pandamon.cern.ch/tasks/listtasks1?name=mc12_8TeV.(12689)|(16196)*&hours=5000&formats=TOP,N*
where the task format should match either 'TOP' or 'N*' regexp pattern.

The "single value' is the default format. Check the module API documentation to see whether some parameter accepts the comma-separated or "regexp" formats as well.

Browser API

The module "alist - List Active Modules" lists all available top-level modules. In addition some methods from about 120 methods of Panda Server TaskBuffer package can be used via the monitor "taskBuffer" module API:

http://pandamon.cern.ch/taskBuffer?method=some_method

Upon invocation of some module the framework publishes the module generated content as well as the automatically generated module API documentation. To access the documentation the user should click the "'?" help button.

python doc string

CLI API

Since the framework always publishes the content in the jSON format no special trick is needed to use the Panda Monitor as Web service for the CLI / Ajax based front-end clients. The automatically generated API document always contains the up-to-date reference to the correct "curl" command to fetch the information.

One should take in account that the Apache Web server compresses (’gzips’) the content. Therefore one is recommended to apply the "--compressed" curl option to conserve the network traffic and get the result faster.

Custom Monitoring Client Code

example repository

CLI example: Get the Panda Job Info:

#!/usr/bin/python
import commands,pprint
cmd ="curl -s --compressed  'http://pandamon.cern.ch/jobinfo?jobparam=jobStatus,cpuConsumptionTime,modificationHost,computingSite,prodUserName,prodSourceLabel&jobtype=production&hours=2&dump=yes'"
out = commands.getoutput(cmd)
# find the json response
null  = None  # map JSON 'null'  to the python "None" (see: http://docs.python.org/2/library/json.html#json-to-py-table for details )
false = False # map JSON 'false' to the python "False"
true  = True  # map JSON 'true'  to the python "True"
jtxt = eval(out)
jdict =jtxt['pm']
jobs1 = jdict[0]
jobs = jobs1['json']['info']
header = jobs1['json']['header']
pprint.pprint(jobs,indent=1, width=240)
for j in jobs:
   rowdict =  dict(zip(header,j))
   print  '*******'
   pprint.pprint(rowdict,indent=2, width=300)

CLI example: Get the Panda Production Task Info:

#!/usr/bin/python
""" Custom Monitoring Client Code """
# $Id: PandaPlatform.txt,v 1.23 2013/05/30 15:43:30 fine_40bnl_2egov Exp $
# see: https://twiki.cern.ch/twiki/bin/viewauth/Atlas/PandaPlatform#Custom_Monitoring_Client_Code
import commands,pprint
cmd ="curl -s --compressed  'http://pandamon.cern.ch/tasks/listtasks1'"
out = commands.getoutput(cmd)
# find json responce
null  = None  # map JSON 'null'  to the python "None" (see: http://docs.python.org/2/library/json.html#json-to-py-table for details )
false = False # map JSON 'false' to the python "False"
true  = True  # map JSON 'true'  to the python "True"
jtxt = eval(out)
jdict =jtxt['pm']
jobs1 = jdict[0]
tasks = jobs1['json']['tasks']['rows']
header = jobs1['json']['tasks']['header']
for t in tasks:
   rowdict =  dict(zip(header,t))
   print  '*******'
   pprint.pprint(rowdict,indent=2, width=300)

References

• Motivation ( M.Potekhin, 11-Oct-2010 )
• [[][ADC Weekly Feb 9, 2011]] - Analyze the requirements and constrains.
• ADC Weekly Feb 12, 2011 - First prototype
• S & C Workshop, April 4, 2011 - “User request” driven approach monitoring …
• S&C, July 18,2011 - First EC2 cloud-based prototype:
• S&C Workshop, Oct 17, 2011 - Refined framework requirements.
• S&C Workshop, March 12, 2012 - Panda Monitor development.
• S&C Workshop, March 14, 2012 - ATLAS Nightlies Monitor https://atlas-nightlies-browser.cern.ch
• S&C Workshop, June 11, 2012 - Steady grows; already useful.
• S&C Workshop, June 13, 2012 - ANSII UI Example: Administrative Interface.
• S&C Workshop, March 13, 2013 - PanDA Monitor
• Workshop on Analysis Tools Development, May 16, 2013 - Overview of PanDA Monitor

Responsible: ValeriFine

Never reviewed