PandaPlatform < PanDA

PanDA Web>AtlasDistributedComputing>PanDA>PandaPlatform (2013-12-18, ValeriFine) (raw view)
<!--
   * Set PandaPlatformHomePage = pandamon.cern.ch
   * Set PandaClassicHomePage = panda.cern.ch
-->
<!-- This is the default ATLAS template. 
Please modify it in the sections indicated to create your topic! In particular, notice that at the bottom there are some sections that must be filled for publicly accessible pages.
If you have any comments/complaints about this template, then please email : Patrick Jussel (patrick dot jussel at cern dot ch) and/or Maria Smizanska (maria dot smizanska at cern dot ch)

By default the title is the WikiWord used to create this topic
if you want to modify it to something more meaningful, just replace %TOPIC% below with i.e "My Topic"
===========  you can remove the lines above ============================================= -->
%CERTIFY%
---+!! The !PanDA Monitor Web Platform
<!--
---+!!<nop>%TOPIC%
-->
%TOC%     <!-- this line is optional -->
%STARTINCLUDE%
---+Introduction
<!-- Add an introduction here, describing the purpose of this topic. --> 
The [[https://indico.cern.ch/getFile.py/access?resId=1&materialId=slides&contribId=22&sessionId=7&subContId=3&confId=210656][new Panda Web Platform]] was developed to back the Panda Monitor Web server that is more maintainable, supports the JSON/JQuery architecture, is easily extensible, and integrates well with other ADC monitoring tools and components  

Main components:

   * [[PandaServer][Panda Server]] 
   * [[PandaTaskBuffer][TaskBuffer]] 
   * [[PandaDB][PandaDB]]
   * [[PandaPlatform][Panda Web Platform]]
   * The library of the custom end-user back-end applications - modules
   * The library of the custom end-user front-end applications - JavaScript function

 The new  [[PandaMonitor][PanDA Monitor]] allows to  encapsulate the PandaDB [[http://en.wikipedia.org/wiki/Schema_evolution][schema evolution]]
 within its Data Access Layer ( [[PandaTaskBuffer][TaskBuffer]] )  and hide it from the Web applications.

<!-- Add the main topic here. i.e. create some new headings as follows: -->
<center>
<img src="%ATTACHURLPATH%/PandaWebPlatform.png" alt="diagram"  width="60%"  />
</center>

[[http://en.wikipedia.org/wiki/JSONP][JSONP]]  [[http://en.wikipedia.org/wiki/JSON][JSON]] [[http://en.wikipedia.org/wiki/Wget][wget]] [[http://en.wikipedia.org/wiki/CURL][curl]] [[http://en.wikipedia.org/wiki/Ajax_(programming)][ajax]] [[http://en.wikipedia.org/wiki/Web_Server_Gateway_Interface][WSGI]] 
[[http://www.w3schools.com/webservices/default.asp][Web service]]

One can separate here:

   * *[[http://en.wikipedia.org/wiki/Front_and_back_ends][Back-end]]*  - to access the data source, generate and publish both the [[http://en.wikipedia.org/wiki/Web_content][Web content]] as well as the user interface
   * *[[http://en.wikipedia.org/wiki/Front_and_back_ends][Front-end]]* - to provide the custom  representation of the published content.

One can see that the framework publishes the content generated by any custom application in json format "by default" 

The design allows for the 3d party front-ends to be used to process the back-end published content. The front-end is free to choose the technology to treat the content. It can be CLI  [[http://en.wikipedia.org/wiki/CURL][curl]] / [[http://en.wikipedia.org/wiki/Wget][wget]] / python-script based , [[http://en.wikipedia.org/wiki/Ajax_(programming)][ajax]]  or the regular JavaScript applications (see the [[PandaPlatformDevelopers][Developer Guide]] for details )


#PandaMonitorHomePage
---+Panda Monitor Home Page

   The [[http://%PandaPlatformHomePage%][New Web Platform-based Panda Monitor]] web site provides the Home page that shows the list of the  new top-level functions
<center> <img src="%ATTACHURLPATH%/PandaNewMonitorHome.png" alt="diagram"  width="40%"  /> </center> as well as the links to all [[http://%PandaClassicHomePage%][classic Panda]] Pages  <center><img src="%ATTACHURLPATH%/PandaNewClassicMonitorHome.png" alt="diagram"  width="40%"  /> </center> 
The top menu items of the "classic" Page are available via the new interface as well. These items were moved from the top menu bar in the dedicated "Operations" accordion position onto the  left pane of the new page <center><img src="%ATTACHURLPATH%/Old2NewMenuItemTransition.png" alt="diagram"  width="64%"  /> </center> 


---+API
#PandaMonitorAPI
<verbatim>
http[s]://pandamon.cern.ch/[~version/][context/][application][?param1=value1[&param2=value2[& . . . . [&paramN=valueN]
</verbatim>


where
<verbatim>
   http: | https:     - the network protocol 
   //pandamon.cern.ch - the platform Web Server host name
   /~dev              - The framework version (optional)
   /context           - the arbitrary application context name (optional)
   /application       - the application name (optional) followed by the list of the  non default parameter values
</verbatim>

The new platform-based [[http://%PandaPlatformHomePage%][Panda Monitor]] Web Application is backward compatible with the [[http://%PandaClassicHomePage%][classical Panda Monitor]]. It does support the legacy "classic" Panda Monitor Interface. 

For example, the 3 different URLs, 
   * the "classical" one : [[http://%PandaClassicHomePage%?dash=prod][http://panda.cern.ch?dash=prod]]
   * the new [[http://%PandaPlatformHomePage%?dash=prod ][http://pandamon.cern.ch?dash=prod]] 
   * and the [[http://pandamon-eu.atlascloud.org?dash=prod][http://pandamon-eu.atlascloud.org?dash=prod]] 
redirect the user query to one and the same Web Server and generate the same "classic" Web page.

The Platform Application is a custom python program invoked by  [[http://en.wikipedia.org/wiki/Web_Server_Gateway_Interface][WSGI]] server.  It is to create and publish the python object. The framework is to encode the published object into the [[http://en.wikipedia.org/wiki/JSON][JSON]] format and ship it to the remote Web client over http/https protocol.  Optionally, 'on demand' only, the application can publish the rendering [[http://en.wikipedia.org/wiki/JavaScript][JavaScript]] [[http://en.wikipedia.org/wiki/JQuery][JQuery-based]]  function. The function can be used by  the Web Browser to visualize the content the application has  published (see the [[PandaPlatformDevelopers][Developer Guide]] for details )

       The module [[http://%PandaPlatformHomePage%/alist]["alist - List Active Modules"]] lists all available top-level modules. In addition some methods from  [[https://pandamon.cern.ch/taskBufferList][the set of  about  120 methods]] of the [[PandaServer][Panda Server]] [[PandaTaskBuffer][TaskBuffer]] package can be used via the monitor "taskBuffer" module API: %BR% &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  [[http://%PandaPlatformHomePage%/taskBuffer][http://%PandaPlatformHomePage%/taskBuffer?method=some_method]]

---++API parameter value format

    Each 'pmModule' is free to define the   format of the value it accepts. However, one can distinguish  thee different formats,
#SingleValueParam.
   $ *single value*: *This is default format* The single value is the string defining the value that has no "," comma or '*' symbols. This is default format  %BR% For example, %BR% &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  http://%PandaPlatformHomePage%/listusers?topsize=20&days=3  %BR% to get the 20 names long  PanDA user "top list" for the last 3 days.
#RegexpParam
   $ *regexp pattern*: Slightly modified [[http://pubs.opengroup.org/onlinepubs/007908799/xbd/re.html][*regexp pattern*]] can be used to define the range of  the possible parameter values. %BR%  For example, %BR% &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; http://%PandaPlatformHomePage%/tasks/listtasks1?name=mc12_8TeV.(12689)|(16196)*&hours=5000  %BR% to get the list of the Monte-Carlo production tasks from 'mc12_8TeV' project with the 'dsn' the fist 5 digits are either 12689 or 16196 for the last 5000 hours %BR% Check the example above. It  demonstrates how the  [[http://pubs.opengroup.org/onlinepubs/007908799/xbd/re.html][standard *regexp]] was modified*.  %BR% The symbol *dot* -  <bold>"."</bold>  is always just the ordinary character  "." rather the  special  "pattern" for "any symbol". %BR% The  <bold> symbol '*' </bold>  is treated as the regexp expression <bold>  '.*'  </bold>  *dot+asteric*, i.e. it is a "wildcard-like' pattern for   "any number of any symbol"*.
#CommaParam
   $ *comma-separated*: *The list of the "comma-separated"* values. Each value can be either "single value" or "regexp pattern". %BR%  For example, %BR% &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; http://%PandaPlatformHomePage%/tasks/listtasks1?tidm=1134632,1134629  %BR%  to query  the parameters of two concrete tasks.%BR% If any element of the 'comma-separated' values is "regexp" then all comma-separated values are treated as [[#RegexpParam]["regexp"]] rather as [[#SingleValueParam]["single value"]] . %BR% Compare the query: %BR% &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; http://%PandaPlatformHomePage%/tasks/listtasks1?name=mc12_8TeV.(12689)|(16196)*&hours=5000&formats=NTUP_TOP %BR% to list all tasks  with the 'formats = NTUP_TOP' sharp vs.%BR% &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; http://pandamon.cern.ch/tasks/listtasks1?name=mc12_8TeV.(12689)|(16196)*&hours=5000&formats=TOP,N* %BR% where the task format should match either 'TOP' or 'N*' regexp pattern.


The [[#SingleValueParam]["single value']] is the default format. Check the module API  documentation to see whether some parameter accepts the comma-separated or "regexp" formats as well.

---++Browser API
      
       The module [[http://%PandaPlatformHomePage%/alist]["alist - List Active Modules"]] lists all available top-level modules. In addition some methods from [[https://pandamon.cern.ch/#/taskBufferList][about 120 methods]] of   [[PandaServer][Panda Server]] [[PandaTaskBuffer][TaskBuffer]] package can be used via the monitor "taskBuffer" module API:

   [[http://%PandaPlatformHomePage%/taskBuffer][http://%PandaPlatformHomePage%/taskBuffer?method=some_method]]

Upon invocation of some module the framework publishes the module generated content as well as the automatically generated module API documentation. To access the documentation the user should click the "'?" help button.<center><img src="%ATTACHURLPATH%/ModuleApiDoc.png" alt="diagram"  width="64%"  /> </center> 
Since the help is generated automatically by extracting the [[http://www.python.org/dev/peps/pep-0257/ ][python doc string]] from the module code  it is supposed to be always up-to-date

---+++'Drill Down" Web GUI

Many Panda Monitor pages  share the common "drill down" Web GUI to narrow the items selection  like this,

<center> <img src="%ATTACHURLPATH%/DrillDownPage.png" alt="DrillDownPage"  width="80%"  /> </center> 

The diagram below explains the purpose of the different GUI controls and indicators:

<center> <img src="%ATTACHURLPATH%/DrillDownControl.png" alt="DrillDownControl"  width="70%"  /> </center> 

The current selection parameters are checked and displayed by  the "Selection Bar"  that can be found over the "item summary. For example, the screenshot below shows the "Selection Bar" and "Summary" screen for user who wanted to know how many her jobs the Panda  is going retried yet  or is retrying using the site ANALY_AGLT2_SL6

<center> <img src="%ATTACHURLPATH%/SelectionBar.png" alt="SelectionBar"  width="90%"  /> </center> 

---+++'Drill Up" Web GUI

Many Panda Monitor pages  share the common "drill up" Web GUI that allows removing  some item selectors to get the wider selection of the items. For example, un-checking the  "ComputingSite" check box from the example above 

     <center>  <img src="%ATTACHURLPATH%/SelectionBarDrillUp.PNG" alt="SelectionBarDrillUp.PNG"  width="90%" /> </center> 

one can get the list of the jobs  to be retried world-wide as follows:

     <center>  <img src="%ATTACHURLPATH%/SelectionBarDrillUp2.PNG" alt="SelectionBarDrillUp2.PNG"  width="90%" /> </center> 

---++CLI API

Since the framework always publishes the content in the  jSON format no special trick is needed to use the Panda Monitor as Web service for the CLI / Ajax based front-end clients. The automatically generated API document always contains the up-to-date reference to the correct "curl" command to fetch the information. 

One should take in account that the Apache Web server compresses (’gzips’) the content. Therefore one is recommended to apply the "--compressed" curl option to conserve the network traffic and get the result faster.
---+++Custom Monitoring Client Code
Using CLI API is trivial  for anybody to  create his our custom monitoring code.  Try to copy /paste the code below from any your local computer word-wide and execute. ( You may want to browse the [[https://svnweb.cern.ch/trac/panda/browser/pandamon/trunk/pandamon/examples?order=date][example repository]]  too. *Do not hesitate contributing your own examples. It is welcome and appreciated!*  )
---++++CLI example: [[https://svnweb.cern.ch/trac/panda/browser/pandamon/trunk/pandamon/scripts/getJobInfo.py][Get  the Panda Job Info]]: 
*<verbatim>
#!/usr/bin/python
import commands,pprint
cmd ="curl -s --compressed  'http://pandamon.cern.ch/jobinfo?jobparam=jobStatus,cpuConsumptionTime,modificationHost,computingSite,prodUserName,prodSourceLabel&jobtype=production&hours=2&dump=yes'"
out = commands.getoutput(cmd)
# find the json response
null  = None  # map JSON 'null'  to the python "None" (see: http://docs.python.org/2/library/json.html#json-to-py-table for details )
false = False # map JSON 'false' to the python "False"
true  = True  # map JSON 'true'  to the python "True"
jtxt = eval(out)
jdict =jtxt['pm']
jobs1 = jdict[0]
jobs = jobs1['json']['info']
header = jobs1['json']['header']
pprint.pprint(jobs,indent=1, width=240)
for j in jobs:
   rowdict =  dict(zip(header,j))
   print  '*******'
   pprint.pprint(rowdict,indent=2, width=300)
</verbatim>*

---++++CLI example: [[https://svnweb.cern.ch/trac/panda/browser/pandamon/trunk/pandamon/scripts/getTaskInfo.py][Get  the Panda Production Task Info]]: 
*<verbatim>
#!/usr/bin/python
""" Custom Monitoring Client Code """
# $Id: PandaPlatform.txt,v 1.27 2013/12/18 20:09:41 fine_40bnl_2egov Exp $
# see: https://twiki.cern.ch/twiki/bin/viewauth/Atlas/PandaPlatform#Custom_Monitoring_Client_Code
import commands,pprint
cmd ="curl -s --compressed  'http://pandamon.cern.ch/tasks/listtasks1'"
out = commands.getoutput(cmd)
# find json responce
null  = None  # map JSON 'null'  to the python "None" (see: http://docs.python.org/2/library/json.html#json-to-py-table for details )
false = False # map JSON 'false' to the python "False"
true  = True  # map JSON 'true'  to the python "True"
jtxt = eval(out)
jdict =jtxt['pm']
jobs1 = jdict[0]
tasks = jobs1['json']['tasks']['rows']
header = jobs1['json']['tasks']['header']
for t in tasks:
   rowdict =  dict(zip(header,t))
   print  '*******'
   pprint.pprint(rowdict,indent=2, width=300)
</verbatim>*



---+References 


   * [[PanDjango#Motivations][Motivation ( M.Potekhin, 11-Oct-2010 ) ]]
   * [[][ADC Weekly Feb 9, 2011]] - Analyze the requirements and constrains.
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=127099][ADC Weekly Feb 12, 2011]] - [[https://indico.cern.ch/getFile.py/access?contribId=0&resId=5&materialId=slides&confId=127099][First prototype]]
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=119169][S & C Workshop, April 4, 2011]] - [[https://indico.cern.ch/conferenceDisplay.py?confId=119169][“User request” driven approach]] monitoring …
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=119170][S&C, July 18,2011]] - [[https://indico.cern.ch/getFile.py/access?contribId=68&sessionId=10&resId=0&materialId=slides&confId=119170][First EC2 cloud-based]] [[http://atlascloud.org][prototype:]]
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=119171][S&C Workshop, Oct 17, 2011]] - [[https://indico.cern.ch/getFile.py/access?contribId=76&sessionId=3&resId=1&materialId=slides&confId=119171][Refined framework requirements]].
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=169695][S&C Workshop, March 12, 2012]] - [[https://indico.cern.ch/getFile.py/access?contribId=52&sessionId=2&resId=1&materialId=slides&confId=169695][Panda Monitor development]].
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=169695][S&C Workshop, March 14, 2012]] - [[https://indico.cern.ch/getFile.py/access?contribId=8&sessionId=7&resId=1&materialId=slides&confId=169695][ATLAS Nightlies Monitor]] [[https://atlas-nightlies-browser.cern.ch][https://atlas-nightlies-browser.cern.ch]]
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=169698][S&C Workshop, June 11, 2012]] - [[https://indico.cern.ch/getFile.py/access?contribId=2&sessionId=2&resId=2&materialId=slides&confId=169698][Steady grows; already useful]].
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=169698][S&C Workshop, June 13, 2012]] - [[https://indico.cern.ch/getFile.py/access?contribId=30&sessionId=6&resId=3&materialId=slides&confId=169698][ANSII UI Example: Administrative Interface]].
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=210656][S&C Workshop, March 13, 2013]] - [[https://indico.cern.ch/getFile.py/access?resId=1&materialId=slides&contribId=22&sessionId=7&subContId=3&confId=210656][PanDA Monitor]]
   * [[https://indico.cern.ch/conferenceDisplay.py?confId=250199][Workshop on Analysis Tools Development, May 16, 2013]]  - [[https://indico.cern.ch/getFile.py/access?contribId=7&sessionId=1&resId=0&materialId=slides&confId=250199][Overview of PanDA Monitor]]
   *  [[https://indico.cern.ch/conferenceDisplay.py?confId=210657][S&C Workshop, June 12, 2013]] - [[https://indico.cern.ch/getFile.py/access?resId=1&materialId=slides&contribId=23&sessionId=9&subContId=4&confId=210657][New Panda Monitor ( tutorial)]]
   *  [[https://indico.cern.ch/conferenceOtherViews.py?view=cdsagenda&confId=266191][PanDA Workshop @ UTA, Oct 3-4, 2013]] - [[https://indico.cern.ch/getFile.py/access?contribId=4&sessionId=0&resId=1&materialId=slides&confId=266191][Panda Monitoring]]


<!-- *********************************************************** -->
<!-- Do NOT remove the remaining lines, but add requested info as appropriate -->
<!-- *********************************************************** -->

-----
<!-- For significant updates to the topic, consider adding your 'signature' (beneath this editing box) -->
*Major updates*:%BR%
-- Main.ValeriFine - 13-Feb-2013

<!-- Person responsible for the page: 
Either leave as is - the creator's name will be inserted; 
Or replace the complete REVINFO tag (including percentages symbols) with a name in the form Main.TwikiUsersName -->
%RESPONSIBLE% %REVINFO{"$wikiusername" rev="1.1"}% %BR%
<!-- Once this page has been reviewed, please add the name and the date e.g. Main.StephenHaywood - 31 Oct 2006 -->
%REVIEW% *Never reviewed*

%STOPINCLUDE%
<!--
   * The PanDA Web Platform diagram: <br />
     <img src="%ATTACHURLPATH%/PandaWebPlatform.png" alt="PandaWebPlatform.png" width="960" height="720" />

   * The Monitor Page with the "Drill Down" GUI: <br />
     <img src="%ATTACHURLPATH%/DrillDownPage.png" alt="DrillDownPage.png" width="879" height="472" />

   * "Drill Down" Panda Monitor GUI: <br />
     <img src="%ATTACHURLPATH%/DrillDownControl.png" alt="DrillDownControl.png" width="1800" height="900" />

   * The "Selection Bar" followed by the items' summaries screen: <br />
     <img src="%ATTACHURLPATH%/SelectionBar.png" alt="SelectionBar.png" width="1065" height="236" />

   * Uncheck the  item to "Drill Up": <br />
     <img src="%ATTACHURLPATH%/SelectionBarDrillUp.PNG" alt="SelectionBarDrillUp.PNG" width="1065" height="243" />

   * The list job: <br />
     <img src="%ATTACHURLPATH%/SelectionBarDrillUp2.PNG" alt="SelectionBarDrillUp2.PNG" width="1011" height="275" />
Topic revision: r27 - 2013-12-18 - ValeriFine
PanDA
Public webs
- Cern Search
- TWiki Search
- Google Search
PanDA All webs
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback