Panda Integration Testing

Introduction

This document describes tests on how to run Eowyn with Panda to transfer CERN DB jobs to PandaDB and how to run jobscheduler to execute jobs in OSG.

For a page with just the installation of Panda Components you can refer to PandaInstall.

Installation

  • Put python2.3.4 into your PATH (this should be added in the pacman file as dependency and provided)
  • Create one directory to install PandaJDE package via pacman. You need at least the Oct. 19th pre-release version of pacman (pacman -get Pacman:Pacman-pre-release) for installing PandaSrv (this should have been solved pacman-current should work).
   pacman -get GCL:PandaJDE
  • If you want to send out your own pilot jobs (users may ise Pilot sent out by others) you will need another directory to install PandaJS with pacman.
   pacman -get GCL:PandaJS

How to run Eowyn

  • cd into PandaJDE directory and edit the setup file to add the following ones into your PYTHONPATH for now, these will be included in the pacman file soon, so you will not need to do this. Below "your installation area" is where PandaJDE, PandaSrv, PandaJS directories sit:
   PYTHONPATH="/your installation area/PandaJDE/Protocol/:$PYTHONPATH"
   PYTHONPATH="/your installation area/PandaJDE/Executors/:$PYTHONPATH"
   PYTHONPATH="/your installation area/PandaJDE/pandajde/:$PYTHONPATH"
   PYTHONPATH="/your installation area/PandaJDE/pandajde/executorInterface/:$PYTHONPATH"
  • Source the setup file
  • Edit PandaJDE/pandajde/executorInterface/execute.py file: change numJobs=20 to numJobs=1 for easy testing. Executor will want only one job when asked by supervisor. Executor will get jobs from devdb defined by taskfk=38 and inputhint='OSG'.
  • Edit PandaJDE/Eowyn/start.py file to choose a supervisor name and password:
   #supname=raw_input('sup name:')
   #suppw=raw_input('sup password:')
   supname='name of your supervisor'
   suppw='password for your supervisor'
  • Edit PandaJDE/Eowyn/start.py file to choose devdb instead of fakedb, comment out the lines related to fakedb, and include the ones related to devdb:
   #import ProdDBFake
   #pdb=ProdDBFake.ProdDBFake(lmg)
   #pdb.avGFJTime=0                        
   #pdb.defaultcpu=20                     
   #pdb.avTime=0                          
   #pdb.failrate=0                         
   #pdb.initfailrate=0                 
   #pdb.maxjobs=100000                   

   import ProdDBProxyOracle
   #user=raw_input('db user:')
   #dbpw=raw_input('db password:')
   user='atlas_prodsys_dev'
   dbpw='password for devdb'
   pdb=ProdDBProxyOracle.ProdDBProxyOracle(lmg,user,dbpw,'devdb',None)
   pdb.connect()
  • Edit PandaJDE/Eowyn/start.py file to choose to run Panda executor:
   #import ExecutorFake
   #exe=ExecutorFake.ExecutorFake(lmg,'exezorro')
   import execute
   exe=execute.execute('panda')
  • Edit PandaJDE/Eowyn/start.py file to change the number of maximum active jobs from sup.maxActive = 50 to sup.maxActive = 5 for easy testing, this number can be controlled interactively without stopping Eowyn.
  • To run Eowyn change the background color of your terminal to white, if yours is black you will not see what is printed. Run Eowyn by:
   python start.py
  • To shutdown Eowyn, open another window and go to PandaJDE/Eowyn directory and type:
   echo "self.eowyn.shutdown = True" > Eowyn.msg
  • For other commands to control Eowyn, see PandaJDE/Eowyn/Eowyn.msg.cheat file.

How to run jobscheduler

  • cd into PandaJS directory and source the setup file.
  • Start your grid-proxy (grid-proxy-init -valid 200:00) and start condor (condor_master).
  • cd into pandajs/jobscheduler directory and edit siteinfo.py file: change 'OK' to 'NOTOK' for each entry. Comment out the entry for agt_bu_edu and add the one works for now:
   "agt_bu_edu":['atlas.bu.edu','pbs','/atlasgrid/Grid3-app','/data3','data3','
   /tmp','http://doe-dhcp241.bu.edu:8000/dq2/', {},'/atlasgrid/Grid3-app/atlas_app/
   python/python-2.4.1/Python-2.4.1/python',2,'OK'],

   #"agt_bu_edu":['agt.bu.edu','pbs','/data/apps','/data/osgdata','/data/scratc
   h','/scratch','',{},'/usr/local/IT/python-2.4.1/bin/python',2,'NOTOK'],
  • To run jobscheduler type the following, this will submit 3 pilot jobs at a time:
   python pusher.py
  • To look at the log file of a job, pilot.out, open another window and go to PandaJS/pandajs/jobscheduler/myjobs directory.


Major updates:
-- MarcoMambelli - 27 Oct 2005



Responsible: MarcoMambelli

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2007-06-15 - StefanoAntonelli
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback