Panda Integration Testing

Introduction

This document describes how to run Eowyn with Panda to transfer CERN DB jobs to PandaDB and how to run jobscheduler to execute jobs in OSG.

Installation

   $ pacman -get GCL:PandaJDE
   $ pacman -get GCL:PandaSrv
   $ pacman -get GCL:PandaJS

How to run Eowyn

  • cd into PandaJDE directory and edit the setup file to add the following ones into your PYTHONPATH for now, these will be included in the pacman file soon, so you will not need to do this. Below "your installation area" is where PandaJDE, PandaSrv, PandaJS directories sit:
   PYTHONPATH="/your installation area/PandaJDE/Protocol/:$PYTHONPATH"
   PYTHONPATH="/your installation area/PandaJDE/Executors/:$PYTHONPATH"
   PYTHONPATH="/your installation area/PandaJDE/pandajde/:$PYTHONPATH"
   PYTHONPATH="/your installation area/PandaSrv/pandasrv/:$PYTHONPATH"
   PYTHONPATH="/your installation area/PandaJDE/pandajde/executorInterface/:$PYTHONPATH"
  • Source the setup file
  • Edit PandaJDE/pandajde/executorInterface/execute.py file: change numJobs=20 to numJobs=1 for easy testing. Executor will want only one job when asked by supervisor. Executor will get jobs from devdb defined by taskfk=38 and inputhint='OSG'.
  • Edit PandaJDE/Eowyn/start.py file to choose a supervisor name and password:
   #supname=raw_input('sup name:')
   #suppw=raw_input('sup password:')
   supname='name of your supervisor'
   suppw='password for your supervisor'
  • Edit PandaJDE/Eowyn/start.py file to choose devdb instead of fakedb, comment out the lines related to fakedb, and include the ones related to devdb:
   #import ProdDBFake
   #pdb=ProdDBFake.ProdDBFake(lmg)
   #pdb.avGFJTime=0                        
   #pdb.defaultcpu=20                     
   #pdb.avTime=0                          
   #pdb.failrate=0                         
   #pdb.initfailrate=0                 
   #pdb.maxjobs=100000                   

   import ProdDBProxyOracle
   #user=raw_input('db user:')
   #dbpw=raw_input('db password:')
   user='atlas_prodsys_dev'
   dbpw='password for devdb'
   pdb=ProdDBProxyOracle.ProdDBProxyOracle(lmg,user,dbpw,'devdb',None)
   pdb.connect()
  • Edit PandaJDE/Eowyn/start.py file to choose to run Panda executor:
   #import ExecutorFake
   #exe=ExecutorFake.ExecutorFake(lmg,'exezorro')
   import execute
   exe=execute.execute('panda')
  • Edit PandaJDE/Eowyn/start.py file to change the number of maximum active jobs from sup.maxActive = 50 to sup.maxActive = 5 for easy testing, this number can be controlled interactively without stopping Eowyn.
  • To run Eowyn change the background color of your terminal to white, if yours is black you will not see what is printed. Run Eowyn by:
   $ python start.py
  • To shutdown Eowyn, open another window and go to PandaJDE/Eowyn directory and type:
   $ echo "self.eowyn.shutdown = True" > Eowyn.msg
  • For other commands to control Eowyn, see PandaJDE/Eowyn/Eowyn.msg.cheat file.

How to run jobscheduler

  • cd into PandaJS directory and source the setup file.
  • Start your grid-proxy (grid-proxy-init -valid 200:00) and start condor (condor_master).
  • cd into pandajs/jobscheduler directory and edit siteinfo.py file: change 'OK' to 'NOTOK' for each entry. Comment out the entry for agt_bu_edu and add the one works for now:
   "agt_bu_edu":['atlas.bu.edu','pbs','/atlasgrid/Grid3-app','/data3','data3','
   /tmp','http://doe-dhcp241.bu.edu:8000/dq2/', {},'/atlasgrid/Grid3-app/atlas_app/
   python/python-2.4.1/Python-2.4.1/python',2,'OK'],

   #"agt_bu_edu":['agt.bu.edu','pbs','/data/apps','/data/osgdata','/data/scratc
   h','/scratch','',{},'/usr/local/IT/python-2.4.1/bin/python',2,'NOTOK'],
  • To run jobscheduler type the following, this will submit 3 pilot jobs at a time:
   $ python pusher.py
   (or, to keep it running in the background, do "python pusher.py >& pusher.out &")
  • To look at the log file of a job, pilot.out, open another window and go to PandaJS/pandajs/jobscheduler/myjobs directory.

Major updates:
-- NurcanOzturk - 25 Oct 2005



Responsible: NurcanOzturk

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2007-06-15 - StefanoAntonelli
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback