pRun

Introduction

prun allows users to submit general scripts to Panda.

Getting started

Installation

Installation procedure is different from pathena's one since Athena Runtime is not mandatory.

via tarball

$ wget https://twiki.cern.ch/twiki/pub/Atlas/PandaRun/panda-client-0.1.tar.gz
$ tar xvfz panda-client-*
$ cd panda-client-*
$ python setup.py install --prefix=[install dir]

via rpm

$ wget https://twiki.cern.ch/twiki/pub/Atlas/PandaRun/panda-client-0.1-1.noarch.rpm
$ rpm -Uvh panda-client-*

Setup

First, setup the grid runtime, or set PATHENA_GRID_SETUP_SH if you don't want to pollute environment variables. e.g.,
$ source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
or
$ export PATHENA_GRID_SETUP_SH=/afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
This section explains what is PATHENA_GRID_SETUP_SH. Then, source the prun setup script according to your installation.

via tarball

$ source [install dir]/etc/panda/panda_setup.[c]sh

via rpm

$ source /etc/panda/panda_setup.[c]sh

How to run

The usage of prun is
$ prun [options] script
Try
$ prun -h
to see all available options.

Here is a toy example. A script xoxo.py

$ cat xoxo.py

#!/usr/bin/python

import os
import sys
inNames = sys.argv[1]

print sys.argv[2]

totalSize = 0
for inName in inNames.split(','):
    totalSize += os.path.getsize(inName)

outName = 'xyz.dat'
f = open (outName,'w')
f.write(totalSize)
f.close()

import commands
print commands.getoutput('echo %s > %s' % (sys.argv[2],'abc.txt'))

sys.exit(0)
this script calculates the total size of input files, and produces two output files on each WN. How to submit this script is
$ prun --inDS fdr08_run2.0052280.physics_Bphys.merge.AOD.o3_f47_m26 --outDS user08.TadashiMaeno.test123 --outputs xyz.dat,abc.txt -p "%IN 123" --nFiles 2 xoxo.py
prun gathers all files under --workDir (default=./) and sends them to WNs. Single prun job instantiates one buildGen job and some runGen jobs like pathena job. buildGen stores your source files to the remote SE and activates runGen jobs as soon as it finishes. The script is executed in runGen jobs. The argument of the -p option is given to the script after converting %IN to a list of input files. So the job executes something like
$ xoxo.py AOD.01234._0002.pool.root.1,AOD.01234._0002.pool.root.1 123
on WNs. Input files are available in the 'current directory' on the WN. Output files are auomatically renamed to DatasetName_SerialNumber_OriginalName, e.g, user08.TadashiMaeno.test123._00001.xyz.dat. Output files can be retrieved by using dq2 tools.

More Examples

Run jobs without input files

If you don't specify --inDS, --nJobs (default=1) subjobs will be instantiated. Perhaps you may set random seeds using %RNDM in the -p option. e.g,
$ prun -p "%RNDM=123 %RNDM=456" myscript --outDS user...
where %RNDM=basenumber (e.g., %RNDM=100) will be incremented per sub-job.

Use Athena Runtime on WN

Sometimes you may need Athena Runtime on WNs. e.g.,
$ prun --athenaTag=14.2.24,32,setup ...
This will setup Athena-14.2.24 runtime on remote WNs. The syntax of --athenaTag may be familiar to Athena users. If you want to use caches, 14.2.24.3,32,AtlasProduction,setup..., for example.

Archive output files

When you have a lot of output files on WN, you may want to archive them.
$ prun --outputs "abc.data,JiveXML_*.xml" ...
this will produce DatasetName_SerialNumber_abc.data and DatasetName_SerialNumber_JiveXML_XYZ.xml.tgz. The latter will contain all JiveXML_*.xml files.

Send jobs to a particular site

prun automatically chooses an appropriate site by using information about dataset location, site occupancy, and user's VOMS FQAN. But users can send jobs to a particular cloud/site using --cloud= or --site option. e.g.,
$ prun --cloud FR ...
$ prun --site TRIUMF ...


TODO

  • development of book-keeping tool


Major updates:
-- TadashiMaeno - 14 Nov 2008



Responsible: TadashiMaeno

Never reviewed

Edit | Attach | Watch | Print version | History: r99 | r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r1 - 2008-11-15 - TadashiMaeno
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback