pRun

Introduction

prun allows users to submit general jobs to Panda. Those jobs may run ROOT(CINT,C++,pyRoot), ARA, Python, user's executable, shell script and so on. All files under the current directory are sent to remote woker-nodes (WNs) except root files by default. Users can construct arbitrary runtime environment on the WNs and can do anything in principle. However, please avoid operations which connect to remote servers (e.g., cmt checkout, wget, and lcg-cp) unless administrators of the remote servers allow. Your job is split to many sub-jobs and they are executed in parallel, so that those operations tend to become a DDoS attack and easily break the remote servers.

Getting started

Installation

Installation procedure is different from pathena's one since prun is independent of Athena.

via tarball

$ wget https://twiki.cern.ch/twiki/pub/Atlas/PandaRun/panda-client-0.1.tar.gz
$ tar xvfz panda-client-*
$ cd panda-client-*
$ python setup.py install --prefix=[install dir]

via rpm

$ wget https://twiki.cern.ch/twiki/pub/Atlas/PandaRun/panda-client-0.1-1.noarch.rpm
$ rpm -Uvh panda-client-*

Setup

First, setup the grid runtime, or set PATHENA_GRID_SETUP_SH if you don't want to pollute environment variables. e.g.,
$ source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
or
$ export PATHENA_GRID_SETUP_SH=/afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
This page explains PATHENA_GRID_SETUP_SH. Next, source the prun's setup script according to your installation.

via tarball

$ source [install dir]/etc/panda/panda_setup.[c]sh

via rpm

$ source /etc/panda/panda_setup.[c]sh

How to run (an example of ARA job)

The usage of prun is
$ prun [options]
where --exec and --outDS are mandatory. Try
$ prun -h
to see all available options.

Here is an example of ARA job.

$ cat aratest.py

#!/bin/bash

"exec" "python" "-Wignore" "$0" "$@"

def main():
    import sys
    import user
    import ROOT
    import PyCintex
    # output
    outF = ROOT.TFile('out1.root','recreate')
    import AthenaROOTAccess.transientTree
    CollectionTree = ROOT.AthenaROOTAccess.TChainROOTAccess('CollectionTree')
    # input
    inputFiles = sys.argv[1].split(',')
    for inputFile in inputFiles:
        print "add %s" % inputFile
        CollectionTree.Add(inputFile)
    tt = AthenaROOTAccess.transientTree.makeTree(CollectionTree)
    # event loop
    for i in range(tt.GetEntries()):
        tt.GetEntry(i)
        photons = tt.PhotonAODCollection
        print [e.eta() for e in photons]

if __name__ == "__main__":
    main()
This script gets the list of input filenames via sys.argv[1] and produces an output file out1.root. You may run this script locally like
$ aratest.py AOD1.pool.root,AOD2.pool.root
Now you can submit this to Panda by using prun. All you need is essentially to convert the execution string (aratest.py AOD1.pool.root,AOD2.pool.root) to --exec "aratest.py %IN" and to give it to prun, where %IN represents a list of input files and will be replaced to the real list on each WN.
$ prun --exec "aratest.py %IN" --outDS user08.TadashiMaeno.test123 --inDS valid1.006384.PythiaH120gamgam.recon.AOD.e322_s412_r577 --outputs out1.root --nFiles 5 --athenaTag=14.2.24
prun gathers all files under --workDir (default=./) and sends them to WNs. A single prun job instantiates one buildGen job and some runGen jobs like pathena jobs. Once jobs are submitted to Panda, you can see what's going on using PandaMonitor. buildGen stores your source files to the remote SE and activates runGen jobs as soon as it finishes. The aratest.py runs in runGen jobs. The argument of the --exec option is executed after converting %IN to a list of input files on each WN. The list is a comma-separated string including names of input files available in the 'current directory' on the WN. Output files are defined by the --outputs option and are auomatically renamed to DatasetName_SerialNumber_OriginalName, e.g, user08.TadashiMaeno.test123._00001.out1.root. You can retrieve output files by using dq2 tools. See another example as well.


More Examples

The following shows some example implementations to run ROOT and/or Python jobs. There are other possible implementations, of course. Basically the prun framework has no constraint, so users can write their own scripts or applications as they like. The point is that the --exec option defines how the list of input files is given to scripts/applications.

Run CINT macro

Here is an example of CINT macro.
$ cat macrotest.C

#include <TROOT.h>
#include <TChain.h>
#include <TFile.h>

gROOT->Reset();

void macrotest()
{
  std::string argStr;

  std::cin >> argStr;

  // split by ','
  std::vector<std::string> fileList;
  for (int i=0,n; i <= argStr.length(); i=n+1)
    {
      n = argStr.find_first_of(',',i);
      if (n == string::npos)
        n = argStr.length();
      string tmp = argStr.substr(i,n-i);
      fileList.push_back(tmp);
    }

  // open input files
  TChain fChain("CollectionTree");
  for (int iFile=0; iFile<fileList.size(); ++iFile)
    {
      std::cout << "open " << fileList[iFile].c_str() << std::endl;
      fChain.Add(fileList[iFile].c_str());
    }

  Int_t           EventNumber;
  TBranch        *b_EventNumber;
  fChain.SetBranchAddress("EventNumber", &EventNumber, &b_EventNumber);

  // main loop
  Long64_t nentries = fChain.GetEntriesFast();
  for (Long64_t jentry=0; jentry<nentries;jentry++)
    {
      Long64_t ientry = fChain.LoadTree(jentry);
      if (ientry < 0)
        break;
      fChain.GetEntry(jentry);

      std::cout << EventNumber << std::endl;
    }
}
This macro reads a string via stdin and splits the string to the list of input files. You may run this locally like
$ echo NTUP1.root,Ntup2.root | root.exe macrotest.C
So you can submit it like
$ prun --exec "echo %IN | root.exe macrotest.C" --athenaTag=14.2.24 --inDS ...
ROOT applications requires the --athenaTag option which is explained in this section. Any stand-alone ROOT is not available on the grid WNs. So you have to use ROOT included in an Athena release.

Run C++ ROOT

Although pre-compiled binaries can be sent to WNs, it is safe to re-compile on remote WNs to avoid library mismatch. In this case, you need to prepare source files, Makefile and a wrapper script. Here is a C++ example.
$ cat cpptest.cc

#include <string>
#include <vector>
#include <iostream>
#include <stdlib.h>

#include "TROOT.h"
#include "TFile.h"
#include "TTree.h"
#include "TChain.h"
#include "TBranch.h"

int main(int argc, char **argv)
{
  Int_t nFile = atoi(argv[1]);
  std::vector<std::string> fileList;
  for (Int_t iFile=0; iFile<nFile; ++iFile)
    {
      fileList.push_back(argv[2+iFile]);
    }
  // open input files
  TChain fChain("CollectionTree");
  for (unsigned int iFile=0; iFile<fileList.size(); ++iFile)
    {
      std::cout << "open " << fileList[iFile].c_str() << std::endl;
      fChain.Add(fileList[iFile].c_str());
    }

  Int_t           EventNumber;
  TBranch        *b_EventNumber;
  fChain.SetBranchAddress("EventNumber", &EventNumber, &b_EventNumber);

  // main loop
  Long64_t nentries = fChain.GetEntriesFast();
  for (Long64_t jentry=0; jentry<nentries;jentry++)
    {
      Long64_t ientry = fChain.LoadTree(jentry);
      if (ientry < 0)
        break;
      fChain.GetEntry(jentry);

      std::cout << EventNumber << std::endl;
    }
}
Makefile could be
$ cat Makefile

ROOTCFLAGS    = $(shell root-config --cflags)
ROOTLIBS      = $(shell root-config --libs)
ROOTGLIBS     = $(shell root-config --glibs)

CXX           = g++
CXXFLAGS      =-I$(ROOTSYS)/include -O -Wall -fPIC
LD            = g++
LDFLAGS       = -g
SOFLAGS       = -shared

CXXFLAGS     += $(ROOTCFLAGS)
LIBS          = $(ROOTLIBS)
GLIBS         = $(ROOTGLIBS)

OBJS          = cpptest.o

cpptest: $(OBJS)
        $(CXX) -o $@ $(OBJS) $(CXXFLAGS) $(LIBS)

# suffix rule
.cc.o:
        $(CXX) -c $(CXXFLAGS) $(GDBFLAGS) $<

# clean
clean:
        rm -f *~ *.o *.o~ core
Now you need a wrapper script which compiles source files and runs the executable.
$ cat cpptestwrapper.py

#!/bin/bash

"exec" "python" "-Wignore" "$0" "$@"

import os
import sys

# make
os.system('make')

# execute
inFiles = sys.argv[1].split(',')
com = './cpptest %d' % len(inFiles)
for inFile in inFiles:
    com += " %s" % inFile
status = os.system(com)

# return
sys.exit(status)
Then
$ ls
cpptest.cc
cpptestwrapper.py
Makefile
...
$ prun --exec "cpptestwrapper.py %IN" --athenaTag=14.2.24 --inDS ...

Run pure Python job

This is an example to execute a completely general job. Here is a python script.
$ cat purepython.py

#!/usr/bin/python

import sys
print sys.argv
f =open('out.dat','w')
f.write('xxxxxxxx')
f.close()
sys.exit(0)
Then
$ prun -v --exec "purepython.py %IN" --inDS ...
Python is always available on remote WNs. So you don't need to setup Athena on WNs.

Run jobs without input files

If --inDS is not specified, --nJobs (default=1) subjobs will be instantiated. Perhaps you may set random seeds using %RNDM in the --exec option. e.g,
$ prun --exec "somescript %RNDM=123 %RNDM=456" --outDS user...
where %RNDM=basenumber (e.g., %RNDM=100) will be incremented per sub-job.

Run pyAthena+ARA

Essentially the difference from the above example is that this runs athena.py instead of python. This time the list of input files is given via sys.stdin.
$ cat arestest.py

theApp.initialize()

import sys
import user
import ROOT
import PyCintex
# output
outF = ROOT.TFile('out1.root','recreate')
import AthenaROOTAccess.transientTree
CollectionTree = ROOT.AthenaROOTAccess.TChainROOTAccess('CollectionTree')
# input
inputFiles = sys.stdin.readline()[:-1].split(',')
for inputFile in inputFiles:
    print "add %s" % inputFile
    CollectionTree.Add(inputFile)
tt = AthenaROOTAccess.transientTree.makeTree(CollectionTree)
# event loop
for i in range(tt.GetEntries()):
    tt.GetEntry(i)
    photons = tt.PhotonAODCollection
    print [e.eta() for e in photons]

sys.exit(0)
Then
$ prun --exec "echo %IN | athena.py arestest.py" --athenaTag=14.2.24  --inDS ... 
Of course the --athenaTag option is required. Note that if you need to compile multiple Athena packages it would be more convenient to use pathena --ara/--ares instead. See this page.


Tips

Use Athena Runtime on WN

Sometimes you may need Athena Runtime on WNs. e.g.,
$ prun --athenaTag=14.2.24 ...
This will setup Athena-14.2.24 runtime on remote WNs. The syntax of --athenaTag may be familiar to Athena users. If you want to use caches, --athenaTag=14.2.24.3,AtlasProduction, for example.

Archive output files

When you have a lot of output files on WN, you may want to archive them.
$ prun --outputs "abc.data,JiveXML_*.xml" ...
this will produce DatasetName_SerialNumber_abc.data and DatasetName_SerialNumber_JiveXML_XYZ.xml.tgz. The latter will contain all JiveXML_*.xml files. You need " or \ to disable shell-globing when the wild-card is used.

Send jobs to a particular site

prun automatically chooses an appropriate site by using information about dataset location, site occupancy, and user's VOMS FQAN. But users can send jobs to a particular cloud/site using --cloud= or --site option. e.g.,
$ prun --cloud FR ...
$ prun --site TRIUMF ...

Use selected files in the input dataset

prun has the --match option which allows user to choose files matching a given pattern.
$ prun --match "*AOD*" ...


TODO

  • development of book-keeping tool


Major updates:
-- TadashiMaeno - 14 Nov 2008



Responsible: TadashiMaeno

Never reviewed

Edit | Attach | Watch | Print version | History: r99 | r12 < r11 < r10 < r9 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r10 - 2008-11-16 - TadashiMaeno
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback