TTP Job Submission to UniGeneva Network

This wiki is a brief overview and example tutorial of submitting TTP jobs on the unige batch system. Recipe type tutorial begins at Recipe from 25.09.2009 for release 15.3.0. Background info is in following section:Background.

/atlas/users/olya/genevaBatch/HOWTO
Provides updated short tutorial. The other sections are for reference.

Background

Knowledge of setup of atlas framework account and grid access is assumed at the level of AtlasWorkBookAdvancedSetup.
AtlasWorkBook - basic initial configuration of framework and grid access
AtlasWorkBookAdvancedSetup - advanced configuration of framework

Unige is accessible through user machines:

  • atlas00#.unige.ch where # is from 1 to 3
    • unige can be accessed solely from lxlpus
    • user name and password are identical for lxplus and unige
    • machines atlas001 and atlas002 have "screen" command loaded

genevaBatch

All files for job submission will be located under directory genevaBatch

A template version is available at
/atlas/users/eptacek/Template/genevaBatch TEMPLATE HAS NOT BEEN TESTED

This should be copied into user directory
/atlas/users/"username"/genevaBatch

Summary of directories in genevaBatch

Releases
Is the directory that contains the releases

Releases/"releasedir"
Release directory
Ex: genevaBatch/Release/15.3.0/

ntuples
Contains output ntuples that are generated by TTPAthena per run on batch jobs

logs
Contains log files that are generated by TTPAthena per run on batch jobs

Summary of Environment Configuration files (a.k.a "requirements" used to configure for specific environment setup)

Permanent

requirements in genevaBatch/Releases
set   CMTSITE  CERN
set   SITEROOT /afs/cern.ch
macro ATLAS_DIST_AREA ${SITEROOT}/atlas/software/dist

macro ATLAS_TEST_AREA /atlas/users/###User Name (e.g. "olya")###/genevaBatch/Releases

use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)

######################################### EOF$PWD ##################################

requirements in genevaBatch
set   CMTSITE  CERN
set   SITEROOT /afs/cern.ch
macro ATLAS_DIST_AREA ${SITEROOT}/atlas/software/dist

macro ATLAS_TEST_AREA /atlas/users/###User Name (e.g. "olya")###/genevaBatch
apply_tag oneTest

use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)

######################################### EOF$PWD ##################################
setup.sh in genevaBatch/Releases
setup.sh generated by cmt config in Release/ with
-path = /atlas/user/"User Name"/genevaBatch/
replaced by
-path=$PWD

setup.sh in genevaBatch
copy of setup.sh from genevaBatch/Releases

Dynamically generated

Dynamically generated files are saved on batch machine per job; deleted afterwards
requirements
Generated per batch job on batch machine by makeSetup.sh

set   CMTSITE  CERN
set   SITEROOT /afs/cern.ch
macro ATLAS_DIST_AREA ${SITEROOT}/atlas/software/dist
apply_tag oneTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
macro ATLAS_TEST_AREA /tmp/scratch

  • directory "/tmp/scratch" is on the batch machine

linked directories
These are set in bathena.sh

ln -s /afs/cern.ch/atlas/project/database/DBREL/packaging/DBRelease/current/geomDB geomDB
ln -s /afs/cern.ch/user/a/atlcond/coolrep/sqlite200 sqlite200
ln -s $extraSources/InstallArea InstallArea

  • $extraSources is /atlas/users/###User Name (e.g. "olya")###/genevaBatch/Releases/###Release Directory (e.g. "AtlasOff
line-15.3.0")###
    • Ex:
      /atlas/users/eptacek/genevaBatch/Releases/AtlasOffline-15.3.0

Summary of Script Files (files that are executable or are called by executable files)

submitBatch.sh

Called by user

Master file
Run to submit batch jobs. Divides dataset into groups of files. Runs a batch job for each group of files.

Functions called are:

Line 3 requires replacement of text inside ### ###

Primary commands

Lines 6 or 23

geometry= ... 
set detector geometry - line above group of datasets being used is un-commented.

Line groups 7-20 and 24-30

dirSet="..."
#dirSet="..."
#dirSet="..."
list possible datasets available. one of these lines is uncommented per run of batch jobs.

Line 34

bunchSize
Number of files sent into one output root ntuple file.


Lines 42 - 51(oexecute function) are executed once per group of files from input dataset. oexecute submits batch job.
Line 46, 47
echo "#!/bin/sh
$bathenascript  $fileList $geometry "  > tmp.sh
generate tmp.sh; $bathenascript is bathena.sh
$fileList is string containing group of bunchSize(5 usually) consecutive files that will be used as inputs to generate single root ntuple file for single batch job.
$geometry is detector geometry.

Line 49

qsub -q public -l mem=1500mb ./tmp.sh
submits batch job



Lines 55 - 91 divide input dataset into groups of "bunchSize" number of files and call oexecute to submit batch job per group.

bathena.sh

Called by submitBatch.sh

MasterServant file that runs all other files in batch job
run on the batch machine per batch job
Configures the directory and environment inside batch system to run job
Runs athena on the batch machine to produce each output ntuple root file
Saves results

Accepts 2 inputs "$fileList" and "$geometry"

  • $fileList is $inputFile
  • $fileList is the list of input files that will be run over and output to single ntuple.root file. Passed from submitBatch.sh
  • $geometry is the detector geometry (a.k.a. ATLAS-CSC-01-02-00)

Functions called are:

Lines 2-4, 23-25, 53, 70, 74, 76, and 84 require replacement of text inside ### ### with that indicated

  • To reconfigure for different release and TTP replace
    • line 2 ###Release Number ###
    • line 3 ###Release Directory ###
    • line 4 ###TTP Number ###

Primary commands

The line 24

 source /atlas/users/###User Name (e.g. "olya")###/genevaBatch/setup.sh -tag=opt,32,simpleTest,runtime,releases,${Release}
sets up environment variables

Line 66

 athena.py test.py > ttp.log 
Runs job
reference:test.py

makeSetup.sh

Called by bathena.sh

Create requirements file used by batch job on batch machine.

No modifications required

makeJobOption.sh

Called by bathena.sh

Creates test.py.
Is called once per batchjob.

Accepts inputs $fileList and $geometry

  • $fileList is inputFile
  • $fileList is a string of input files for single run of TTPAthena generating single root ntuple file.
  • both inputs are passed directly from bathena.sh
  • inputs originate from submitBatch.sh

No modifications required

cleanup

Called by user

  • script used to clean directory after run of batch submission
  • eliminates the "tmp" files that are automatically generated by batch system
  • saves old logs and batch system output and old ntuples under
     oldTest

To run

 ./cleanup 

Summary of dynamically generated script files

tmp.sh

Generated by oexecute function of submitBatch.sh once per batch job.

Calls bathena.sh which runs the batch job and saves results.
Strings $fileList and $geometry are passed to bathena.sh as inputs.

Generic Form
#!/bin/sh
$bathenascript  $fileList $geometry

  • $bathenascript is bathena.sh passed from line 3 of submitBatch.sh
    • Ex:
      /atlas/users/eptacek/genevaBatch/bathena.sh
  • $fileList is string of input file names.
    • Ex:
       \"/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04792.pool.root.1\",
      \"/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04793.pool.root.1\"
  • $geometry is detector geometry configuration
    • Ex:
      ATLAS-GEO-02-01-00

Ex from test run
#!/bin/sh
   /atlas/users/eptacek/genevaBatch/bathena.sh  \"/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04004.pool.root.1\" ATLAS-GEO-02-01-00

test.py

Generated by makeJobOption.py which is called by bathena.sh

Is executed in the line

athena.py test.py > ttp.log
of bathena.sh to run TTPAthena

Generic Form
PoolRDOInput=[$1]
DetDescrVersion = "$2"
EvtMax = -1
include("tauPerform_RDO_topOptions.py")

  • $1 is $fileList from makeJobOption.sh <- bathena.sh <- submitBatch.sh. String of input file names for TTPAthena.
    • Ex:
      "/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04792.pool.root.1","/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04793.pool.root.1"
  • $2 is $geometry from makeJobOption.sh <- bathena.sh <- submitBatch.sh. Detector geometry
    • Ex:
      ATLAS-GEO-02-01-00

Ex from test run
PoolRDOInput=["/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04792.pool.root.1","/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04793.pool.root.1","/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04794.pool.root.1","/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04795.pool.root.1","/atlas/data3/datasets/mc08.106052.PythiaZtautau.digit.RDO.e347_s462_d154_tid045023/RDO.045023._04796.pool.root.1"]
DetDescrVersion = "ATLAS-GEO-02-01-00"
EvtMax = -1
include("tauPerform_RDO_topOptions.py")

Output Directories

Recipe from 25.09.2009 for release 15.3.0

DOES NOT WORK Test directory is stored under:
/atlas/users/eptacek/genevaBatch
Template directory is
/atlas/users/eptacek/Template/genevaBatch

Setup

Initial

copy
 /atlas/users/eptacek/Template/genevaBatch 
to user directory
 /atlas/users/"user name"/genevaBatch

For bathena.sh, submitBatch.sh, genevaBatch/requirements and genevaBatch/Releases/requirements replace text inside "### ###" with indicated replacement.

Per Release

% cd genevaBatch/Releases/
% source /afs/cern.ch/sw/contrib/CMT/v1r20p20090520/mgr/setup.sh
% cmt config
% mkdir 15.3.0

in setup.sh change -path=/atlas/users/"user name/genevaBatch/Releases to -path=$PWD

% cp setup.sh ../
% source /atlas/users/eptacek/genevaBatch/Releases/setup.sh -tag=opt,32,simpleTest,runtime,releases,15.3.0
% cd 15.3.0
% cmt co -r TrigTauPerformAthena-00-06-05 Trigger/TrigAnalysis/TrigTauAnalysis/TrigTauPerformAthena
% cmt co -r TrigTauPerformNtuple-00-06-03 Trigger/TrigAnalysis/TrigTauAnalysis/TrigTauPerformNtuple
% cd Trigger/TrigAnalysis/TrigTauAnalysis/TrigTauPerformAthena/cmt
% cmt bro gmake

errors


Failure to compile can be due to broken path to installArea
source /atlas/users/eptacek/genevaBatch/Releases/setup.sh -tag=opt,32,simpleTest,runtime,releases,15.3.0
should be executed from "genevaBatch/Releases" directory.
The path in setup.sh should be reset to $PWD before executing source command
The correct installArea /genevaBatch/Releases/15.3.0/InstallArea should appear at beginning of $PYTHONPATH... etc. after execution of source command.

Run

% source submitBatch.sh

More Info

Scripts are written in shell scripting language:
shell scripting tutorial

sed command section in shell scripting tutorial

Additional information is also available in:

/atlas/users/olya/genevaBatch/HOWTO

Information on Unige System:
Twiki for the unige system
twiki for submitting batch jobs on the unige system

-- PtacekE - 2009-09-25

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2009-09-29 - Eptacek
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback