-- ClemenciaMora - 18-Feb-2010

How to run transforms on the Grid

MinBias configuration that we can base our jobs upon:

The job_trf used by the minbias group are in: /afs/cern.ch/atlas/groups/fatras/minbias_pp0_scripts/geant_minbias/geant_job.sh The only difference is that we need to run the standard configuration (they were switching off calo and muons) The parameters used are in: /afs/cern.ch/atlas/groups/fatras/minbias_pp0_scripts/geant_minbias/geant_submitter.sh

Currently we're using Jamie's configuration for the Tier0 reconstruction of MinBias (except w/o the Lucid stuff and with Calo digi turned on), so the final setup looks like:

export GEO_TAG='ATLAS-GEO-08-05-00'
export ALIGNMENT_TAG='Nominal'
export SIM_CONDTAG='OFLCOND-SIM-BS900-00'
export RECO_CONDTAG='OFLCOND-DR-BS900-ANom-03'

source cmthome/setup.sh AtlasProduction,15.3.1.21,32,opt,runtime

# simulation step
csc_atlasG4_trf.py inputEvgenFile=evgen.pool.root outputHitsFile=hits.pool.root maxEvents=200 skipEvents=$SKIP_EVTS randomSeed=$RANDOM geometryVersion=$GEO_TAG physicsLi
st=QGSP_BERT jobConfig=VertexFromCondDB.py conditionsTag=$SIM_CONDTAG

source cmthome/setup.sh AtlasProduction,15.6.1.5,32,opt,runtime,slc4

# digitization step
csc_digi_trf.py inputHitsFile=hits.pool.root outputRDOFile=digit.pool.root maxEvents=-1 skipEvents=0 digiSeedOffset1=$RANDOM digiSeedOffset2=$RANDOM geometryVersion=$GEO
_TAG conditionsTag=$RECO_CONDTAG digiRndmSvc=AtRanluxGenSvc addCaloDigi=True IgnoreConfigError=False beamGasHitsFile=NONE beamHaloHitsFile=NONE samplingFractionDbTag=QG
SP_BERT minbiasHitsFile=NONE cavernHitsFile=NONE jobConfig=NONE doAllNoise=NONE triggerConfig=MC_InitialBeam_v1_no_prescale

# reco
source cmthome/setup.sh AtlasProduction,15.6.3.6,32,opt,runtime
Reco_trf.py autoConfiguration=everything inputRDOFile=digit.pool.root outputESDFile=esd.pool.root preInclude=RecJobTransforms/SetJetConstants-02-000.py,RecJobTransforms/
NoTrackSlimming.py preExec="rec.Commissioning.set_Value_and_Lock(True);jobproperties.Beam.energy.set_Value_and_Lock(900*Units.GeV);muonRecFlags.doCSCs=False;muonRecFlags
.useAlignmentCorrections=True;muonCombinedRecFlags.doMuidVertexAssociator=False;muonCnvFlags.MdtCablingMode='new';from AthenaCommon.BFieldFlags import jobproperties;jobp
roperties.BField.solenoidOn.set_Value_and_Lock(True);jobproperties.BField.endcapToroidOn.set_Value_and_Lock(True);jobproperties.BField.barrelToroidOn.set_Value_and_Lock(
True)" postExec="import MDTcabling.MDTcablingConfig;ServiceMgr.MDTcablingSvc.RODfile='AtlasRODmap.data';ServiceMgr.MDTcablingSvc.MEZfile='AtlasMEZmap.data';ServiceMgr.MD
TcablingSvc.ShiftLinks=False" triggerConfig="MCRECO:InitialBeam_v1"

Translated into pAthena syntax, the evgen command is for example:

pathena --trf="csc_evgen_trf.py  runnumber=107002 firstevent=1 maxevents=5000 randomseed=%RNDM:100  jobconfig=MC8.107002.singlepart_e_Et20.py outputevgenfile=%OUT.EVNT.pool.root" --outDS=user10.ClemenciaMoraHerrera.pathena_test1 --cloud=DE --site=ANALY_CSCS

Where the last 2 parameters are optional if one wants to send the jobs to the T2 in Manno.

What EGamma group really needs:

  • The 4 samples we would like to produce are: 10k photons and 10k electrons, pT = 40 GeV, 2.0 < eta< 2.5, flat phi, Geometries: ATLAS-GEO-08-00-02 (the standard one we use as reference) and ATLAS-GEO-08-05-00 (the new one). Below is the example evgen file for the electron file (the photon is the same using PDGcode: constant 22 )
# Single electrons
# J.F. Arguin, Dec. 2005

# Set output level threshold (2=DEBUG, 3=INFO, 4=WARNING, 5=ERROR, 6=FATAL )

from AthenaCommon.AlgSequence import AlgSequence 
topAlg = AlgSequence("TopAlg") 

from ParticleGenerator.ParticleGeneratorConf import ParticleGenerator
topAlg += ParticleGenerator()


ParticleGenerator = topAlg.ParticleGenerator

# For VERBOSE output from ParticleGenerator.
ParticleGenerator.OutputLevel = 1

ParticleGenerator.orders = [
 "PDGcode: sequence -11 11",   #HERE FOR PHOTONS IS  "PDGcode: constant 22",
 "et: constant 40000",
 "eta: flat 2.0 2.5",
 "phi: flat -3.14159 3.14159"
 ]

#==============================================================
#
# End of job options file
#
###############################################################
#--------------------------------------------------------------
# Configuration for EvgenJobTransforms
#--------------------------------------------------------------
from EvgenJobOptions.SingleEvgenConfig import evgenConfig

More useful things

  • In order to get the available transforms in a release (once one has setup athena) you can run trf_ls. Then, for example, grepping for those that do event generation you get:
> trf_ls | grep ev 
csc_evgen08_trf.py
csc_evgen08new_trf.py
csc_evgen900_trf.py
csc_evgenHI_trf.py
csc_evgenHInew_trf.py
csc_evgenTruthJetMet08_trf.py
csc_evgenTruthJetMet08new_trf.py
csc_evgenTruthJetMet_trf.py
csc_evgen_input_trf.py
csc_evgen_trf.py
csc_evgennew_trf.py

Pathena commands for job submission

This section describes the setup and exact pathena commands used to produce single electron/photon samples from generation, simulation, digitization to reconstruction using a particular geometry.

Generation step

# setup release 
source cmthome/setup.sh -tag=AtlasProduction,15.3.1.21,32,opt,runtime
# setup pathena environment
export PATHENA_GRID_SETUP_SH=/afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
source ~/pandabin/etc/panda/panda_setup.sh

The pathena transform command for event generation is:

Single photons

pathena --trf="csc_evgen_trf.py  runnumber=107041 firstevent=1 maxevents=25 randomseed=%RNDM:400  jobconfig=genJO_photons.py outputevgenfile=%OUT.EVNT.pool.root" 
--outDS=user10.MoritzBackes.pathena_photons_evgen 
--cloud=DE 
--site=ANALY_CSCS 
--split=400 
Single electrons
pathena --trf="csc_evgen_trf.py  runnumber=107004 firstevent=1 maxevents=25 randomseed=%RNDM:400  jobconfig=genJO_electrons.py outputevgenfile=%OUT.EVNT.pool.root" 
--outDS=user10.MoritzBackes.pathena_electrons_evgen 
--cloud=DE 
--site=ANALY_CSCS 
--split=400 
where the modified generator jobOptions file (genJO_photons.py, genJO_electrons.py) can be found here. Before submission make sure your outDS name is unique and uses your name othervise the submission will fail.

Simulation step

Here we are using the same release as for generation. However, to run on a recent geometry you also need to get the latest database release tarfile since the latest DB release might not be installed at your grid site. Look here to find the location of this file. In this particular example we copied

/afs/cern.ch/atlas/www/GROUPS/DATABASE/pacman4/DBRelease/DBRelease-8.9.1.tar.gz
to the working directory. The pathena command for simulation is:

Single photons

pathena --trf="csc_atlasG4_trf.py inputEvgenFile=%IN outputHitsFile=%OUT.HITS.pool.root maxEvents=-1 skipEvents=0 randomSeed=%RNDM:100 geometryVersion=ATLAS-GEO-08-05-00 
physicsList=QGSP_BERT jobConfig=VertexFromCondDB.py DBRelease=DBRelease-8.9.1.tar.gz  conditionsTag=OFLCOND-SIM-BS900-00" 
--inDS=user10.MoritzBackes.pathena_photons_evgen
--outDS=user10.MoritzBackes.pathena_photons_sim_geo-08-05-00 
--cloud=DE 
--split=400 
--dbRelease=ddo.000001.Atlas.Ideal.DBRelease.v080901:DBRelease-8.9.1.tar.gz
Single electrons
pathena --trf="csc_atlasG4_trf.py inputEvgenFile=%IN outputHitsFile=%OUT.HITS.pool.root maxEvents=-1 skipEvents=0 randomSeed=%RNDM:100 geometryVersion=ATLAS-GEO-08-05-00 
physicsList=QGSP_BERT jobConfig=VertexFromCondDB.py DBRelease=DBRelease-8.9.1.tar.gz  conditionsTag=OFLCOND-SIM-BS900-00" 
--inDS=user10.MoritzBackes.pathena_electrons_evgen
--outDS=user10.MoritzBackes.pathena_electrons_sim_geo-08-05-00
--cloud=DE 
--split=400 
--dbRelease=ddo.000001.Atlas.Ideal.DBRelease.v080901:DBRelease-8.9.1.tar.gz

Digitization step

Here we are using a different release, so again:

# setup release 
source cmthome/setup.sh -tag=AtlasProduction,15.6.1.5,32,opt,runtime,slc4
# setup pathena environment
export PATHENA_GRID_SETUP_SH=/afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
source ~/pandabin/etc/panda/panda_setup.sh
Again you need to copy the DBRelease tar file to your working directory. The pathena command for digitization is:

Single photons

pathena --trf="csc_digi_trf.py inputHitsFile=%IN outputRDOFile=%OUT.digit.pool.root maxEvents=-1 skipEvents=0 digiSeedOffset1=$RANDOM digiSeedOffset2=$RANDOM 
geometryVersion=ATLAS-GEO-08-05-00 conditionsTag=OFLCOND-SIM-BS900-00 digiRndmSvc=AtRanluxGenSvc addCaloDigi=True IgnoreConfigError=False beamGasHitsFile=NONE
beamHaloHitsFile=NONE samplingFractionDbTag=QGSP_BERT minbiasHitsFile=NONE cavernHitsFile=NONE jobConfig=NONE DBRelease=DBRelease-8.9.1.tar.gz doAllNoise=NONE 
triggerConfig=MC_InitialBeam_v1_no_prescale" 
--inDS=user10.MoritzBackes.pathena_photons_sim_geo-08-05-00 
--outDS=user10.MoritzBackes.pathena_photons_digi_geo-08-05-00 
--cloud=DE 
--split=200 
--dbRelease=ddo.000001.Atlas.Ideal.DBRelease.v080901:DBRelease-8.9.1.tar.gz
Single electrons
pathena --trf="csc_digi_trf.py inputHitsFile=%IN outputRDOFile=%OUT.digit.pool.root maxEvents=-1 skipEvents=0 digiSeedOffset1=$RANDOM digiSeedOffset2=$RANDOM 
geometryVersion=ATLAS-GEO-08-05-00 conditionsTag=OFLCOND-SIM-BS900-00 digiRndmSvc=AtRanluxGenSvc addCaloDigi=True IgnoreConfigError=False beamGasHitsFile=NONE
beamHaloHitsFile=NONE samplingFractionDbTag=QGSP_BERT minbiasHitsFile=NONE cavernHitsFile=NONE jobConfig=NONE DBRelease=DBRelease-8.9.1.tar.gz doAllNoise=NONE 
triggerConfig=MC_InitialBeam_v1_no_prescale" 
--inDS=user10.MoritzBackes.pathena_electrons_sim_geo-08-05-00 
--outDS=user10.MoritzBackes.pathena_electrons_digi_geo-08-05-00 
--cloud=DE 
--split=200 
--dbRelease=ddo.000001.Atlas.Ideal.DBRelease.v080901:DBRelease-8.9.1.tar.gz

Reconstruction step

Again a different release so do:

# setup release 
source cmthome/setup.sh -tag=AtlasProduction,15.6.3.6,32,opt,runtime
# setup pathena environment
export PATHENA_GRID_SETUP_SH=/afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
source ~/pandabin/etc/panda/panda_setup.sh
Again you need to copy the DBRelease tar file to your working directory. The pathena command for reconstruction is:

Single photons

pathena --trf="Reco_trf.py autoConfiguration=everything inputRDOFile=%IN outputESDFile=%OUT.esd.pool.root 
preInclude=RecJobTransforms/SetJetConstants-02-000.py,RecJobTransforms/NoTrackSlimming.py DBRelease=DBRelease-8.9.1.tar.gz 
preExec='rec.Commissioning.set_Value_and_Lock(True)',,'jobproperties.Beam.energy.set_Value_and_Lock(900*Units.GeV)',,'muonRecFlags.doCSCs=False',,
'muonRecFlags.useAlignmentCorrections=True',,'muonCombinedRecFlags.doMuidVertexAssociator=False',,'muonCnvFlags.MdtCablingMode=\`new\`',,'from AthenaCommon.BFieldFlags import jobproperties',,
'jobproperties.BField.solenoidOn.set_Value_and_Lock(True)',,'jobproperties.BField.endcapToroidOn.set_Value_and_Lock(True)',,'jobproperties.BField.barrelToroidOn.set_Value_and_Lock(True)' 
postExec='import MDTcabling.MDTcablingConfig',,'ServiceMgr.MDTcablingSvc.RODfile=\`AtlasRODmap.data\`',,'ServiceMgr.MDTcablingSvc.MEZfile=\`AtlasMEZmap.data\`',,
'ServiceMgr.MDTcablingSvc.ShiftLinks=False'
triggerConfig='MCRECO:InitialBeam_v1'" 
--inDS=user10.MoritzBackes.pathena_photons_digi_geo-08-05-00 
--outDS=user10.MoritzBackes.pathena_photons_reco_geo-08-05-00 
--cloud=DE 
--split=200 
--dbRelease=ddo.000001.Atlas.Ideal.DBRelease.v080901:DBRelease-8.9.1.tar.gz
Single electrons
pathena --trf="Reco_trf.py autoConfiguration=everything inputRDOFile=%IN outputESDFile=%OUT.esd.pool.root 
preInclude=RecJobTransforms/SetJetConstants-02-000.py,RecJobTransforms/NoTrackSlimming.py DBRelease=DBRelease-8.9.1.tar.gz 
preExec='rec.Commissioning.set_Value_and_Lock(True)',,'jobproperties.Beam.energy.set_Value_and_Lock(900*Units.GeV)',,'muonRecFlags.doCSCs=False',,
'muonRecFlags.useAlignmentCorrections=True',,'muonCombinedRecFlags.doMuidVertexAssociator=False',,'muonCnvFlags.MdtCablingMode=\`new\`',,'from AthenaCommon.BFieldFlags import jobproperties',,
'jobproperties.BField.solenoidOn.set_Value_and_Lock(True)',,'jobproperties.BField.endcapToroidOn.set_Value_and_Lock(True)',,'jobproperties.BField.barrelToroidOn.set_Value_and_Lock(True)' 
postExec='import MDTcabling.MDTcablingConfig',,'ServiceMgr.MDTcablingSvc.RODfile=\`AtlasRODmap.data\`',,'ServiceMgr.MDTcablingSvc.MEZfile=\`AtlasMEZmap.data\`',,
'ServiceMgr.MDTcablingSvc.ShiftLinks=False'
triggerConfig='MCRECO:InitialBeam_v1'" 
--inDS=user10.MoritzBackes.pathena_electrons_digi_geo-08-05-00 
--outDS=user10.MoritzBackes.pathena_electrons_reco_geo-08-05-00 
--cloud=DE 
--split=200 
--dbRelease=ddo.000001.Atlas.Ideal.DBRelease.v080901:DBRelease-8.9.1.tar.gz

D3PD making

The D3PDMaker page explains how to produce D3PDs locally. Working with release 15.6.5.3 one did not need to check out any code. One would just do this:

> source cmthome/setup.sh -tag=15.6.5.3,AtlasProduction
> athena.py EgammaD3PD.py &> test.log

When running with pathena, one needs to have an existing input file, which can be found locally, specified in the athenaCommonFlags.FilesInput in the job options. It was also necessary to send the geometry tar ball together with the job. The DBRelease-8.9.1.tar.gz (or a link to it, to be exact) was in the directory where pathena was run. This test job was a success:

pathena EgammaD3PD.py --nFiles 1 --nFilesPerJob 1 
--inDS user10.SzymonGadomski.pathena_photons_reco_geo-08-05-00.p 
--outDS user10.SzymonGadomski.pathena_photons_d3pd_test2 
--cloud=DE 
--dbRelease=ddo.000001.Atlas.Ideal.DBRelease.v080901:DBRelease-8.9.1.tar.gz

The production was done with 50 input files per job, using the following script:

> cat d3pd-sample.sh
pathena EgammaD3PD.py --nFilesPerJob 50 
--inDS user10.SzymonGadomski.pathena_${1}_reco_${2} 
--outDS user10.SzymonGadomski.pathena_${1}_d3pd_${2} 
--cloud=DE 
--dbRelease=ddo.000001.Atlas.Ideal.DBRelease.v080901:DBRelease-8.9.1.tar.gz

One could do:

> ./d3pd-sample.sh electrons geo-08-05-00.m
> ./d3pd-sample.sh photons geo-08-00-02.p

and so on for all input datasets.

-- SzymonGadomski - 29-Mar-2010

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2010-03-29 - SzymonGadomski
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback