Difference: KatesCTIDEDevelopment (1 vs. 4)

Revision 42018-10-23 - KatherinePachal

Line: 1 to 1
 
META TOPICPARENT name="KatherinePachal"
Active notes for working on CTIDE.
Line: 99 to 99
 # Last step: load common job options include ("InDetRecExample/jobOptions.py")
Added:
>
>

Flags for controlling input

Example here for other items which can be read from a provided file:

https://gitlab.cern.ch/atlas/athena/blob/master/InnerDetector/InDetExample/InDetAlignExample/share/jobOption_ConditionsOverrider.py#L77

Example here of instructions Sourav was given once he had a local DB set up:

Reco_tf.py \
 --AMI q221 \
 --preInclude="CTIDENN_IDrecon_preInclude.py" \
 --inputRDOFile /eos/user/s/sosen/RDO_CTIDE_timing/user.rjansky.mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.RDO_20180110_EXT0/user.rjansky.12944101.EXT0._005040.RDO.root \
 --maxEvents 10 \
 --preExec='from PerfMonComps.PerfMonFlags import jobproperties as pmjp;pmjp.PerfMonFlags.doPostProcessing=True;pmjp.PerfMonFlags.doSemiDetailedMonitoringFullPrint=True' \
 --outputAODFile myAOD.root 2>&1 | tee _log_localtestWithPerfMonSD_q221.txt;

where contents of preInclude file are:

 
from IOVDbSvc.CondDB import conddb;
conddb.setGlobalTag(globalflags.ConditionsTag()); 
# Updated NN folder
conddb.blockFolder("/PIXEL/PixelClustering/PixelClusNNCalib")
conddb.addFolder("","<dbConnection>sqlite://;schema=newpixelNNdb.db;dbname=OFLP200</dbConnection> /PIXEL/PixelClustering/PixelClusNNCalib <tag>PixClusNNCalib-SuperS10122018</tag>", force=True);
..
 

Assess test results

Input files

Changed:
<
<
We want to automatically reproduce CTIDE validation plots, which means running on appropriate inputs. For CTIDE performance note, we had:
>
>
We want to automatically reproduce CTIDE validation plots, which means running on appropriate inputs. For CTIDE performance note, we had:

Revision 32018-10-23 - KatherinePachal

Line: 1 to 1
 
META TOPICPARENT name="KatherinePachal"
Active notes for working on CTIDE.
Line: 81 to 81
 # conddb.blockFolder("/PIXEL/PixelClustering/PixelCovCorr") # conddb.addFolder("","sqlite://;schema=newpixelErrRescale.db;dbname=OFLP200 /PIXEL/PixelClustering/PixelCovCorr PixelCovCorr-SIM-RUN12-000-01", force=True);
Added:
>
>

To run any ATHENA job

Let's say I want to test a specific job options found somewhere in the InDetRec package. How?

With my sparse checkout,

athena.py ../athena/InnerDetector/InDetExample/InDetRecExample/share/jobOptions.py

If I have a local jobOptions which calls the InDetRecExample job options, it can be set up quite simply and I can just run on that. Thus, it suffices to call

athena.py jobOptions_testCTIDEPerformance.py 

where jobOptions_testCTIDEPerformance.py just contains the lines:

# Last step: load common job options
include ("InDetRecExample/jobOptions.py")
 

Assess test results

\ No newline at end of file
Added:
>
>

Input files

We want to automatically reproduce CTIDE validation plots, which means running on appropriate inputs. For CTIDE performance note, we had:

Revision 22018-10-08 - KatherinePachal

Line: 1 to 1
 
META TOPICPARENT name="KatherinePachal"
Active notes for working on CTIDE.
Line: 30 to 29
  # Sparse checkout git atlas addpkg InDetExample
Changed:
<
<
git atlas addpkg InDetRecTools
>
>
git atlas addpkg SiClusterizationTool
  # Get ready to develop my code. Make sure I use an up to date branch! git fetch upstream

Revision 12018-10-08 - KatherinePachal

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="KatherinePachal"
Active notes for working on CTIDE.

Development sequence

  • Make sparse checkout of Athena packages that I want to change.
  • Make relevant changes. Ensure they compile
  • Make sure I can see my changes in local tests
  • Make MR back into Athena for 21.0 (?) What do I need to do this?

Ultimate goal:

  • Timing information which can be easily turned on or off from config
  • Easy development system for testing stuff

Athena development log

Sparse checkout InDetRecTools, InDetExample and compile. Make my own branch so I can merge my developments back later. My development location is: /afs/cern.ch/work/k/kpachal/CTIDE/AthenaDevelopment

Process based on instructions in ATLAS tutorial. As of Oct 8, best release is 21.0.82.

setupATLAS
lsetup git
# Currently using my own fork.
git atlas init-workdir ssh://git@gitlab.cern.ch:7999/kpachal/athena.git
cd athena

# Sparse checkout
git atlas addpkg  InDetExample
git atlas addpkg  InDetRecTools

# Get ready to develop my code. Make sure I use an up to date branch!
git fetch upstream
# Template:
# git checkout -b master-my-topic upstream/[parent_branch] --no-track
# My command:
git checkout -b 21.0-add-CTIDE-timing-diagnostics upstream/21.0 --no-track

# To check what releases are available:
asetup --listreleases
# Compile with correct release
mkdir ../build && cd ../build
asetup 21.0.[my_release],Athena

# If I used a sparse checkout I shouldn't need to muck about with the
# package_filters. It should just only compile the packages I checked out.
cmake ../athena/Projects/WorkDir
make -j

# Now test it.
mkdir ../run && cd ../run
source ../build/x86_64-slc6-gcc49-opt/setup.sh
Reco_tf.py --AMI q431

Perform tests

#!/usr/bin/python

import os

ref =   """Reco_tf.py --AMITag=r8618 --autoConfiguration=everything --conditionsTag "default:OFLCOND-MC16-SDR-10" --digiSteeringConf="StandardSignalOnlyTruth" --geometryVersion="default:ATLAS-R2-2016-00-01-00" --numberOfCavernBkg=0 --preInclude="preInclude.IDonly_reconstruction.py" --postExec "all:CfgMgr.MessageSvc().setError+=[\\"HepMcParticleLink\\"]" "HITtoRDO:streamRDO.ItemList+=[\\"SiHitCollection#*\\",]" "RAWtoESD:StreamESD.ItemList+=[\\"InDetSimDataCollection#*\\",\\"SiHitCollection#*\\"]" "ESDtoAOD:fixedAttrib=[s if \\"CONTAINER_SPLITLEVEL = '99'\\" not in s else \\"\\" for s in svcMgr.AthenaPoolCnvSvc.PoolAttributes];svcMgr.AthenaPoolCnvSvc.PoolAttributes=fixedAttrib" "ESDtoDPD:xAOD_PixelPrepDataToxAOD.UseTruthInfo=True" --postInclude "default:PyJobTransforms/UseFrontier.py" --preExec "all:rec.Commissioning.set_Value_and_Lock(True);from AthenaCommon.BeamFlags import jobproperties;jobproperties.Beam.numberOfCollisions.set_Value_and_Lock(20.0);from LArROD.LArRODFlags import larRODFlags;larRODFlags.NumberOfCollisions.set_Value_and_Lock(20);larRODFlags.nSamples.set_Value_and_Lock(4);larRODFlags.doOFCPileupOptimization.set_Value_and_Lock(True);larRODFlags.firstSample.set_Value_and_Lock(0);larRODFlags.useHighestGainAutoCorr.set_Value_and_Lock(True); from InDetRecExample.InDetJobProperties import InDetFlags; InDetFlags.pixelClusterSplitProb1.set_Value_and_Lock( 0.6 ); InDetFlags.pixelClusterSplitProb2.set_Value_and_Lock( 0.2 ); InDetFlags.doTIDE_RescalePixelCovariances.set_Value_and_Lock(False);InDetFlags.writeRDOs.set_Value_and_Lock(True)" "ESDtoAOD:TriggerFlags.AODEDMSet=\\"AODFULL\\"" "all:from egammaRec.egammaRecFlags import jobproperties;jobproperties.egammaRecFlags.doSuperclusters.set_Value_and_Lock(False)" "HITtoRDO:from Digitization.DigitizationFlags import digitizationFlags;digitizationFlags.doBichselSimulation.set_Value_and_Lock(True)" --runNumber=361027  --inputHITSFile=/afs/cern.ch/work/k/kastanas/public/mc16_valid.361027.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ7W.simul.HITS.e3668_s2995_tid09640850_00/HITS.09640850._001314.pool.root.1 --outputDAOD_IDTIDEFile=OUT.pool.root --maxEvents=2"""

os.system( ref)

where

cat preInclude.IDonly_reconstruction.py

from IOVDbSvc.CondDB import conddb;
conddb.setGlobalTag(globalflags.ConditionsTag()); 
# Updated NN folder
conddb.blockFolder("/PIXEL/PixelClustering/PixelClusNNCalib")
#conddb.addFolder("","<dbConnection>sqlite://;schema=newpixelNNdb_v2.db;dbname=OFLP200</dbConnection> /PIXEL/PixelClustering/PixelClusNNCalib <tag>PixClusNNCalib-00-00-42</tag>", force=True);
conddb.addFolder("","<dbConnection>COOLOFL_PIXEL/OFLP200</dbConnection> /PIXEL/PixelClustering/PixelClusNNCalib <tag>PixelClusNNCalib-SIM-RUN12-000-04</tag>", force=True);

# conddb.blockFolder("/PIXEL/PixelClustering/PixelCovCorr")
# conddb.addFolder("","<dbConnection>sqlite://;schema=newpixelErrRescale.db;dbname=OFLP200</dbConnection> /PIXEL/PixelClustering/PixelCovCorr <tag>PixelCovCorr-SIM-RUN12-000-01</tag>", force=True);

Assess test results

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback