(r34) TrigEventTPSeparation < Main

Main Web>TWikiUsers>AndrewHamilton>TriggerEDMCoordination>TrigEventTPSeparation (revision 34) (raw view)~~EditAttachPDF~~
This page documents Transient-Persistent (TP) separation in the trigger Event Data Model (EDM).  The page contains guidelines for developers needing to introduce new classes or evolve old classes.  Since much of the trigger EDM in in the !TrigEvent packages, this twiki uses !TrigEvent examples.     

---+++ What is !TP Separation?

Transient-Persistent (!TP) separation means that the classes stored on disk (ie. the persistent classes =xxx_p1= etc.) are different from the classes the user interacts with (the transient classes).  When someone asks to read an object from POOL, StoreGate uses the "persistent to transient" converter to read the instance of persistent class from disk and deliver an instance of the transient class to the user.  When someone asks to write an object to POOL, StoreGate uses the "transient to persistent" converter to create a persistent instance of the class and write it to disk.   The primary functionality of !TP separation is to enable backwards compatibility, so that data written with !ATHENA release N can be read with !ATHENA release N+1.

 A complete description of the !TP separation framework can be found on Marcin Nowak's !TP separation description: [[https://twiki.cern.ch/twiki/bin/view/Atlas/TransientPersistentSeparation][TransientPersistentSeparation]].

---+++ Introducing a New !TrigEvent Class

If you would like to persistify (ie. write to !ESD/AOD/DPD) a new class into the !TrigEvent !EDM (any class in !Trigger/TrigEvent) you will need to include suitable persistent classes and converters in the !Trigger/TrigEvent/TrigEventTPCnv and !Trigger/TrigEvent/TrigEventAthenaPool packages.  The !TrigEventAthenPool classes (!AthenaPoolConverters) act as the interface between !POOL and your persistent classes and converters - this is the code that decides which persistent class (ie =p1=, =p2=, =tlp1=, etc) should be read from, or written to, disk.  The persistent classes and their corresponding converters are found in the !TrigEventTPCnv package.

_Example:_  The =TrigEventAthenaPool/src/TrigElectronContainerCnv= class describes the !POOL interface to the !TrigElectronContainer converters.  The !TrigEventTPCnv package contains the =TrigElectron_p1= and =TrigElectronCnv_p1= classes which describe a persistent class and it's corresponding converter.  There are other =TrigElectron= related classes that are described below.

The following is a loose "step-by-step" set of instructions you can follow to create introduce your new !TrigEvent class.  It does not explain all details or use cases, but should be used as a starting point for understanding what to do:

   * Write the persistent version of your class in the !TrigEventTPCnv package.  The name will be !YourClassName_p1, the "_p1" reflecting that it is the first persistent version of the class.  The persistent class should contain private data members with the information you want to persistify, but no accessor methods (since the persistent class will never be accessed by the user).   If the persistent class contains a data member of a non-basic type, then you must decide to either "embed" or "link" (using an ElementLink) the contained class.  If you "embed" the object then you will be making a copy of the object inside your class (likely wasting disk space and reducing flexibility), while if you "link" the object to your class then you must be sure to persistify the object linked to so that the link points to!  Any base classes (=P4EtaPhiM= for example) must be embedded as a data member of the persistent class.  Note that there should be no pointer data members in the persistent class since this duplicates the information pointed to.
      * _Example_: !TrigElectron_p2 has links to the !TrigEMCluster and !TrigInDetTrack, and embeds the !P4EtaPhiM base class.  The p2 here represents the fact that it is actually the second persistent !TrigElectron version.

   * Write the converter for your class in the !TrigEventTPCnv package, the name should be !YourClassNameCnv_p1.  The converter will simply inherit from =T_AthenaPoolTPCnvBase= and contain a =persToTrans= and =transToPers= methods which do the conversion from the current transient version of the class to the p1 persistent version.
      * _Example_: !TrigElectronCnv_p2 shows many types of conversions.  Note that basic types, embedded classes, base classes, and !ElementLinks are all dealt with differently, see TransentPersistentSeparation for more details.

   * Write a persistent container and the corresponding container converter in the !TrigEventTPCnv package, !YourClassNameContainer_p1 and !YourClassNameContainerCnv_p1.  Normally, the container only requires a header file, no =cxx= file, because the implementation comes from the inheritance.  Note that every !TrigEvent class that can have more than one instance per event must be contained in a container (or collection).  This is due to the underlying structure of the trigger !EDM and navigation.  If your transient class does not have a container (or collection) you will have to add one.  
      * _Example_: See !TrigElectronCollection_p2 and !TrigElectronCollectionCnv_p2

   * Write the "Top Level" persistent class and converter for your class in the !TrigEventTPCnv package, !YourClassNameContainer_tlp1 and !YourClassNameContainerCnv_tlp1.  This "!TLP" class is used to eliminate the chain effect caused when contained classes evolve.  For example, if you embedded a !P4EtaPhiM_p1 class into your class, you would need to change this if !P4EtaPhiM was evolved to !P4EtaPhiM_p2.  This is obviously not maintainable in the long term.  Therefore, the !TLP allows you to specify classes by reference, called a !TPObjRef, and you do not need to change your class if a base class evolves under you.
      * _Example_: See !TrigElectronCollection_tlp2 and !TrigElectronCollectionCnv_tlp2

   * Include your class in the !TrigEventTPCnv/TrigEventTPCnvDict.h

   * Include your class in the !TrigEventTPCnv/selection.xml.  The =id= number needs to be unique for each class and all letters must be capital.  You get a unique id by doing =uuidgen | tr 'a-z' 'A-Z'= (the =tr= part of the command converts the id to all capitols).

   * Write the !AthenaPoolConverters in the !TrigEventAthenaPool package, !YourClassNameContainerCnv.  These converters are used to 'steer' the user request for an object to the correct persistent version to be read or written.  The =id= values from the !TrigEventTPCnv/selection.xml are used in this converter to identify the persistent version present on disk (using the =compareClassGuid= method)

---+++ Instructions to Schema Evolve (change the !EDM)


The actual converters and persistent representations (=_pN= objects) are located in the !TrigEventTPCnv package.  Every persistified object (as well as every data member and object in the inheritence structure of the persistified object) must have both a converter and a persistent class definition.  For example, =TrigElectronContainer= is a !DataVector of =TrigElectron= objects, =TrigElectron= objects inherit from =P4PtEtaPhiM= objects.  Therefore we will need the persistent classes: =TrigElectronContainer_p1=, =TrigElectron_p1=, and =P4PtEtaPhiM_p1= and their converters =TrigElectronContainerCnv_p1=, =TrigElectronCnv_p1=, and =P4PtEtaPhiMCnv_p1=.  

Because the structure of =TrigElectron_p1= depends on =P4PtEtaPhiM_p1=, if =P4PtEtaPhiM_p1= evolves to =P4PtEtaPhiM_p2=, then =TrigElectron_p1= must also be evolved to =TrigElectron_p2=.  This can create an undesirable chain effect in classes with large or complex data member and inheritence structure.  To get around this, we define "Top Level" converters and persistent classes, denoted by =_tlpN= (these do not replace the =pN= objects, they are in addition to the =pN= objects).  The top-level converter classes (denoted with =*Cnv_tlpN=) contain data members of *all* =_pN= converters used in the converter class, while the top-level persistent classes (denoted with =_tlpN=) contain =std::vector= data members of every =_pN= class used in the top-level class.  For example, this means that =TrigElectronContainer= has a =TrigElectronContainer_tlp1= class (containing =std::vectors= of =TrigElectronContainer_p1=, =TrigElectron_p1=, and =P4PtEtaPhiM_p1= types), and a =TrigElectronContainerCnv_tlp1= class (containing data members of type =TrigElectronContainerCnv_p1=, =TrigElectronCnv_p1=, and =P4PtEtaPhiMCnv_p1=.

So, as an example, here is what you need to do to schema evolve !TrigElectron (note that you are *adding* the =_p2= classes, you are not *replacing* the =_p1= classes):

   * you will need to modify two packages =Trigger/TrigEvent/TrigEventAthenaPool= and =Trigger/TrigEvent/TrigEventTPCnv=
   * in !TrigEventTPCnv:
      * create a new file =TrigEventTPCnv/TrigElectron_p2.h= containing the new !TrigElectron persistent class (obviously, use =TrigElectron_p1= as a template)
      * create the new converter in =TrigEventTPCnv/TrigElectronCnv_p2.h= and =src/TrigElectronCnv_p2.cxx= (use =_p1= as template)
      * electrons are in "containers", so you need to create =TrigEventTPCnv/TrigElectronContainer_p2.h=, =TrigEventTPCnv/TrigElectronContainerCnv_p2.h= (inheritance takes care of the content of the converter, so in this case =TrigElectronContainerCnv_p2.cxx= is not needed) 
      * create the new "top level object" =TrigEventTPCnv/TrigElectronContainerCnv_tlp2.h= and =src/TrigElectronContainerCnv_tlp2.cxx=
      * add entries into =TrigEventTPCnv/selection.xml= for the new classes (to get the a =id= do =uuidgen | tr 'a-z' 'A-Z'=), follow examples already in =selection.xml=
      * add headers and dummy vectors into =TrigEventTPCnv/TrigEventTPCnvDict.h=

   * in !TrigEventAthenaPool:
      * note that all header and source files are in =TrigEventAthenaPool/src=
      * edit =TrigElectronContainerCnv.cxx= to contain another =if= statement for your new =guid= (the =id= from the =TrigEventTPCnv/selection.xml= file) 
      * edit the =TrigElectronContainerCnv.h= to typedef the new =_tlp2= object, and create a new =m_TPConverter= data member (probably with a name something like =m_TPConverter_tlp2=)

   * now you should have a functioning converter which produces an =_tlp2= object when you write an !AOD and reads either a =_tlp1= or =_tlp2= object when you read an !AOD depending on which object exists in the !AOD.

To create a converter for a new class you will need to add the class to =TrigEventAthenaPool/cmt/requirements=.  For example: 
<verbatim>-s=${TrigInDetEvent_root}/TrigInDetEvent TrigInDetTrackCollection.h TrigVertexCollection.h TrigTauTracksInfo.h \ </verbatim>

If you have trouble, please email me, I will be glad to help, Main.AndrewHamilton

*Below this point are notes for myself, developers trying to evolve a !TrigEvent class need to read no further!*

---++++ Memory Leaks in 13.0.40

Large memory leaks seen in black hole events due to a conditional ownership problem.  Leaks found in:

!TrigEFBphysCnv_p1
In transient class !TrigEFBphys.h, the pointer =m_secondaryDecay= is not deleted - conditional ownership.

!TrigL2BphysCnv_p1
In transient class !TrigL2Bphys.h, the pointers =m_pVertex= and =m_secondaryDecay= are not deleted - conditional ownership.
 
!TrigPhotonCnv_p1
In transient class !TrigPhoton.h, the pointer to =m_cluster= is not deleted - conditional ownership problem...  

!TrigVertexCnv_p1
In transient class !TrigVertex.h, the pointer to =m_tracks= is deleted, but the tracks pointed to in the list are not deleted - conditional ownership.

The conditional ownership problem arises because during the initial creation of a transient class, say !TrigPhoton, the object does not own the object pointed to, !TrigEMCluster.  But, when the transient objects are created with the T/P converters, the TrigPhoton does own the TrigEMCluster, thus should delete it.

Scott Snyder's fix for the conditional ownership (in !TrigPhoton example):
   * add a =bool m_ownsCluster= to transient class !TrigPhoton
   * initialize =m_ownsCluster= to =false= in constructors of !TrigPhoton (need to be careful in copy constructor)
   * in the destructor of transient class, =if(m_ownsCluster) delete m_cluster=
   * in the =persToTrans= of the converter, set =transObj->m_ownsCluster= to =true= 
   * therefore, !TrigPhotons created by the T/P converter will have =m_ownsCluster= = =true=, while !TrigPhotons created elsewhere will have =m_ownsCluster= = =false=

---++++ All Except !TrigInDetTruthMap

The work started from =TrigEventAthenaPool-00-01-08= and =TrigEventTPCnv-00-00-00=, so to remove all TP separation of !TrigEvent one can use those two tags.  

Testing:
   * produce and AOD from RDO
   * dump variables of !TrigEvent classes to log file
      * using the !AnalysisSkeleton 'framework' and the modified following files: [[%ATTACHURL%/dumpAOD.tgz][dumpAOD.tgz]]
      * also add the following to the requirments file:
         * =use !TrigEventAthenaPool       !TrigEventAthenaPool-*           Trigger/!TrigEvent=
         * =use !TrigEventTPCnv            !TrigEventTPCnv-*                Trigger/!TrigEvent=
         * =use !TrigSteeringEvent         !TrigSteeringEvent-*             Trigger/!TrigEvent=
         * =use !TrigCaloEvent             !TrigCaloEvent-*                 Trigger/!TrigEvent=

   * check variables are consistent compared to non-TP split AOD
      * the following variables are not initialized, so differences are expected in TP to non-TP split !AODs:
         * =TrigL2Bphys->isValid=
         * =TrigMissingET->RoIword=
         * =pVertex->energyFraction=
         * =pVertex->dist= 
         * =TrigL2Bphys->dist= 
      * use =egrep -v "persToTrans|create|tlp1|AthenaSealSvc|AthenaPool|TrigMissingET->RoIword|pVertex->energyFraction|pVertex->dist|TrigL2Bphys->dist|pSecondDecay->dist" dumpAll.noTPSep.log > dumpAll.noTPSep.trim= to remove these benign log file differences
      * rounding differences due to =double -> float= conversion are expected in:
         * =TrigInDetTrackFitPar->ea0=
         * =TrigInDetTrackFitPar->ez0=
         * =TrigInDetTrackFitPar->ephi0=
         * =TrigInDetTrackFitPar->eeta0=
         * =TrigInDetTrackFitPar->epT=
         * =pVertex->x=
         * =pVertex->y=
         * =pVertex->z=
         * =pVertex->chi2=
         * =pVertex->massVariance=


Notes to myself: 
   * dump files in: =unidisk/trigger/noTPSep/PhysicsAnalysis/AnalysisCommon/UserAnalysis/run=
   
<!---
---++++ Worst Case Scenerio Tags (=TrigEventAthenaPool-00-01-14= and  =TrigEventTPCnv-00-00-04=)

In =TrigEventAthenaPool/src/=, removed:
<verbatim>
TrigElectronContainerCnv.cxx
TrigElectronContainerCnv.h
TrigEMClusterCnv.cxx
TrigEMClusterCnv.h
TrigInDetTrackCollectionCnv.cxx
TrigInDetTrackCollectionCnv.h
TrigRoiDescriptorCnv.cxx
TrigRoiDescriptorCnv.h
TrigTauClusterCnv.cxx
TrigTauClusterCnv.h
TrigTauCnv.cxx
TrigTauCnv.h
</verbatim>

In =TrigEventTPCnv/TrigEventTPCnv/=, removed:
<verbatim>
TrigCaloClusterCnv_p1.h
TrigCaloCluster_p1.h
TrigElectronCnv_p1.h
TrigElectronContainerCnv_p1.h
TrigElectronContainerCnv_tlp1.h
TrigElectronContainer_p1.h
TrigElectronContainer_tlp1.h
TrigElectron_p1.h
TrigEMClusterCnv_p1.h
TrigEMClusterCnv_tlp1.h
TrigEMCluster_p1.h
TrigEMCluster_tlp1.h
TrigInDetTrackCnv_p1.h
TrigInDetTrackCollectionCnv_p1.h
TrigInDetTrackCollectionCnv_tlp1.h
TrigInDetTrackCollection_p1.h
TrigInDetTrackCollection_tlp1.h
TrigInDetTrackFitParCnv_p1.h
TrigInDetTrackFitPar_p1.h
TrigInDetTrack_p1.h
TrigRoiDescriptorCnv_p1.h
TrigRoiDescriptor_p1.h
TrigTauClusterCnv_p1.h
TrigTauClusterCnv_tlp1.h
TrigTauCluster_p1.h
TrigTauCluster_tlp1.h
TrigTauCnv_p1.h
TrigTauCnv_tlp1.h
TrigTau_p1.h
TrigTau_tlp1.h
</verbatim>

In =TrigEventTPCnv/src/=, removed:
<verbatim>
TrigCaloClusterCnv_p1.cxx
TrigElectronCnv_p1.cxx
TrigElectronContainerCnv_tlp1.cxx
TrigEMClusterCnv_p1.cxx
TrigEMClusterCnv_tlp1.cxx
TrigInDetTrackCnv_p1.cxx
TrigInDetTrackCollectionCnv_p1.cxx
TrigInDetTrackCollectionCnv_tlp1.cxx
TrigInDetTrackFitParCnv_p1.cxx
TrigRoiDescriptorCnv_p1.h
TrigRoiDescriptor_p1.h
TrigTauClusterCnv_p1.cxx
TrigTauClusterCnv_tlp1.cxx
TrigTauCnv_p1.cxx
TrigTauCnv_tlp1.cxx
</verbatim>

Edit =TrigEventTPCnv/TrigEventTPCnvDict.h= and =TrigEventTPCnv/selection.xml= to remove all references to above classes.


-- Main.AndrewHamilton - 03 Sep 2007
--->
<!--
---++++ !TrigInDetTrackCollection Version 2

List of files in =~ahamil/trigger/testTPSep/=:

   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackCollection_tlp1.h= (no .cxx file)
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackCollection_p1.h= (no .cxx file) - is =m_RoI_ID= used correctly?
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrack_p1.h= (no .cxx file)
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackFitPar_p1.h= (no .cxx file)

   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackCollectionCnv_tlp1.h= and =TrigEventTPCnv/src/TrigInDetTrackCollectionCnv_tlp1.cxx=
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackCollectionCnv_p1.h= (no .cxx file)
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackCnv_p1.h= and =TrigEventTPCnv/src/TrigInDetTrackCnv_p1.cxx=
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackFitParCnv_p1.h= and =TrigEventTPCnv/src/TrigInDetTrackFitParCnv_p1.cxx=

   * =TrigEventAthenaPool/src/TrigInDetTrackCollectionCnv.h= and =TrigEventAthenaPool/src/TrigInDetTrackCollectionCnv.cxx=
 
   * =TrigEventTPCnv/TrigEventTPCnv/TrigEventTPCnvDict.h= (add all =TrigInDetTrackCollection= related classes)
   * =TrigEventTPCnv/TrigEventTPCnv/selection.xml= 

---++++ T/P Separation Version 2 (using !TrigTau) 

List of files created:

   * =TrigEventTPCnv/TrigEventTPCnv/TrigTau_tlp1.h= (no .cxx file) 
   * =TrigEventTPCnv/TrigEventTPCnv/TrigTau_p1.h= (no .cxx file)
   * =TrigEventTPCnv/TrigEventTPCnv/TrigTauCnv_tlp1.h= and =TrigEventTPCnv/TrigEventTPCnv/TrigTauCnv_tlp1.cxx=

   * =TrigEventTPCnv/TrigEventTPCnv/TrigTauCluster_p1.h= (no .cxx file)
   * =TrigEventTPCnv/TrigEventTPCnv/TrigTauClusterCnv_p1.h= and =TrigEventTPCnv/src/TrigTauClusterCnv_p1.cxx=
   * (there is only one cluster, not a collection of clusters)

   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrack_p1.h= (no .cxx file)
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackCnv_p1.h= and =TrigEventTPCnv/src/TrigInDetTrackCnv_p1.cxx= 
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackCollection_p1.h= (no .cxx file)
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackCollectionCnv_p1.h= (no .cxx file)

   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackFitPar_p1.h= (no .cxx file)
   * =TrigEventTPCnv/TrigEventTPCnv/TrigInDetTrackFitParCnv_p1.h= and =TrigEventTPCnv/src/TrigInDetTrackFitPar_p1.cxx=
 

   * =TrigEventAthenaPool/TrigEventAthenaPool/TrigTauCnv.h= and =TrigEventAthenaPool/src/TrigTauCnv.cxx=

   * =TrigEventTPCnv/TrigEventTPCnv/selection.xml=
   * =TrigEventTPCnv/TrigEventTPCnv/TrigEventTPCnvDict.h=

 
To get rid of !TrigTauContainer:
   * remove =TrigParticle/TrigTauContainer.h=
   * comment out =#include !TrigTauContainer.h= from =TrigParticle/TrigParticleDict.h=
   * the following warning is repeated (seamingly infinite) when I try to compile:
      * =TrigEventAthenaPoolPoolCnvGen.make Rebuilding ../i686-slc4-gcc34-opt/TrigEventAthenaPoolPoolCnvGen_dependencies.make WARNING >> You should provide a target for /afs/cern.ch/user/a/ahamil/trigger/tauTPSep/Trigger/TrigEvent/TrigParticle/TrigParticle/TrigTauContainer.h=

-- Main.AndrewHamilton - 31 Jul 2007

---++++ !T/P Separation version 1

Start from the T/P Separation wiki: [[https://twiki.cern.ch/twiki/bin/view/Atlas/TransientPersistentSeparation][TransientPersistentSeparation]] %BR%
Use !SiHit as an example: =/afs/cern.ch/atlas/software/builds/AtlasEvent/13.0.10/InnerDetector/InDetEventCnv/InDetEventAthenaPool= %BR%

Setting up test area:
   * =/afs/cern.ch/user/a/ahamil/trigger/indetTPSep=
   * =source ./setup.sh -tag=13.0.10,oneTest,setup,setupCMT,opt,32=
   * =cmt co -r !TrigEventAthenaPool-00-01-05 Trigger/TrigEvent/TrigEventAthenaPool=
   * create/edit the following files:
      * =TrigEventAthenaPool/TrigDecisionAthenaPoolCnvDict.h=
      * =TrigEventAthenaPool/selection.xml=
      * =TrigEventAthenaPool/TrigInDetTrack_p1.h=
      * =TrigEventAthenaPool/TrigInDetTrackCollection_p1.h=
      * =src/TrigInDetTrackCnv_p1.h= and =.cxx=
      * =src/TrigInDetTrackCollectionCnv.h= and =.cxx=
      * =src/TrigInDetTrackCollectionCnv_p1.h=

Testing:
   * =get_files -jo  !testElectronSliceAthenaRDO.py=
   * edit =!testElectronSliceAthenaRDO.py=
      * uncomment the =PoolRDOInput blah, blah= line
      * make =doWriteAOD=True= (maybe also =doAOD=True=)
   * =athena -s testElectronSliceAthenaRDO.py >& test.log &=
   * should produce an aod file, use root !TBrowser to look for your _p1 objects in the !CollectionTree

The code compiles, but does not produce a !TrigInDetTrackCollection_p1 object..

-- Main.AndrewHamilton - 05 Jul 2007

-->