HQTCrossFrameworkValidation < Sandbox

%TOC%
---+ News

---+ HQT Software Task Force

---++ Mandate and current activities
   * Facilitate communication among analysis regarding analysis software
   * Help with basic validation of software frameworks used in HQT
   * Perform basic cross-framework validation to ensure reproducibility of physics results from different frameworks

---++ Contacts
   * TF Members: %ICON{person}% Main.YuHengChen %ICON{person}% Main.TrishaFarooque %ICON{person}% Main.ElhamEKhoda
   * HQT Conveners: %ICON{person}% Main.ChristophAnders %ICON{person}% Main.KatharinaBehr

---+ Major analysis frameworks used in HQT

According to the early survey, frameworks used in individual analysis for Rel.20.7 and their corresponding derivations are listed in the following:

%TABLE{ dataalign="center" headeralign="center" valign="center" }%
|* Analysis            *| *Framework* |* Derivation(s)   *|* Processing time *|
| tt resonance (1-lep.) | AT-based    | EXOT4             | ~1 WK             |
| tt resonance (0-lep.) | AT-based    | EXOT7+EXOT4+TOPQ1 | ~1 WK             |
| tb resonance (0-lep.) | AT-based    | EXOT7             | -                 |
| VLQ OS-ML             | AT-based    | EXOT4             | ~2-4 WK           |
| VLQ all-had.          | AT-based    | EXOT7             | -                 |
| VLQ Zt+X high-MET     | ST-based    | SUSY5             | -                 |
| VLQ Ht+X              | ST-based    | TOPQ1+TOPQ4       | -                 |
| VLQ B->b/Zb           | CxAOD       | -                 | in Development    |

---+ Cross-Framework Comparison
---++ Setups
---+++!! Softwares
*Validation Tool*: https://gitlab.cern.ch/CrossFrameworkValidation/CrossFrameworkValidation

*Frameworks*
%TABLE{ dataalign="center" headeralign="center" valign="center" }%
|               |* Validated/Latest Version *|* Contact *|* Source Code *|* Build *|* Converge *|
|* [[https://twiki.cern.ch/twiki/bin/view/AtlasProtected/AnalysisTop21][AnalysisTop]] *| 21.2.19/21.2.37            | hn-atlas-top-reconstruction@cern.ch | [[https://gitlab.cern.ch/atlas/athena/tree/21.2/PhysicsAnalysis/TopPhys/xAOD][Link]] |  <a href="https://gitlab.cern.ch/CrossFrameworkValidation/RunAnalysisTop"><img src="https://gitlab.cern.ch/CrossFrameworkValidation/RunAnalysisTop/badges/master/build.svg?private_token=XDDyUC_Pqtke9cwrresJ"/></a> | <a href="https://gitlab.cern.ch/CrossFrameworkValidation/CrossFrameworkValidation/commits/master"><img alt="coverage report" src="https://gitlab.cern.ch/CrossFrameworkValidation/CrossFrameworkValidation/badges/master/coverage.svg" /></a> |
|* [[https://twiki.cern.ch/twiki/bin/view/AtlasProtected/SUSYWorkingGroup][SUSYTools]]   *| ^                          | atlas-phys-susy-backgroundforum@cern.ch | [[https://gitlab.cern.ch/atlas/athena/tree/21.2/PhysicsAnalysis/SUSYPhys/SUSYTools][Link]] | <a href="https://gitlab.cern.ch/CrossFrameworkValidation/RunSUSYTools"><img src="https://gitlab.cern.ch/CrossFrameworkValidation/RunSUSYTools/badges/master/build.svg?private_token=XDDyUC_Pqtke9cwrresJ"/></a> |^ |
*CP Tools* %BR%
General information about how to configure CP Tools can be found here: https://twiki.cern.ch/twiki/bin/view/AtlasProtected/PhysicsAnalysisWorkBookRel21CPRec

---+++!! Object Definitions
*Jets* %BR%
%GREEN%Container name: %ENDCOLOR% !AntiKt4EMTopo %BR%
%GREEN%Selection cuts: %ENDCOLOR% JetMC15
   * pT&gt;25GeV
   * abs(&eta;)&lt;2.5
   * B-tagging: MV2c10 at 70% WP

*Electrons* %BR%
%GREEN%Container name: %ENDCOLOR% !Electrons %BR%
%GREEN%Selection cuts: %ENDCOLOR% ElectronLikelihoodMC15
   * pT&gt;25GeV
   * abs(&eta;)&lt;2.47
   * Veto 1.37&lt;abs(&eta;)&lt;1.52
   * Loose (baseline): Identification: MediumLH, Isolation: no isolation
   * Tight (signal): Identification: TightLLH, Isolation: FixedCutTightTrackOnly

*Muons* %BR%
%GREEN%Container name: %ENDCOLOR% !Muons %BR%
%GREEN%Selection cuts: %ENDCOLOR% MuonMC15
   * pT&gt;25GeV
   * abs(&eta;)&lt;2.5
   * Loose (baseline): Identification: Medium, Isolation: no isolation
   * Tight (signal): Identification: Medium, Isolation: FixedCutTightTrackOnly

*MET* %BR%
%GREEN%Selection cuts: %ENDCOLOR%
   * Track Soft Term (TST), no fJVT
   * *All* Calibrated jets
   * electrons and muons passing loose selections
   * %RED% No photons and taus

*Large-R Jets* %BR%
%GREEN%Container name: %ENDCOLOR% !AntiKt10LCTopoTrimmedPtFrac5SmallR20Jets %BR%
%GREEN%Selection cuts: %ENDCOLOR%JetMC15
   * pT&gt;25GeV
   * abs(&eta;)&lt;2.0

*Overlap Removals* %BR%
%GREEN%Working Point: %ENDCOLOR%Standard
   1. remove electrons that shares track with muons
   2. remove single jet closest to an electron (within &#8710;R < 0.2)
   3. remove electrons/mouns within &#8710;R < 0.4 of any jet

---+++!! Input Sample
%TABLE{ dataalign="center" headeralign="center" valign="center" caption="TOPQ1 Powheg+Pythia8 ttbar MC sample" }%
|* SCOPE:NAME *|* GUID *|* ADLER32 *|* FILESIZE *|* EVENTS *|
| <a href="https://ami.in2p3.fr/AMI/servlet/net.hep.atlas.Database.Bookkeeping.AMI.Servlet.Command?Command=FormGetElementInfo+-GUI%3D%22yes%22+-amiAdvanced%3D%22ON%22+-amiLang%3D%22english%22+-elementID%3D%2256839770%22+-entityName%3D%22files%22+-processingStep%3D%22production%22+-project%3D%22mc16_001%22+-uimode%3D%22bootstrap%22&Converter=%2Fxsl%2FAMIProdFrameHTML.xsl&Valider=Execute&Type=AtlasProduction">mc16_13TeV:DAOD_TOPQ1.12453997._001723.pool.root.1</a> | F1CEF10F-5BB8-5045-B4E3-46503FED3B79 | ad:8aafca9e | 1.007 GB   |     6176 |

---++ Results 
---+++!! Common Tools Configurations
Utilizing the asg::ToolStore::dumpToolConfig() function, CP Tool Configurations as dependent variables of [[#Setups][Setups]] are studied.%BR%
Full List can be found here: https://docs.google.com/spreadsheets/d/14Lgpz19N_2ilQedNxoZ74kUp63hEJKLZLpqrEvab0-g/edit?usp=sharing%BR%
In Summary, most of the Tools Configurations can be easily configured and some hard-to-find discrepancies usually happens when CDI files of certain framework are not up-to-dated.

---+++!! Cutflows & Yields
*Cutflows*
<sticky><table><tr><td valign="top"></sticky>
%TABLE{ dataalign="right" headeralign="center" valign="center" caption="e+jets" }%
|* Selection         *|* Nevents (AT) *|* Nevents (ST) *|* NevST-NevAT *|
| INITIAL             | 6176           | 6176           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Ne(pT > 25 GeV) &#8805; 1 | 2144           | 2145           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Ne(pT > 30 GeV) &#8805; 1 | 1962           | 1963           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Ne(pT > 30 GeV) = 1 | 1903           | 1904           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| N&#956;(pT > 30 GeV) = 0 | 1776           | 1777           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Pass jet cleaning   | 1776           | 1777           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| ETmiss  > 20 GeV    | 1627           | 1627           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| ETmiss+mTW >60GeV   | 1532           | 1533           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Njet &#8805; 1            | 1528           | 1529           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Njet &#8805; 2            | 1448           | 1449           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Njet &#8805; 3            | 1193           | 1194           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Njet &#8805; 4            | 760            | 760            |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Nbjet &#8805; 1           | 649            | 649            |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| NlargeR-jet &#8805; 1         | 157            | 157            |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Nbad-jet &#956; = 0          | 157            | 157            |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
<sticky></td><td>&nbsp; &nbsp;</td><td valign="top"></sticky>
%TABLE{ dataalign="right" headeralign="center" valign="center" caption="&#956;+jets" }%
|* Selection         *|* Nevents (AT) *|* Nevents (ST) *|* NevST-NevAT *|
| INITIAL             | 6176           | 6176           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| N&#956;(pT > 25 GeV) &#8805; 1 | 2478           | 2478           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| N&#956;(pT > 30 GeV) &#8805; 1 | 2227           | 2227           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| N&#956;(pT > 30 GeV) = 1 | 2152           | 2152           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Ne(pT > 30 GeV) = 0                    | 2025           | 2025           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Pass jet cleaning   | 2025           | 2025           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| ETmiss  > 20 GeV    | 1842           | 1842           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| ETmiss+mTW >60GeV   | 1725           | 1726           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Njet &#8805; 1            | 1721           | 1722           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Njet &#8805; 2            | 1663           | 1664           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Njet &#8805; 3            | 1394           | 1394           |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Njet &#8805; 4            | 922            | 922            |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Nbjet &#8805; 1           | 794            | 794            |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| NlargeR-jet &#8805; 1         | 138            | 138            |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
| Nbad-jet &#956; = 0          | 138            | 138            |%CALC{$EVAL($T(R$ROW(0):C$COLUMN(-1))-$T(R$ROW(0):C$COLUMN(-2)))}%|
<sticky></td></tr>
<tr><td>
|*weights*|*Nevents (AT)*|*Nevents (ST)*|*NevST/NevAT*|
|unweighted |157|157|100.0000%|
|weight mc|115589.492188|115589.492188|100.0000%|
|weight lep|153.322937|153.322937|100.0000%|
|weight btag|154.620438|154.620438|100.0000%|
|weight pu|156.81871|156.81897|100.0002%|
|weight jvt|156.280106|152.133163|97.3465%|
|All weights||||
</td><td>
&nbsp; &nbsp;</td><td>
|*weights*|*Nevents (AT)*|*Nevents (ST)*|*NevST/NevAT*|
|unweighted |138|138|100.0000%|
|weight mc|98668.734375|98668.734375|100.0000%|
|weight lep|136.900772|136.900772|100.0000%|
|weight btag|137.132629|137.132629|100.0000%|
|weight pu|123.823524|123.823769|100.0002%|
|weight jvt|136.602066|133.905701|98.0261%|
|All weights||||
</td></tr>

</table></sticky>

---+++!! Event-by-Event Comparison
<a href="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_0.png">
<img src="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_0.png"  width="500"/>
</a>
<a href="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_1.png">
<img src="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_1.png"  width="500"/>
</a>%BR%
<a href="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_2.png">
<img src="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_2.png"  width="500"/>
</a>
<a href="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_3.png">
<img src="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_3.png"  width="500"/>
</a>%BR%
<a href="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_4.png">
<img src="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_4.png"  width="500"/>
</a>
<a href="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_5.png">
<img src="https://cern.ch/hqtsoftwaretf/CrossFrameworkValidation/Comparisons/Benchmark/Events/EventDisplay_5.png"  width="500"/>
</a>%BR%
---++ Discussions
---+++ JVT (In)Efficiency Scale Factors %ICON{todo}%
%ICON{tag-purple}% open issue %ICON{tag-purple}% not confirmed%BR%
A significant discrepancy in the JVT scale factors obtained from the AnalysisTop and SUSYTools based frameworks is observed [[#Cutflows & Yields][here]].
The JVT weight distributions of both is shown below:
%IMAGE{"JVTWeight.png"
  type="frame" 
  align="none" 
  size=320
  caption="JVT weight distributions of SUSYTools (in red) and AnalysisTop (in black)."
}%
It is later understood that this "long tail" above 1.0 from AnalysisTop is featured by the JVT inefficiency SF. 
The current understanding is that because _Efficiency scale factors are to be applied to hard scatter (HS) jets passing JVT, inefficiency scaling factors are to be applied to HS jets failing JVT._ 
However, while OverlapRemoval itself is an input to the JVT inefficiency SF calculation, OverlapRemoval has to be done after the JVT cut.
Before, this had caused an process issue in JVT SF calculation until the priority decorator feature was introduced.
To conclude, today, instead of being _thrown away_ directly, JVT-faling jets will be _ignored_ by the OR Tool.

Given this historical factor, the survey now goes in the direction of checking if this is also implemented accordingly in SUSYTools. 
More details can be found in the discussion thread with AnalysisTop developers 
[[https://groups.cern.ch/group/hn-atlas-top-reconstruction/Lists/Archive/Flat.aspx?RootFolder=%2fgroup%2fhn%2datlas%2dtop%2dreconstruction%2fLists%2fArchive%2fOverlap%20removal%20not%20checked%20for%20JVT%20efficiency%20calculation%20in%20AnalysisTop&FolderCTID=0x0120020089BEAE4E849F164F8F31AB03D17234C0][Link]] and their presentation of the statement of intent for that time [[https://indico.cern.ch/event/609560/contributions/2457623/attachments/1406413/2149079/TopReco2017_01_12.pdf][Link]].

-- Main.YuHengChen - 2018-07-27
Topic revision: r24 - 2018-09-14 - HQTSoftwareTaskforceService
Account
- Log In
- Cern Search
- TWiki Search
- Google Search
Sandbox All webs
Edit
Attach
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback