Comments for AN-21-061

This twiki discusses the comments for the analysis note AN-21-061 of the H^\pm + h -> l + 4b analysis with CADI line HIG-22-005.

link: https://cms.cern.ch/iCMS/analysisadmin/cadilines?line=HIG-22-005

Overview of note versions

AN v8

link: https://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2021_061_v8.pdf

general info: Futher studies for pre-approval

changes:

  • Added plot of limits calculated with toys in Appendix I
  • Added a sanity check for the usage of the DeepAK8 in Appendix H
  • Added more description in Section 3 for the DeepAK8 usage
  • Changed DeepAK8 uncertainty values in Table 10
AN v7

link: https://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2021_061_v7.pdf

general info: Implementation of comments from pre-approval

changes:

  • Added study of different signal region binning in Appendix G
  • Added a 2D plot of expected cross section limits (Figure 21)
  • Updated several plots in Section 9 (Figure 16, 17-19, 20) and Appendix B, C, D (Systematic uncertainty band added)
AN v6

link: https://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2021_061_v6.pdf

general info: Added study of extrapolation uncertainty

changes:

  • Structured Section 9 with subsections
  • Added Section 9.3 and Appendix E with extropolation uncertainty study
AN v5

link: https://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2021_061_v5.pdf

general info: Implementation of the comments provided by Jan into AN v4

changes:

  • Added DeepAK8 uncertainty and PS scale variation in Table 10
  • Corrected scale uncertainty values in Table 10 (put wrong small numbers there in v4)
  • Added explanation of calculation of PDF uncertainties
  • Added impact plots for low and high mass point (Figure 17-19)
  • Updated all figures in section 9 with recalculated limits
AN v4

link: https://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2021_061_v4.pdf

general info: Implementation of the comments provided by Jan into AN v3

changes:

  • Added ttH MC and nJet-binned W+j samples, compare Table 1
  • Added subsection (2.3) explaining stitching of W+j samples
  • Added clarification about b-tag input used in the DNN
  • Reworked section 8, added more details and an overview table (Table 10)
  • Updated all figures in section 9 with recalculated limits
  • Added several limit comparison plots in Appendix E, starting from page 164)
AN v3

link: https://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2021_061_v3.pdf

general info: Implementation of the comments provided by Agni and Rainer into AN v2

changes:

  • Correction of known typos in text and figure
  • Added a section about the normalisation of the MC samples (section 2.2)
  • Added discussion about the systematic uncertainty on the non-prompt lepton background (line 434-439)
  • Added table with thresholds determined for the definition of the signal and validation bins for on example mass point and analysis channel (table 10)
  • Swap of figure 14 and 15
  • Added discussion about the treatment of systematic uncertainties as nuisances parameter in the fit (line 479-489)
  • Add pull and impact plot for an asimov dataset for the example mass point (figure 17)
AN v2

link: https://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2021_061_v2.pdf

general info: First complete iteration for the documentation of the analysis

AN v1

link: https://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2021_061_v1.pdf

general info: Empty dummy file for first upload

Comments from pre-approval

Q: Please check how much the analysis sensitivity would be improved if you were to split the signal region into 3 or 4 bins, compared with the 2 that you have now

A: Since AN v7 in Appendix G a study of different binnings with 4 and 7 equidistant bins has been performed. The conclusion of the study is that these alternative binnings are generally perform worse and therefore the binning approach with two optimised signal bins as shown in the preapproval is favoured.

Q: No scale factors are applied for the DeepAK8 classifier, please double check with the experts whether the uncertainties that are applied to cover for the missing scale factors are indeed sufficient

A: In a private chat with Loukas Gouskos the issue was discussed. A short summary of the discussion:

  • Resolved some misunderstandings in the usage of the DeepAK8 in the analysis. While the DeepAK8 is shown in 5 classes (see e.g. Figure A.3 in AN v7) for illustration purpose, the DeepAK8 is effectively used in a binary way. In the analysis, each AK8 jet is selected as

    argmax(sum(probZ*), sum(probH*), sum(probT*), sum(probW*), sum(probQCD*)) = 0 (Z like) or 1 (Higgs like)

    All other jets are not selected and not used in any control region study. This is clearified in AN v8.

  • For the background MC Loukas proposed following approach: A conservative 20% uncertainty should be applied. In addition a sanity check should be performed. For this sanity check a di-boson selection is performed. The idea is to selected ZZ events, with one clear Z->ll and another Z->qq clustered as AK8 jet. The AK8 jet has to pass the same requirement as in the analysis. In case no big deviation is seen, one may assume 20% is okay to cover everything.

    Such check is shown in AN v8 in Appendix H.

  • The signal MC Loukas proposed to a test a another genator, e.g. sherpa. Compare the signal with different generators and extract from the difference an uncertainty.

    This study is ongoing, but can be finalised also after the greenlight for preapproval, as no major change in sensitivity is expected.

Comments to v4 [from Jan, 23.05.2022]

Q: Are there any uncertainties in the DeepAK8 tagging efficiencies? (there should be)

A: In fact there are uncertainties for the DeepAK8 efficiencies, see https://twiki.cern.ch/twiki/bin/view/CMS/DeepAK8Tagging2018WPsSFs. The problem is, I dont use the binary score TvsQCD and the WP as described in the twiki. As described in Section 3 and shown in the figures A.7 - A.10, from all raw probabilities I calculate the predicted class and select basically the X->bb class. As a compromise, I have now added as systematic uncertainty on the DeepAK8 by calculating the mean uncertainty over all WPs for the given pt slice from the figures under section "Nominal Top quark tagging".


Q: How are the PDF uncertainties evaluated?

A: For all 100 PDF variation, 100 histograms additional histograms are filled. From this 100 histograms the envelope is calculated and used as uncertainty with the formula unc = max(abs(h_i->GetBinContent(x) - h_nom >GetBinContent(x)) for i in [0, 99]


Q: Signal cross section uncertainty: 1.8% looks pretty small - how is it evaluated?

A: The uncertainty is taken from XSDB, see for example https://cms-gen-dev.cern.ch/xsdb/?searchQuery=DAS=HPlusAndH_ToWHH_ToL4B_200_100_TuneCP5_PSWeights_13TeV-madgraph-pythia8


Q: For ttbar, besides considering mu_R and mu_F, do you also consider the parton shower weights (uncertainties)?

A: It will be added to the next version.


Q: What does the leading uncertainty (JME_2018) stand for?

A: Sorry for the ambiguous naming convention. This uncertainty is related to the jet energy resolution arising from the smearing procedure in MC.


Q: The "Scale" parameter: Which processes are affected by this parameter? The mu_R/mu_F variations should lead to independent uncertainties per process.

A: Right now for all processes the muR/muF is calculated, but is given fully correlated over all processes to the fit. It will be split into uncertainties for each process.


Q: The "PDF" parameter: Depending on how it is evaluated, this should also be decorrelated between the different processes

A: Same as for the muR/muF, it is calculated for all processes, but is given fully correlated over all processes to the fit. It will be split into uncertainties for each process.


Q: Can you also add an impact plot for a high-mass signal.

A: It will be added, in additional I will also add the low-mass signal.


Q: What motivates the range of neutral scalar masses tested (70-110 GeV)?

A: The range is driven by the heavy Higgs mass m_H, which by construction is m_H > m_h and is fixed as 125 GeV in the inverted model. To suppress the decay H->hh, which opens another interesting (but out of scope) phase space, the lower bound is chosen to be 70 GeV. The natural upperbound is 125 GeV, and because 5 mass points for the charged Higgs mass are chosen, in the same manner 5 mass points for the neutral mass have been chosen starting from the lower bound going in 10 GeV steps.

Comments to v3 [from Jan, 16.05.2022]

Q: Choice of h->bb jet tagger: Do you use DeepAK8 V1 or V2? Have you compared with ParticleNet?

A: From the documentation of the DeepAK8 (https://twiki.cern.ch/twiki/bin/viewauth/CMS/DeepAKXTagging), V2 is used as standard since NanoAOD v5. I use NanoAOD v9, so I would assume the version of DeepAK8 I use also should be V2. I have not looked into ParticleNet. In fact, I was not aware of the existence of ParticleNet in the latest NanoAOD v9 (it is not listed in the documentation).


Q: Dilepton rejection with IsoTracks: Is this method applied in other analyses that you took as example, or is it something specifically done for this analysis?

A: This method is taken from SUS-21-007, one of the main contributor of this analysis are in my institute working group and recommended this method.


Q: How has the DeepJet b-tag working point been optimised?

A: The working point is chosen to be loose to keep as much signal events as possible. The optimisation is performed by the DNN, the b-tag is given as an input, compare Table 9. The b-tag is encoded as a number (0 = not-tagged, 1 = loose, 2 = medium, 3 = tight) and given the DNN. This was not mentioned in the AN, it will be added to the next version.


Q: What the the 2HDM parameters used to define the theoretical cross sections and branching fractions? How different are they for 4 types of 2HDMs? Could we tune the 2HDM parameters to get larger cross sections?

A: A comprehensive study of bosonic channel of the charged Higgs was performed by our theory colleagues at DESY (https://arxiv.org/abs/2103.07484), on which our analysis is based on. The main parameter defining the model is fixing the heavy CP-even Higgs H to the standard model Higgs, given us the named "inverted" scenario, with the light CP-even Higgs h as non-SM Higgs. Typically the other way around is chosen, and also possible. In the inverted scenario, which is considered, the optimal model is already chosen. The highest branching ratio is found in the Type I model and the alignment limit, with cos(alpha - beta) = 1, in which the branching ratio W to HPlus + h is maximal. Tan(beta) has no great impact, compare Fig. 1, so it is set a value which is not excluded in combination of the set of other parameter chosen.


Q: What is the relative importance of the three overall categories (non-boosted, semi-boosted, fully boosted)? Can you add plots of comparisons of expected limits for these three combined categories? As a cross check, it will also be interesting to see further such comparisons, e.g. the limits by year, and the limits by lepton final state

A: In Appendix E, the limits by year and by lepton channel + jet boost are shown, but not compared to the overall limit. The explicit comparison plots will be added to the next version.


Q: Fig. 15: The W+jets contribution seems to have very large statistical uncertainty. Looking at section 2, do you really only use the inclusive W+jets sample? Given the high-multiplicity final state, the N-jet-binned samples should probably be used in addition. (From Fig. 16, one could also conclude that the W+jets background is negligible, but it's hard to tell - maybe there's no need to process the jet-binned sample if the contribution of the statistical uncertainty in the W+jets yields is negligible compared to other statistical uncertainties)

A: It will be added.


Q: Given the overall final state, I would have expected ttbb background to play an important role. Have you considered checking the dedicated MC samples? The ttH(bb) team should be able to help you.

A: TT + W and TT + Z is included in the analysis already, in the plots shown as orange histogram. Only the ttH MC was not considered, but will be added in the next version.


Q: Do you apply any theory uncertainties in the ttbar background that could give different numbers in the different final analysis bins? For example scale variations, PS ISR/FSR, UE, top pT reweighting

A: PDF and factorization/renormalization scale have been added as systematics in the fit and in the documentation.


Q-1: There seem to be overall jet uncertainties (which are tightly constrained) and some components. Would recommend to use the jet uncertainties split into the different sources and make sure there's no double-counting. (Figure 17)

A-1: It will be checked.

Q-2: What is the motivation for having separate uncertainties for ttbar single-lepton and dilepton? (but not have the theory shape uncertainties mentioned above) (Figure 17)

A-2: The splitting seems redundant after receiving this comment, it will be removed.

Q-3: It would be good to expand the systematic uncertainty section 8 such that all uncertainties appear with some motivation and numerical values, in particular also all background normalisation uncertainties. (Figure 17)

A-3: It will be added to the next version.

Comments to v2 [from Rainer, 02.05.2022]

Q: appendix C shows that the MC overshoots the data systematically in various plots. How is this handled? In Fig. 16 this is not visible any more in postfit, but the plot is not very differential

A: There is no special treatment for this overshoot. If you check the plot D.32 in Appendix D (page 137), which are the DNN scores for the semi-boosted muon final state for 2018 in which the overshoot occurs, in the DNN score the overshoot is more or less a constant offset. This is translated into the 1Lep and 2Lep bin of plot D.31 right above D.32, which are exactly the bins used in Fig. 16 at the "semi-boost mu 2018" block. The fit handles the overshoot then by itself.


Q: it is not obvious which scale factor scheme is used for the MC-based backgrounds. I think you have nuisance parameters for their cross sections but I do not see in mentioned in the AN. Maybe I missed it

A: A more detailed discussion will be added in the next version.


Q: in the same context, it would be good if you could include pulls & impacts (initially still blinded) in the AN. You showed such plots in the talks

A: These plots will be included in the next version.

Comments to v2 [from Agni, 26.04.2022]

Q: typo, I think this paragraph is about ak4 (line 162)

A: Yes, it is a typo, it is about AK4


Q: No combined triggers for the muons? (table 5)

A: The trigger selection is based on the recommended trigger stated in the Muon POG twiki (https://twiki.cern.ch/twiki/bin/viewauth/CMS/MuonHLTv2), in which only non combined single muon trigger are listed. For electrons triggers, the EGamma POG also listed combined trigger in their list of recommended trigger (https://twiki.cern.ch/twiki/bin/view/CMS/EgHLTRunIISummary).


Q: What was the reasoning behind the merging of the categories? Did you test the performance if they are separated? You mention that it is justified by the efficiency of the selection. Do you have any relevant studies? (lines 323-330)

A: The merging of the background into the shown categories was a process of trial and error during the training of the DNN. In the first iterations, a higher number of background classes was used, so additional classes for W+j and single top have been tried. These background classes either performed not well or the training was not stable. Therefore successive merging of the background was performed until the final background classes arised, which had a stable training and accuracy for all year and analysis channels. The intermediate iterations with more background classes are not saved and available in a state which is usable for proper documentation.

The argumentation with the selection efficiency is of qualitative nature. The main idea is to add backgrounds with initially one prompt lepton like W+j and semi-leptonic ttbar and backgrounds with initially two prompt leptons like di-leptonic ttbar and DY and motivate with this the selection efficiency, without a rigorous quantification. Therefore no well documented studies have been performed.


Q: for my understanding x here is the set described in 7.2? (line 337)

A: Exactly. A clarification in the text will be added.


Q: only the energy-related systematics are propagated through the DNN? why is that? (line 394-5)

A: Propagated means in this context reevaluated. Energy related uncertainties changes the numerical values of the momentum of the jets, lepton or of the missing transverse momentum, therefore the DNN score is reevaluated with the down and up shifted momentum values. For other uncertainties from e.g. scale factors, the numerical values of the input variables like the momentum will not change, therefore the DNN is not reevaluated. These uncertainties are treated as usual, the sum of weights is recalculated foe each uncertainty and shift for the histogram filled the the nominal DNN score.


Q: why are both mH and mh divided by mHmax? (Figure 11)

A: This is a typo, the vector for the CP-even Higgs masses h is of course divided by max(m_h) = 110 as stated in line 337.


Q: seems like you have no uncertainty related to your background estimation method. did I miss it? (Section 8)

A: Indeed the uncertainty was not included into the documentation by mistake, it will be added in the next version.


Q: 2 bkg and 2 signal, I count 4 bins. line 423 says 5. I think i have misunderstood something here, can you make it more clear? (lines 425-428)

A: There are two background bins, the 1Lep bin and the 2Lep from the respective DNN background class. In the DNN signal class, three bins are defined, one validation bin (VR) and two signal bins (SRL and SRH). The signal bin definition is indicated in Figure 14 and discussed from line 429 to 442. To make it more clear, a table will be added in addition to Figure 14, which will show the calculated thresholds for the shown example mass point and analysis channel used to the define VR, SRL and SRH.


-- DavidBrunner - 2022-04-27

Edit | Attach | Watch | Print version | History: r24 < r23 < r22 < r21 < r20 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r24 - 2022-10-20 - DavidBrunner
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback