Review of AN-14-049

Most Recent Note Version

• AN2014_049_v5
• Changes from previous version:
  • Added relevant pdf sets to uncertainty
  • Changed pileup uncertainty MinBias Xsec
  • Updated JER to the most recent measurement
  • Updated JES extra CA8 uncertainty from 5% to 3%

Explicit green-lights from experts

Analysis concerns (PRE-APP)

Color code for answers.

• Green -- we agree, changes to analysis/documentations implemented.
• Lime -- we agree, but the item hasn't been done yet. (Open item.)
• Purple -- we disagree, changes to analysis/documentation is not implemented.
• Teal -- explanation offered but no changes to the analysis are implemented
• Blue-- authors/ARC/conveners need to discuss. (Open item.)

Comments from Alex

-have you cross-checked in simulated QCD events whether the top-mistagging rate in the 30GeV<mjet<70GeV || 100GeV<mjet region is the same as in the 70 < mjet < 100 GeV region?

• We wanted to do this, but the QCD Monte Carlo runs our of statistics quickly with the full top-tagging applied. I have attached the cross check, but it is hard to form any conclusions given the large uncertainties. Perhaps thre is a way to improve this? I will put this plot in the appendix.

follow-up: there are QCD samples which have a high-HT preselection. See for example the samples used in B2G-14-002. These samples would add significant statistics in your high-HT region (you are anyway using an high-HT trigger).

• Here is the New sample

-the data-driven closure test of the procedure (Fig 14) is impressive. Can you specify what selection criteria are applied exactly to data here? Is it the 30GeV<mjet<50GeV || 100GeV<mjet <130 GeV region plus top-tagging? Or is this the same as the full procedure, just using another W tag control region?

• Right, here we use the W mass sideband 30GeV<mjet<70GeV || 100GeV<mjet <130 GeV (The 50 was a typo and has been fixed) and also invert the N-subjettiness cut on the W candidate. After this, the same top tagging procedure is used on the data, and the same top-mistagging rates are used for the QCD estimate. The agreement seen in the data compared to the QCD estimate should not be dependent on the W sideband, so we take this as a valid test of procedure.

-line 260: I don't understand this sentence. I thought that "the pre top-tagged tt selection weighted by the top-mistagging rate" defines the QCD background estimate. Should this not better say that the ttbar contribution is subtracted from the pre-top-tagged tt selection weighted by.?

• I am confused here. The pre top-tagged ttbar selection weighted by the top-mistagging rate is a measure of the ttbar that is expected to fall through the QCD background estimate if we just applied the top-mistagging rate to data. Therefore we need to subtract it away in order to fully separate QCD and ttbar. Are you saying that it makes more sense to state that this is subtracted from the pre top tagged data sample BEFORE it is weighted? The end result should be the same, but this is not what the code does as the weight needs to be applied at an event by event level

follow-up: I think my confusion comes from the use of the term “top-mistagging”, which I associate to QCD and not to ttbar. There is no mis-tagging in ttbar. I now understand that you weight the whole pre-tagged sample with the mistagging rate to get the QCD prediction. This means you also weight the ttbar component in this region with the mistagging rate. Afterwards you subtract whatever ttbar is expected in this sample….

• I have added some clarifying text and a reference to the section explaining this

-lines 264 and following: it is not clear what exactly is extracted by fitting procedures. I had the opinion that you use the 130GeV>mjet region for obtaining a ttbar scale factor of 0.78 that is applied on all ttbar contributions in your analysis. Is this correct? However, you write here that the "The fit additionally implements tt subtraction in creation of the top-mistagging rate" . Are you now fitting these ttbar contributions or are you using a scale factor?

• We apply the scale factor to all of the ttbar contributions in the analysis. The sentence has to do with the creation of the scale factor and not the application. The scale factor is created by comparing QCD and ttbar shapes. However it is more complicated because we subtract off the ttbar contamination from the top-mistagging rate which is then used to estimate QCD. Therefore, the ttbar normalization needs to be partially anti-correlated with the QCD normalization when the fit is performed. We take this into account by adding an extra QCD uncertainty based on the ttbar contamination in the top-mistagging rate.

Comments from Dylan (Trigger Review)

Include errors in the efficiency plot (Figure 1).

• This will be in the next version. I think the trigger plot in the data sample section is already quite busy, so I made another trigger plot with the uncertainty to put in the systematics.

Please be more explicit in defining what you mean by "inefficiency" (line 315). If inefficiency is defined as 1-efficiency, can you explain why this is used, as opposed to the error from the efficiency calculation, in determining the trigger systematic?

• Yes, this is what we use. We considered finding a less conservative method of extracting the uncertainty. There is not very good agreement in the comparison of trigger efficiency measured in data and Monte Carlo (even QCD), so we use the data measurement. This is not completely explained but warrants a conservative uncertainty. In the end however, we are very close to the plateau and there is no noticeable effect from the trigger weighting, so having this conservative uncertainty does not really hurt the measurement.

Comments from Thomas (MC Review)

Table1: Single top cross sections are approx. NNLO, not full NNLO; for QCD you could add LO.

• The next version will contain these changes

l45: Single top is never mentioned later in the AN. Where do you use these samples?

• These samples are used in the background estimate for limit setting. I will clarify this point. It is only a small contribution

l116 and Sec 6.6: The recommended PU uncertainty is obtained by varying the minimum bias cross section by +/-5% (i.e. use 65.9 and 72.3mb as up and down variation, not 73.5mb). See https://twiki.cern.ch/twiki/bin/viewauth/CMS/PileupSystematicErrors for details.

• This has been implemented

Sec. 4.4: Is this selection overlapping with the region described in l184/185? Can you perform a closure test in a region that is not statistically independent from the sideband used to determine the scale factors?

• There is no overlap due to the fact that the sideband used to obtain the mistagging rate has the N-subjettiness cut of the signal region, and and sideband in section 4.4 has an inverted N-subjettiness selection. I should put this in words instead of just the table to make it clear.

Sec 6.7: Which Powheg MC samples are used to evaluate the Q^2 systematic, the inclusive fastsim samples or the high-mass fullsim samples?

• The high mass fullsim samples: TT_Mtt-1000toInf_CT10_scaledown_TuneZ2star_8TeV-powheg-tauola, TT_Mtt-1000toInf_CT10_scaleup_TuneZ2star_8TeV-powheg-tauola, TT_Mtt-700to1000_CT10_scaledown_TuneZ2star_8TeV-powheg-tauola, TT_Mtt-700to1000_CT10_scaleup_TuneZ2star_8TeV-powheg-tauola

Comments from Ivan (B-tagging Review)

- Section 3.6, lines 147 and following: with sentences like "The scalefactor is obtained from the plots shown in Figure 7" it is not fully clear that you refer to a measurement performed in JME-13-007, even if you include the reference, and that the reader should refer there for details on the extraction of the SF.

• I have attempted to make this more clear

- You don't apply dedicated subjet b-tagging SF, but use the inclusive top-tagging SF from JME-13-007 which includes the b-tagging selection. This is fine and it's not necessarily required to check the Delta R between the subjets, as you don't apply b-tagging SF to individual subjets. However, as the inclusive SF in not binned in pT you should add a statement that it holds and that it can be applied to your selection, as the kinematic region you cover and your pT spectrum is compatible with the selection used in JME-13-007 to measure the SF. The large uncertainty of 13% should account for differences.

• This will be in the next version.

- Has the 13% uncertainty a significant impact on the limits? In this case an in situ measurement of the SF, recently adopted by two Z' analyses, could be considered, which should lead to reduced uncertainties.

• Yes. This is a potential improvement. The ttbar normalization is measured in data, so it would not have an effect on that sample, but we would see an improvement for the signal uncertainty. I will conduct a study looking into this

- Line 274. You quote a ttbar normalization uncertainty of 11%. What else is then contributing to the 19-25% ttbar normalization uncertainty quoted in Table 7?

• Good catch! The uncertainty should read 18%. This was updated in the plots, tables and text from the paper, but not the text in the note. The uncertainty contains two sources. The main source is the uncertainty from the Theta fit. The other is the uncertainty in the sideband scale factor measurement. Ie we could not use the W-tagging scale factor because it was inverted in this selection, so we had to extract this scale factor.

Comments from John and Justin (JME Review)

L147 (V3): (response needed to give green light) Have you checked the deltaR between your subjets in the b* signal samples? If they are closer than 0.3 your scale factor will not be valid. My guess is that these will not be too boosted to be a problem.

• I think this is in reference to the subjet b-tagging portion of the scale factor? Here we use the JME-13-007 scale factor (as was used in B2G-12-009), not the BTV scale factor. These are extracted using the identical top-tagging selection as presented in the analysis. Overall this results in a direct scale factor measurement at the cost of a more conservative uncertainty. Perhaps there is a requested plot that could be used to validate the use of this scale factor? Also, see that this concern was raised in the b-tagging review, but due to the large uncertainty no change to the analysis was requested.

S3.3: (response needed to give green light) Do you apply JER smearing? You later describe applying a JER systematic, but not the nominal smearing.

• We use the nominal smearing. I realized that this was not mentioned. I will put it in the next version

L268: (response needed to give green light) Do you really take the JES uncertainty as a flat 5%? Why not use the pT and eta dependent uncertainty provided by the JME group?

• We use the pT and eta dependent uncertainty as well as an additional 5% (added in quadrature for using CA8 jets). This was very awkwardly worded in the note. This has been fixed.

General comment: It would be nice to see a couple signal masses overlaid on the plots with the stacked backgrounds.

• I have added the signal points to the full selection plots

L3-6: Why doesn’t b* -> bg dominate, as it is mediated by the strong interaction? How does the pair production cross compare to single production at ~1 TeV?

• Good question. I am not completely sure. at very low mass (300GeV) BR(tW) is ~20% at higher mass it is ~40%. The other main decays are bg,bZ,bH. I am not sure why the tW branching fraction dominates.

L144-145: Is the top-tagging SF applied to ttbar MC as well?

• Yes. The sentence should have just said "applied to the Monte Carlo samples". This has been fixed

S3.7: Does the mass window cut use the ungroomed mass, or do you apply grooming?

• We do not use ungroomed mass, although this is a potential improvement.

L165: Does this cut actually do anything?

• No. It just gives us good candidate jets to start with. In the case that the top decay is resolved, then we dont want the W and b to be the candidate tW jets. But this would not really pass our selection anyway

L176-177: Aside from an inverted W-tag mass requirement, what cuts are applied in this control region?

• The kinematic cuts and preselection are applied. The QCD background is from weighting the pre top-tagged sample. Then the conrol region selection contains top-tagging.

L204 (V3): What does it mean to say the bifurcation point was chosen “manually"? Is there an optimization procedure?

• There is no optimization procedure here. The fit is meant to represent data, so the point is chosen based on observed representation of the data. This could be expanded but it seems like the fit is quite a good approximation for the data, so we never saw the need.

S4.3: This correction confuses me. I am not terribly surprised that the top-jet candidate mass shape changes after application of the top-tagging requirements, since there could be correlations. But this effect should be in your data-driven QCD estimate. Do you not take these cuts into account when calculating the top mistag rate?

• These are taken into account when measuring the top-mistagging rate. But the mistagging rate is binned in pt. The pre top-tagged data selection has an incorrect mass shape and the top-mistagging rate alone can not correct for this except insofar as the pt and mass are directly correlated. The alternative would be to bin the top-mistagging rate in pt,mass but we very quickly run out of statistics here.

F11: What are the dashed lines?

• This is the uncertainty. I will make this clear

F14: By eye, it looks like less ttbar would make for a better fit. Does the RH plot show the normalization of QCD preferred by the fit, or the nominal value?

• All of the components take the shapes that are preferred by the fit in this plot. However we only use the ttbar correction extracted from this measurement. Hmm, the fit prefers the ttbar normaliation shown. It almost looks slightly unbalanced towards the left. Perhaps this is due to the QCD templates that the fit is given. In any case the uncertainty seems to cover the true value

Fig. 16 and 18 are the same, Fig. 17 is missing from PDF (V3)

• Good catch! This will be fixed in the next version

F17: It looks like you have a pretty serious pT(tW) mismodeling. There are two bins in a row, and four out of five, with a 1.5+ sigma deficit. Three of the bins go off the scale. Shouldn’t we be somewhat concerned by this? Do you apply top pT reweighting to the ttbar MC?

• Perhaps. I need to think about this. It is reassuring that the individual t or W pt spectrum does not look as bad. Wouldn't you expect a top pt discrepancy to show up here before the (t+w).pt spectrum? Also I think this should be nearly orthogonal to the M_tW measurement. Top pt reweighting is not applied here. This is the main point of the ttbar normalization measurement. pt reweighting is not extracted for use in our kinematic range or for use with powheg, so it made more sense to get a rough estimate of this in the analysis itself.

L333: One half of the correction seems rather arbitrary. How did you come up with this systematic?

• Yes. It is arbitrary. It is a rough correction and only a small effect on the M_tW variable so we went with a large uncertainty. I would be interested if there is a suggestion of a better method.

Figure 39 (V3) — why is the 1-sigma band so large for M_B = 900 GeV? The other mass points seem to have a more reasonable width.

• This point exists right at our kinematic threshold. The JES uncertainty skyrockets here (small decrease in pt and we lose a lot of events). Table 7 shows a ~50% normalization effect for these points. At higher B* mass points this uncertainty is more manageable. It is likely that by tweaking our selection we could recover a lot of efficiency here, but this lost efficiency is regained by the other channels

Comments from Roman

• I have updated the reference to what appears in the bstar paper. Did you mean the quoted result for the semileptonic/dileptonic channel?

l22: high transverse momentum of what?

• I have attempted to clarify this; Of the two leading jets.

l23-24: sentence is confusing, please rephrase

• I have attempted to make this more clear.

l51-52: please be more specific on how you obtain the vectorlike b* templates. which cross sections? how do you scale them?

• I have attempted to make this more clear. Basically take right and left, scale them to theory cross section and add them together

section 3.2: is there any change w.r.t. the all-hadronic W' analysis? if not, please mention that the trigger usage and efficiencies are the same.

• This has been added

section 3.4: Please motivate the jet veto for jets with pt > 150 GeV. In light of the single-produced b' + jet models, which we would like to have some sensitivity to, one would like to have a jet veto as loose as possible.

• The motivation is that we want to reject three prong type events that would not reconstruct M_tW correctly and may bias the background estimate

l135-140: How do you treat subjets with DeltaR < 0.4? Do you follow the BTV POG's recommendation to apply CA8 jet b-tagging in case the subjets are closer than 0.4?

• Right now we do not do this. The scale factor used for top tagging does not do this either, so at first order we should be ok. However perhaps this offers increased sensitivity?

section 3.8: maybe you should add a sentence saying how you reconstruct the b* mass

• This has been added

table 4: one would like to see the efficiency (rejection efficiency in this case) of the jet veto here.

• This has been inserted into the table. Attached is the figure from W' investigating this. This should stilll be valid

sections 4.1 and 4.2: please provide numbers here, maybe put them in a table: + number of QCD and ttbar events expected in the sideband region used to measure the mistag rate - for untagged and tagged events + number of QCD and ttbar events in the sideband region used to obtain the background templates, before and after the application of the top-tagging

• This table is now in the note.

l221: fix figure reference

• This has been fixed

l222: show how this correction affects the pt(top) and M_tW spectrum.

• See attached plots

l242: what do you mean with scaled ttbar?

• I tried to make the wording more clear. Basically the normalization that goes into the input histogram is very important. ttbar needs to be scaled to theory cross section using the correct scale factors except the extra correction that the fit is meant to measure.

l248-250: not clear what's happening here. I can't judge if this is correct. Please describe it better.

• The wording has changed due to a change in the analysis (which unfortunately makes the measurement more complicated). I tried to make it more clear, but it may still be confusing

l255: how exactly are the background templates obtained? what's the sideband region to which the top-mistag factors are applied? please describe this better, maybe give the cuts explicitly. also it's not clear from the context that this sideband is independent from the sideband that was used to obtain the top-mistag rates.

• The top-mistagging rate is used to weight the pre top-tagged selection to create the background estimate. I have added this to the text

chapter 5: please provide a table with expected and observed number of events in the signal region, with the corresponding uncertainties. also show the selection efficiency for your signal samples.

• I put the number of events observed and expected in the text, as it somewhat reproduces the information in table 4. The signal efficiency table has been created. Right now I just compute this with entries passed/entries total, so there could be some bias with if a shape based weight is used in the signal region, but it is probably pretty close

fig 17: one would like to see MET and HT here. please add these two distributions as well.

• I have added these distributions

l269-l270: how do you take the difference in the measured and simulated W mass into account? is this a subjet energy correction factor?

• This is JES for CA8 jets, which carries an extra factor in quadrature with the standard JEC uncertainties.

section 6.3 and 6.4: do you apply these also to the signal samples (text needs to be changed)

• The text has been changed.

section 6.5: why do you use the MRST2006nnlo PDF set, which is not used in any of your MC samples? Shouldn't this be CT10, which was used for the generation of the ttbar samples? Also, it's not clear what you do here. Do you calculate a PDF error for each of these PDF sets using all available eigenvectors (by summing the contributions in quadrature) and then take the error from the PDF set which gives the largest uncertainty? Or do you take the difference between the PDF sets and sum this in quadrature with the error obtained from the eigenvectors? It's a small uncertainty, but the description needs to be improved.

• I have attempted to clarift the procedure. Take the uncertainty for each set individually then use the largest deviation (same as EXO-12-024).
• The analysis now uses the CT10 PDF as well. The uncertainty is still small

l316-319: are these not applied to the ttbar samples? why not?

• Any purely normalization uncertainty is taken into account in ttbar by our measurement of the ttbar normalization and uncertainty.

l321 and l324: i guess it should be pre W-tagged. remnant from W' analysis?

• Yep (well pre top-tagged anyway). This has been fixed

l324: how are the statistical uncertainties taken into account?

• The statistical uncertainty is taken into account by using include_mc_uncertainties = True in theta

section 6.9.1: do you take into account also the constant fit to determine the uncertainty? it looks like that his overestimates the uncertainty somewhat.

• Yes. Originally the uncertainty was taken only from the difference from constant fit to nominal fit in order to be conservative. The current iteration is to take the mean squared error from many fits which is less conservative. However I think it is important to keep the constant fit, as this is a subjective uncertainty, and not completely crazy to assume a less rigorous approach would just use an unparameterized efficiency correction

l378-384 (also table 3): where do you take the the b* cross section from? is this just the LO prediction from madgraph?

• This has been changed in the latest version to be consistent with the other channels.

-- KevinNash - 22 Jul 2014