Draft Responses to comments on AN19-234

Comments from Petar 01/30/2021 (B2G Concerns)

Sec.11 Please explain how you actually calculate the likelihood. (I wasn't able to find it anywhere, so if this description exists, it's well-hidden.) For example, is this a counting experiment, or a shape-based measurement? For example, you can use the distribution of the S_T variable to further suppress the background, and have improved sensitivity at higher masses. (Your limits are currently ~ flat, which is not great.) The only indication I have that this is a shape-based likelihood are the nuisance pulls and parameter impacts on Figs 40-42. But I don't know of which variable !

JW: This is a shape-based likelihood of the S_T distribution. I have added the description in the AN.

Figs 39-40 Asimov data is that you have a background estimate, and then generate data that exactly matches it. So, naively, one would expect that the b-only fit (blue) has no pull, since the bkg estimate is already at the minimum, since there's a perfect agreement between the data and the model. Whereas there could be disagreements for the s+b fit. So I think something was reversed... Moreover, the amount of pull the fit that is pulled is still a bit large.

JW: I used S+B (signal strength = 1) to produce the Asimov data for fitting. I have updated this in the AN after setting signal strength to 0.

L413 Just to confirm: if an event has 3 or more tight leptons, then it will enters the multilepton analysis and not this one, right?

JW: Yes.

L416 Why do you need to do for ee but not for \mu\mu? Is it because you need these events to measure the charge mis-id for electrons?

JW: If events with ee from Z passed the same-sign dilepton requirement, they are definitely due to charge mis-ID. Since we use this region to measure data-driven charge mis-id background, these events must be vetoed to avoid double-counting. However, since \mu\mu events has negligibly small charge mis-id, we don’t need to worry too many \mu\mu from Z events passing the same-sign dilepton requirement, nor do we need these events for measuring charge mis-id.

L417 So if there are three leptons and two are tight, one of the SS leptons is allowed to be not tight? If you were to require that the SS leptons are the tight ones, what is the fraction of the signal events that would be lost?

JW: Sorry for the confusion. We require that SS leptons must be tight and that they are the only tight leptons in the event. But we allow events to have loose leptons. I have updated in the AN to make this clear.

I did try to do the counting by allowing one of the SS leptons to be loose in 2018 TpTp samples and the fraction of signal lost is only 0.3-0.5%.

TpTp	nEvts (Allow 1T1L SS)	nEvts (Tight SS Only)	% decrease
900	5130	5116	0.3%
1400	4664	4649	0.3%
1800	4197	4174	0.5%

Sec. 13.2 First of all, I should say that this is WAY better than just taking the ratio of SS/OS events! Let me check if I understand the fit from Eq.(8) and (9) correctly: you have 3 pt regions and 4 eta regions, so there is a total of 12 bins, i.e. indices i and j go from 0..11. The efficiency in each bin, eff[i] is what we are trying to measure, so each is a fit parameter; thus this is a 12 parameter fit.

n^SS[i,j] is thus a 12x12 matrix of measurements, and the prediction is n[i,j]*P(from Eq.(7)). Is this correct?

JW: Yes, this is correct.

Ibid. Next, since p_{ij}^{SS} = p_{ji}^{SS}, do you actually fit only 12*(12+1)/2 = 78 and not 144 bins?

JW: This is not actually the case. I use the first label for leading leptons and second label for sub-leading leptons, i.e. n_{ij} = n_{ji} . So the 2D histograms is actually “i” in x-axis with 12 bins and j in y-axis with 12 bins. So I did fit the 144 bins to find the 12 parameters $\epsilon_i$ after minimizing NLL.

Ibid. Also, do we then assume that the mis-id for muons is negligible?

JW: Yes.

Table 24 Uhm, here you actually have 5 eta bins! (In contradiction to L426.)

JW: I forgot to update the text after updating the binning. This is corrected in the AN.

Tables 26-27 Maybe I have asked this before, but I forgot... Why are the muon fake rates a factor of ~ 2.5 higher than the electron fake rates?

JW: Firstly, I would like to note that there has been changes to the fake rate. This is the version before I removed an upper cut on lepton pT. This has been updated in the AN. For the ease of discussion, I have attached the new measurement here.

Technically this should be called a nonprompt rate which allow us to derive the total nonprompt background from data. The nonprompt background is events with nonprompt leptons that can be coming from light quarks, c quarks, b quarks as well as mis-identification at reconstruction (“fake”). This means that nonprompt background also contains contributions from top and QCD processes.


We studied the nonprompt rates using the tt MC samples. In the tt MC samples, we look at all the loose leptons with $p_T$ > 30 and try to match each of them to a GEN particle. Each loose leptons is categorize into different flavor sources as follow:

The categories of nonprompt leptons are in bold : Unmatched, Fake, Bottom Quark, Charm Quark and Light Quark. We find the tight-to-loose ratios in each category respectively and in all categories combined (average). The result are shown in the tables below for 2017 (top) and 2018 (bottom). Since we are measuring nonprompt rate in data with selected leptons with $p_T$ > 30, the “Combined” column is the most representative to our measurement.

In both years, in “Combined” column, the nonprompt rate of muon is about 1.7 times of that of electron in Average. In row of Light Quarks, muon nonprompt rate is greater than 2 times of electron’s. In row of Charm Quarks, muon nonprompt rate is almost 3 times of electron’s. Given that the composition of sources is different in data, it is not alarming to see muon nonprompt rate ~ 2.5 higher than electron’s.

In addition, in 2016 X53 analysis (AN16-242), we observed muon nonprompt rates varying between 0.17 and 0.31. In 2017 and 2018 (Table 27), muon nonprompt rates are between 0.21 and 0.37 which are not unreasonable.


We did measure a lower electron nonprompt rate in 2017 and 2018 (between 0.10 and 0.14) compared to that in 2016 (between 0.16 and 0.25). However, there are changes in ID and triggers and so on, so it is natural that the nonprompt rate changes.

Fig.53 I'm curious, for ee and \mu\mu channels, are the large error bars for events above, say, 400 GeV, mostly driven by the fake-rate or by mis-id rate? (I don't really have the intuition for this...)

It fluctuates with bins as it is a rate uncertainty. Wherever NP (i.e. fake rate) background is non-zero, systematic uncertainty on NP background contributes the largest part to the total error. However, for ee channel, in some bins, systematic uncertainty on mis-id rate is comparable to that on fake rate.

Sec. 13.7 This looks like a counting experiment -- is that right? If so, did you consider subdividing data using one of the sensitive variables, e.g. H_T^lep from Figs.52 and 60? For example, you can split the events into three bins: [0,500] --> essentially all bkg, works as a CR [500,1500] --> a bit of signal but also lots of bkg >1500 --> good S/B ratio.

Yes, this is a counting experiment with a final selection requiring $H_T$ $>$ 1200.

This sounds like a good idea and we are currently developing the transition from counting experiment to binned fits. We observe some weird behavior when processing the binned template in Combine with systematics. We did a quick check in SS2L ’s most sensitive decay mode B’ -> tW (100%) to estimate the benefit of this proposal. Considering statistical uncertainties only, we observe the limit to go from 1141 GeV in simple counting experiments, up to 1175 GeV using binned templates. We will continue to try to debug but if it turns out to be way too time-consuming, this quick check also assures us that we are not missing much with or without this change.

Comments from Grace and Andrea 02/09/2021 (JME Concerns)

1) What are the JEC and JER versions you use?

JW: I have added this in the AN.

	JEC	JER
2017	Fall17_17Nov2017_V32	Fall17_V3
2018	Autumn18_V19	Autumn18_V7

2) We assume you use CHS jets. Is that correct?

JW: Yes. I have added the description in AN.

3) Do you use PF MET? What corrections do you apply on top?

JW: Yes. We applied the “Type-1” corrections on MET based on all jets in the event (with no kinematic thresholds). The description is added in AN.

4) Do you apply MET filters? I see a reference in Table 7, but no elsewhere in the text. Can you comment on this point?

JW: This might be an unclear naming. I have changed that to “Pass filters”. We did apply a series of filters on all events. I have added the description in the text. The list of filters are: - goodVertices (in addition to $z<$24.0 cm,$\rho<$2.0 cm,$\ge$4 degree of freedom) - HBHENoiseFilter - HBHENoiseIsolatedFilter - globalSuperTightHalo2016Filter - eeBadSCFilter (only on data) - EcalDeadCellTriggerPrimitiveFilter - BadPFMuonFilter - ecalBadCalibFilter (recomputed for 2017/2018 using the MiniAOD recipe)

5) Can you confirm you use the prefiring weights for 2017 in both your categories? I could find only a reference in Table 41.

JW: Yes I used prefiring weights for 2017 in both categories. It is mentioned in the systematics tables for both channels (Table 17 and 36). I have added a list in AN to specify all the corrections on MCs.

6) Are you affected by the HEM issue? Do you veto events if jets and or electrons are in that eta-phi region in 2018? I believe that, given the number of leptons in your analysis you should be safe from the fake electron rate, but can you provide a 2D eta-phi plot for 2018 runC-D? What about jets? Do you down-scale the MC?

JW: For 3+L, we have studied the 2D eta-phi plots of electrons and jets in 2018C and D. The plots do not show a very obvious hot spot overall. In addition, we checked the jet energy after down scaling in MC and plotted the ratio of down-scaled distribution to normal distribution. The ratio shows a change < 2.5 %. It seems fairly safe from the HEM issue

For SS2L, I also made the 2D eta-phi plots. This time the hot spot is quite obvious. Unfortunately, with the current framework of SS2L analyzer, it would involve quite some changes to plot pT of all jets, so I only did the jet energy down-scale check on H_T^{lep} (H_T+SS lep pT) distribution, which is what we are now using to make binned templates for Combine. The plot shows the ratio of down-scaled H_T to nominal H_T bin by bin and the changes are < 1.5%. For SS2L, we are vetoing events if any of the selected SS leptons is an electron and falls in the hot spot.

7) How is the data-MC agreement at high values (>1.4TeV) of ST in both years for your CRs? Does this affect your estimation of the SR?

JW: There is no cuts on ST and the plots stopping at 1.4 TeV is just a plotting feature. Also in the plots, all overflow is contained in the last bin.

8) How relevant is MET for your analysis? Especially at low values (<25 GeV). In particular, would a variation of this cut have some effect on your fake rate estimation?

I assume this is referring to the SS2L fake rate measurement. I studied this and plotted the overall fake rate at different cut value on MET, from 0 up to 100 at steps of 5 GeV. There is almost no change in FR with the MET cut.

Comments from Johan S. and John H. 02/15/2021 (PPD Concerns)

CMSSW Versioning: Thanks for including the production CMSSW versions on line 37. Can you also document the analysis-level CMSSW version in that same paragraph?

JW: Added in AN.

Luminosity and Uncertainty: In line 32 the 2017 luminosity is cited as 41.56 fb^-1, the the recommendation is 41.53 fb^1. Here is a link to the Lumi twiki . It is a small change, but can you check the correct value is used throughout the analysis? The uncertainty to the lumi is correctly documented in Table 17.

JW: Thank you. I have updated the entire AN regarding this.

Data campaign: Tables 1 and 2 show you use the ReReco data campaigns, can you specify this succinctly in the text as well? I do spot an issue here with the SingleMuon/EGamma datasets for 2018 EraD. The recommendations for 2018 are 17Sep2018 for Eras ABC, and 22Jan2019 for SingleMuon/EGamma or Prompt otherwise ( link to PdmV table ). For DoubleMuon you are using the correct Prompt datasets, but for EGamma and SingleMuon the wrong set is being used at the moment. Please update this going forward.

JW: This has been updated.

Golden JSON Files: The files cited in line 38-41 are for Prompt data reconstruction. Since you are using ReReco, the corresponding good run files should be used. The long-term fix is to rerun your data using the correct JSON files 2017: Collisions17/13TeV/ReReco/Cert_294927-306462_13TeV_EOY2017ReReco_Collisions17_JSON.txt 2018: Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt If you need a short term fix, John made a script that you can run on lxplus which takes in as arguments the paths of two JSON files to compare ( link to repo ). This will output the lumi blocks that differ between the files, so just filter out any of these that pass your selections.

JW: This have been fixed. Now all data are filtered according to the right JSON files.

Global Tags: I don't see which GT is used in the AN. Can you please document this? The GTs used during production of the miniAOD sets you use are documented in the PdmV table I linked above, but do you apply a newer version at the analysis level? I am especially interested in what you use since the JEC/JER has been updated since the production versions. I noticed this is an open item with the JME contacts' review ( HN link ). As long as the JEC/JER applied is the latest recommendation, you should be fine. Regardless, please document what you are using now.

Pileup Reweighting: I do not find in the AN if pileup reweighting is being applied. I see in Tables 17, 28, 30, and 31 that there is a pileup SF measurement and uncertainty for each process, but I cannot find if this is a custom pileup reweighting technique or something additional? Can you please clarify what it is you are doing for pileup reweighting and perhaps add supporting plots on the matter (especially if it is not the standard pileup reweighting prescription)? For reference the recommendation on pileup reweighting is here .

MET Filters: In Table 7 you mention applying MET filters, but can you explicitly list them so we are sure all of them are appropriately applied? For reference here is the recommendation twiki ( link )

JW: These are now described in the AN.

Comments from Titas and Arne 02/15/2021 (EGM/MUO Concerns)

Section 3.1:

- Why did you choose the ID working points you chose? There are dedicated IDs for high-pT muons (HighPtId) and electrons (HEEP ID). Have you compared signal/background efficiencies for different IDs? - Given that you might have highly boosted Z bosons, you might even profit from a combination of HighPtID + TrkHighPtId a la B2G-19-006. Have you considered this choice?

JW: We haven’t studied the two IDs you mentioned. The choice of IDs is to stay consistent with single-lepton channel since we are going to combine this with singleLepton channel.

- L69-70: What does it mean to apply a tracking scale factor of unity? It is no longer recommended to apply the tracking scale factor, so it should be removed entirely in the next iteration.

JW: It has been removed. Thank you for pointing this out.

Section 5.1:

- Are the SUSY trigger scale factors blessed by the EGM and MUO POGs?

JW: This is from a published SUSY paper SUS-19-008 (AN-2018/280).

- Have those trigger scale factors been measured with respect to exactly the same lepton ID + isolation as applied in your analysis

JW: The lepton ID and isolation is not exactly the same between AN-2018/280 and our analysis. The details of the IDs and isolations are listed in the table below.

	AN-2018/280	Our Analysis
Electron ID	94xMVA cuts at constant value [1,2] (WP“VLoose” similar to WPLoose and WP“Tight” has slightly higher efficiency than WP90)	WP90 and WPLoose
Electron Iso	"miniIso <0.07	miniIso < 0.1
	Lep pT / closest Jet pT > 0.78.	Lep-Jet dR > 0.4
	pT Rel (lep, closest jet) > 8.0.
Muon ID	Medium and Loose cut-based ID	Tight and Loose cut-based ID
Mum Iso	"miniIso <0.11	miniIso < 0.1
	Lep pT / closest Jet pT > 0.74	Lep-Jet dR > 0.4
	pT Rel (lep, closest jet) > 6.8.

[1] https://indico.cern.ch/event/719317/contributions/2963816/attachments/1630110/2598062/MVAidSUSY_10Apr18_SUSYMeeting.pdf [2] https://twiki.cern.ch/twiki/pub/CMS/SUSLeptonSF/MVAwp2018_14Dec2018.pdf

While there are differences between our IDs and isolation from SUSY’s, but these differences are very small when comparing each of these sets to trigger leptons and we do not expect these small differences to propagate and change trigger efficiencies by much.

In terms of ID, trigger leptons have much loser requirements applied on them than any of ours or SUSY’s ID. Thus the trigger is effectively just adding a momentum threshold, which is well below our 30 GeV pT cuts on leptons. Since we are applying SF for our lepton IDs, the trigger SFs should only be dependent on the trigger itself.

In terms of isolation, the effect of SUSY’s pT ratio / ptRel cuts are probably quite similar to our dR cut. In addition, trigger efficiencies are measured with tag-and-probe method which should be less affected by these cuts designed for boosted events.

Our cut value on miniIsolation is <0.03 while trigger isolation is based on lower-level cluster and track isolation. Given the much bigger differences between trigger isolation and miniIsolation 0.1± 0.03 , efficiencies calculated with SUSY’s miniIso cut would be almost identical to what we would have calculated. Moreover, raw efficiencies is not the concern for us, but only the differences between data and MC efficiencies. The propagation of theses small differences to trigger SFs are expected to be almost negligible.

- L96-97: How was the DZ filtering efficiency measured?

JW: It is measured by comparing the efficiency of the same trigger with and without DZ condition. 


- What is the "referenced literature" motivating the uncertainty of 1% per lepton leg and 0.3% for the DZ filtering?


JW: It is taken from AN2018-280 too. 

-- WingYanWong - 2021-03-12