TOP-16-007: Measurement of normalized top-quark-pair production cross section in the dilepton channel in pp collisions at sqrt{s}= 13 TeV

Overall comments:

The paper JHEP 207P 0817 v1 describes the measurement done by the CMS Collaboration of normalised differential cross sections of ttbar events, as function of several kinematic properties exploiting the dilepton final state. It uses a data sample of 2.1 fb-1 collected in 2015 at the LHC center-of-mass energy (c.o.m.) of 13 TeV. Concerning the originality of the work, it is important to mention that similar measurements have already been published by ATLAS using a data sample of 3.2 fb-1 collected in 2015 at the same c.o.m. energy: ATLAS Collaboration, Measurements of top-quark pair differential cross- sections in the eμ channel in pp collisions at ps = 13 TeV using the ATLAS detector, Eur. Phys. J. C 77 (2017) 292, arXiv: 1612.05220 [hep-ex] It is surprising that the Authors of JHEP 207P 0817 v not only do not cite this ATLAS paper, but also seem not aware of it since in the Introduction they state that their measurement is the first at a c.o.m. energy of 13 TeV. There are few differences with respect to the ATLAS publication. The relevant ones are reminded here in order to assess the originality of the JHEP 207P 0817 v1 paper.

* While all the ATLAS differential distributions are measured at particle level in the fiducial phase space, JHEP 207P 0817 v1 includes also few distributions at parton-level extrapolated to the full phase space. They REPORT JHEP_207P_0817 are compared with generators and with theory calculations. Parton- level measurements are more model dependent than particle-level mea- surements. The extrapolation to the full phase space introduces an additional dependency on the models. In spite of the model depen- dency, parton level results facilitate the comparison with theoretical calculations.

* At particle level the JHEP 207P 0817 v1 paper includes few additional differential cross section measurements: the transverse momentum of the leptons and of the jets and the angular difference in the transverse plane between the t and tbar.

* While ATLAS uses dilepton final states with a pair of isolated leptons of different flavours, in the CMS paper also same flavour (SF) dilepton channels (ee and μμ) are included. This is done in order to increase the size of the available data sample which is 2/3 of the ATLAS data sample. It is expected that final states with ee and μμ are affected by an higher level of background due to Z+jets events, therefore to fight this background, the CMS Analysers introduce selection criteria on the invariant mass of the dilepton pair and on the missing energy. It is a pity that they do not cite the final event yield and the fraction of SF contribution. The paper is of some relevance given that it contributes to the understand- ing of the ttbar process with the addition of few measurements and comparisons with theory with respect to the ATLAS paper and is the first CMS result at 13 TeV in the dilepton ttbar final state on the subject. Similar measurements have also been done by ATLAS and CMS at 13 TeV in the semi-leptonic ttbar final state but the dilepton ttbar final state may be in- teresting to use, due to possible different systematics w.r.t. the semileptonic ttbar final state. A better strategy for this paper would have been to include all the Run 2 data sample. Moreover it is a pity that the paper doesn’t extract at the same time the integrated cross section in the fiducial phase space. This is done in a separate paper: CMS Collaboration,“Measurement of the ttbar production cross section using events in the eμ final state in pp collisions at ps= 13 TeV”, Eur. Phys. J. C 77 (2017) 172, doi:10.1140/epjc/s10052-017-4718-8, arXiv:1611.04040. The scientific quality of the analysis appears fair, but in some points it is not easy to judge since the explanations are concise and lack relevant details. Otherwise the paper is well written.

In conclusion, the paper merits to be published, nevertheless some improve- ments and clarifications are mandatory.

INTRODUCTION

** The motivation of the measurements is too concise: there is no discussion what this paper brings of new, why it merits to be published and the choice of the kinematic variables is not explained. Any comment about using the semi-leptonic ttbar final state rather than the di-lepton ? Please add a short discussion.

Based on your comments, introduction is modified as followings:

The measurement of \ttbar differential cross sections can provide a test of perturbative quantum chromodynamic (QCD) calculations and also improve the knowledge of parton distribution functions (PDFs)~\cite{Czakon:2013tha}. Previous measurements of differential cross sections for \ttbar production have been performed in proton-proton (pp) collisions at the CERN LHC at center-of-mass energies of 7~\cite{Chatrchyan:2012saa, Aad:2015eia} and 8\TeV~\cite{Khachatryan:2015oqa,Aad:2015mbv,Khachatryan:2016oou, Khachatryan:2016gxp, Khachatryan:2015fwh,Khachatryan:2015mva,Aaboud:2016omn,Aaboud:2016iot, Sirunyan:2254647}. The dilepton (electron or muon) final state of the ${\ttbar}$ decay has brought benefits to suppress background events well. This paper presents the first CMS measurements at $\sqrt{s} = 13\TeV$ in the dilepton decay channels including the same-flavor lepton channels ($\Pe\Pe$ and $\Pgm\Pgm$), using data corresponding to an integrated luminosity of 2.1\fbinv. The data were recorded by the CMS experiment at the LHC in 2015, and this measurement complements other recent measurements that have been reported in a different decay channel~\cite{Khachatryan:2016mnb} and by a different experiment~\cite{Aaboud:2016xii}.

The \ttbar differential cross sections are performed at the particle and parton levels. Particle-level measurements use final-state kinematic observables that are experimentally measurable and theoretically well defined. Corrections are limited mainly to detector effects that can be determined experimentally. The particle-level measurements are designed to have minimal model dependencies. The visible differential cross section is defined for a phase space within the acceptance of the experiment. Large extrapolations into inaccessible phase-space regions are thus avoided. In contrast, the parton-level measurement of the top quark pair production cross sections is performed in the full phase space. This facilitates comparisons to predictions in perturbative QCD.

The normalized \ttbar differential cross sections are measured as a function of the kinematic properties of the \ttbar system, the top quarks and the top quark decay products: jets coming from the hadronization of bottom quarks and leptons. The particle-level measurements are included the transverse momentum of the leptons and of the jets. Invariance mass and rapidity of the \ttbar system are also measured to be used for better understanding of PDFs. The angular difference in the transverse plane between the top and anti-top are added to compare to new physics beyond standard model. In addition, the normalized \ttbar cross sections are measured as function of the transverse momenta of top quark and top quarks of \ttbar system.

SECTION 2.2

** The higher-order overlap of Wt and ttbar production is not even mentioned. See: S. Frixione, E. Laenen, P. Motylinski, B. R. Webber and C. D. White, Single-top hadroproduction in association with a W boson, JHEP 07 (2008) 029, arXiv: 0805.3067 [hep-ph].

Indeed, we briefly state how each background is generated and normalized. As Wt is not major background for this analysis, we kept the detail the same as the other backgrounds. However, we add the suggested reference.

SECTION 4

** In the 5th line from the top I suggest: “.. leads to results that are largely independent ..” (a residual model dependence is unavoidable)

We follow your suggestion.

This approach avoids theoretical uncertainties in the measurements owing to the different calculations within each generator, and leads to results that are largely independent of the generator implementation and tuning.

** Dressing procedure (this is only a comment): the use of the anti-kt al- gorithm is a cumbersome option. Also: what happens if the two charged leptons (not from hadron decays) are close and fall in the same cluster?

When two leptons are identified as one lepton in reconstruction, it will be treated as one lepton event. Similarly, in simulation, if two leptons are merged into one lepton, there is only one lepton is indicated. Therefore, it is accounted as “ttbar others” not ttbar dilepton event.

** It is not clear if all B-hadrons are ghost-associated or only those with really short lifetimes (B*, B**). If it is the first case, I suggest to drop “short-lifetime” in the expression “short-lifetime B hadrons”.

The expression “short-lifetime” emphasizes that B hadrons can’t be identified by itself due to “short” lifetime in detector, but B hadrons can be provided b-jet identification in simulation by being included jet cluster. Also, because of the expression “short-lifetime”, we don’t have to explain why B hadrons are scaled down to be negligible.

** The sentence: “The ttbar system at the parton level is calculated in the gener- ator at NLO, using the PYTHIA8 PS simulation.” should be improved since PYTHIA8 is not a NLO fixed order generator.

This is probably a misunderstanding because this sentence would point out that parton level calculation is done in NLO generator. Since the parton shower simulation is not necessary for this message, it is removed. Therefore, new sentences is as following:

The \ttbar system at the parton level is calculated in the generator at NLO.

** Please define at the end of the Section the ‘full’ phase space (no cuts at all?).

New sentence is changed from

The measurements at the parton level are performed separately in the full phase space.

to The normalized differential cross sections at the parton level are performed by extrapolating the measurements into the full phase space which is including experimentally inaccessible region.

SECTION 5

** A Table with the final expected and observed event yields before and after the neutrino reconstruction is strongly recommended.

To ARC members: We are not sure that we can add new tables with numbers. It isn’t review by CMS.

** I would remove the sentence “The data distributions are slightly lower than those from the MC simulation.”. This is confusing when looking at the lower right panels of Figure 1 and 2 and it is the following sentence which carries a more important message.

The sentence “The data distributions are slightly lower than those from the MC simulation.” gives motivation of equation (1) and (2). Move the sentence from Sec. 5 to Sec. 6 as following: The non-\ttbar backgrounds are first subtracted from the measured distributions. The data distributions are slightly lower than those from the MC simulation. The \ttbar-others backgrounds are then removed as a proportion of the total \ttbar contribution by applying a single correction factor $k$ shown in Eq.~(\ref{eq:extract}), using Eq. (\ref{eq:extract2}):

SECTION 6

** As it is writen now, reading carefully, one manages to understand that the calculation (2) is done per bin, but it is not completely clear if k is computed in each bin or not (“single correction factor”?). Please clarify in words.

To clarify this, we have changed the text as follows: Here, $N^{\mathrm{MC}}_{\mathrm{non}\textnormal{-}\ttbar}$ is the total estimate for the non-${\ttbar}$ background from the MC simulation, $N^{\mathrm{MC}}_{\ttbar\textnormal{-}\mathrm{sig}}$ is the total MC-predicted ${\ttbar}$ signal yield, and $N^{\mathrm{MC}}_{\ttbar\textnormal{-}\mathrm{others}}$ is the total MC prediction of the remaining \ttbar background. The \ttbar signal yield, $N^{\mathrm{data}}_{\ttbar\textnormal{-}\mathrm{sig}}$, is then extracted from the number of data events, $N^{\mathrm{data}}$, in each bin of the kinematic distributions, as shown in Eq.~(\ref{eq:extract2}).

** It is important to indicate the baseline generator that has been used to get the response matrix (even though we believe and hope that the result is independent of it).

New sentence is: The method relies on a response matrix that maps the expected relation between the true and reconstructed variables taken from the ${\POWHEG}$+\PYTHIA{}8 simulation.

SECTION 7

** Is the MC top-quark pT reweighting of the tt - others to data done after subtracting from data the non - ttbar ? Please, clarify.

To clarify “these”, it is replaced by “the \ttbar-others”, so new sentence is The uncertainty in the shape of the \ttbar-others contribution is obtained by reweighting the \pt distribution of the top quark for the \ttbar-others events to match the data and comparing with the unweighted contribution.

** Is the MC top-quark pT reweighting of the ttbar signal to data done after subtracting from data the non - ttbar and other - ttbar ? Please, clarify.

No, pT reweighting isn’t applied on the ttbar signal. They are only considered when we estimate uncertainty for ttbar-others contribution. That is why the pT measurements in data is than simulation.

** Table 2: it is said (correctly) that particle-level measurements are less model dependent than parton-level measurements. Why the max systematic uncertainty labeled in the Table “MC generator” is higher at particle-level w.r.t. parton level? (also surprising that the max fact./renorm effect is higher at particle level than parton level).

The systematic ranges are determined using median of each distribution. Maxs of median for MC generator and fact./renorm at particle level are higher than parton level, but mins of median are lower.

Updated v2 ** In which lines of Table 2 is included the effect of the uncertainty on the QCD radiation ?

As described as follows in Sec. 7:

The uncertainty from the choice of mF and mR is estimated by varying the scales by a factor of two up and down in POWHEG independently for the ME and PS steps. For the ME calculation, all possible combinations are considered independently, excluding the most extreme cases of (mF, mR) = (0.5, 2) and (2, 0.5) [59, 60]. The scale uncertainty in the PS modeling is assessed using dedicated MC samples with the scales varied up and down together. The uncertainties in the factorization and renormalization scale in the ME and PS calculations are taken as the envelope of the differences with respect to the nominal parameter choice.

Fact./renorm scale variations include fact/renorm variations not only in ME but also in PS, therefore this uncertainty covers QCD radiation uncertainties.

** Table 2. Do the last six lines refer to variations studied only in the signal sample, namely excluding the tt - bar others, and than the effect of these six theoretical uncertainties propagated to the tt - bar others is included in the line “Backgrounds” of the Table ?

We assume that nominal sample can represent true distribution. So, we consider statistical uncertainty estimated by nominal sample for tt-others.

SECTION 8

** The data distributions are said normalised but it is not said to what. Are they divided by the measured total cross-section, obtained by integrating over all bins of the differential distributions? And how are normalised the expected distributions? Please explain.

To add more information, change the sentence from

After subtracting the background contribution and correcting, Figs.~\ref{fig:particle_level_1} and \ref{fig:particle_level_2} show the normalized differential ${\ttbar}$ cross sections as a function of

to The normalized differential ${\ttbar}$ cross sections are performed by subtracting the background contribution, correcting for detector effects and acceptance, and dividing the number of ${\ttbar}$ signal events by the total inclusive ${\ttbar}$ cross section. Figs.~\ref{fig:particle_level_1} and \ref{fig:particle_level_2} show the normalized differential ${\ttbar}$ cross sections

Updated v2 ** Comparing in particular the bottom right panel of Fig 4 and 6 (this vari- able should be sensitive to QCD radiation) it seems that the particle level measurement is affected by a larger systematic uncertinty than the parton level. Is this understood?

Because of additional acceptance cut on ttbar events at particle level, defining of signal events at particle level is tighter than parton level. Also, this acceptance cut on signal definition for particle level is largely dependent to MC generator and hadronization.

Missing reference:

ATLAS Collaboration, Measurements of top-quark pair differential cross- sections in the eμ channel in pp collisions at ps = 13 TeV using the ATLAS detector, Eur. Phys. J. C 77 (2017) 292, arXiv: 1612.05220 [hep-ex]

Added

English and typos:

Section 2.1, third line: “Within the solenoid volume there are a silicon ..” (except in interrogative sentences the subject in written English goes before the verb)

Fixed

Caption Figure 2: the word ‘only’ is repeat two times consecutively.

Fixed

done except ( trigger[ref]: not yet, Used Sec. for Section. p value: no hyphen)

Line Change

Abstract "leptons, jets from bottom quark hadronisation" "several quantum chromodynamic models"

7 "perturbative quantum chromodynamic (QCD) calculations"

10 Delete either "center-of-mass energy" or "sqrt(s)". We don't put both in the same sentence.

12 "measurements that have"

15 "jets coming from the hadronisation of bottom quarks (b jets)"

16 "final-state" hyphen

17 "particle-level"

36, 39, "PYTHIA8" one word. 41, 49, 52

44, "Drell--Yan" 2 dashes for a combined name. 97

46 "tW channel" no hyphen

47 "five-flavour" hyphen

48 "WZ, (comma) and"

51 "final-state"

52 "hard-scattering" hyphen, "collisions in the same or nearby beam crossings (pileup)"

55 "in the data"

59 "from the two", "particle-flow" hyphen

63 "and pseudorapidity

69 "for the track from the silicon tracker"

70 "to an ECAL cluster." You have already defined ECAL.

81 "nonuniformity and nonlinearity"

85 "Jets from the hadronisation of b quarks (b jets) are"

88 "like-flavour"

92 We now use "p_T^miss" for the magnitude. See the publication guidelines.

93 "triggers [reference]" We have been asked to reference our CMS Trigger JINST paper as much as possible. See the detector twiki page.

93 "low-p_T" hyphen.

96 "as a b jet"

101 "radiation, (comma)", "particle-level"

102 "generator-level"

103 "due to the different"

104 "generator" delete the "s"

106, "final-state" 110, 115

111 "muons, (comma)"

113 "Section 4"

114 "generator-level"

117 "reconstructed-object", "the jet, (comma)", "ghost-B"

118 "is used in which B hadrons" Don't start a sentence with a symbol.

119 "as a b jet"

124 "a b jet"

126 "leptons, (comma) and b jets."

128 "non-dilepton"

137 "objects, (comma)"

139 "Breit--Wigner"

142 "b jet", "at the particle and parton levels."

143 "weights", "for the final"

146 "b jets"

147 Put the superscripts in roman font, "Fig. 2"

151 "than the MC"

157 "as the corresponding"

159 "inelastic" is misspelled.

161 "identification, (comma)"

164 "propagating"

Fig. 1 "band shows the combined statistical and systematic caption uncertainties added in quadrature."

172 "coupling strength" We prefer this since the parameter is not really constant.

176 "matrix-element and parton-shower"

184 "parton-shower"

185 "PYTHIA8" one word

188 "variables, (comma)" "Section"

189 "of each"

190 "at the particle and parton levels.", "is the dominant"

191 "source"

192 "modelling"

205, "PYTHIA8" 206

207 "the kinematic"

209 "Drell--Yan"

223 "that maps"

231 "Figs."

233 "parton-level"

234, "PYTHIA8" 237, 244

240 "that includes", "Tables 3 and 4"

242 "p values" no hyphen, "jet, (comma)"

243 "high-p_T"

244 "of the tt-bar"

246 "of the top"

Table "p values", "at the particle" 3 caption

248 "parton-level"

249 "Figs."

252 "leading-logarithm"

255 "double-differential"

256 "momentum (?) space"

257 "considered with"

260 "and using"

261 "on the"

262 "with the dof" you have already defined "dof".

263 "p values", "between the measurements"

266 "M^tt-bar, (comma)"

268 "measured by the CMS experiment in"

271 "in the full"

272 "space" is misspelled!

273 "fairly good"

274 "for both the particle and parton levels"

275 "for the parton level"

276 Put in the Acknowledgement section.

279 Put "C" in the journal name. Delete "no. 3"

286 Same thing.

292 Put spaces around the "=". Put "D" in the journal name. Delete "no. 5".

Do this for the rest of the references.

I notice you are not using pennames, e.g., "e" for electron is not in roman font. \Pe, \Pgm, and \Pgt are the pennames for the electron, muon, and tau. See style manual https://twiki.cern.ch/twiki/bin/view/CMS/Internal/PubGuidelines There are errors in formatting of some of the references (e.g., in [48], "C" should not be in boldface and you should give the first page number only); same thing in [51], etc. in [46] "t-bar" should be \oveline{t} as in [43]; eliminate "no. 3" from [53]

done

OK. I am currently re-reading the complete paper and will have some new comments (I know that already). You could start on one while awaiting the full set. Tables 6-19 in the appendices contain many too many digits. For example, I doubt seriously that you know the normalized differential cross-section in the pT bin 20-30 GeV to be 0.01946 +/- 0.0004478 +/- 0.001131. The best that I could imagine would be 0.0195 +/- 0.0004 +/- 0.0011.

done

L10: "correspond" -> "corresponding"

done

L15: The way it reads now is that the leptons are associated with b jets. Better would be to say "... the top quarks and the top quark decay products: leptons and the jets associated with bottom quarks (b jets)."

done

L18: I'm not quite clear on this. Is "variables mainly corrected for detector effects" a consequence of the reduced dependence or the cause? Thus use of "so" would indicate a consequence.

yes. it is consequence of the reduced dependence.

L39: "at the NLO" -> "at NLO"

done

L63: "The muons are required" -> "Muons are required"

done

LL66-68: I'm not sure what you are trying to say here. The isolation is supposed to account for all particles coming from the primary vertex, correct? Or are you talking about the "underlying event"? At any rate, the last part of the sentence should be rewritten to say "... as well as charged and neutral hadrons and photons from pileup."

Isolations are considered both primary and underlying event. Done as “The isolation requirement accounts for the contribution from additional particles such as charged hadrons from the primary vertex, as well as charged and neutral hadrons and photons from pileup.”

L70: "The electrons are required" -> "Electrons are required"

done

L111: I don't understand "and photons" here. Does this mean that you "dress" photons?

yes

LL121-122: You can remove "in particle system", as it should be clear from the context.

done

L165: "propagated" -> "propagating them"

done

L173: "strong constant" -> "strong coupling constant"

done

L178: "where result" -> "where the result"

done

L181: "are taken the" -> "are taken as the"

done

LL193-195: remove the spaces between the closing parentheses and the commas: "level) , " -> "level), ".

done

L206 (and everywhere this appears!): please make the spacing around the "+" signs consistent: either a space before and after or no spaces at all. So make it either, eg, "POWHEG + PHYTHIA 8" or "POWHEG+PYTHIA 8". Either works, just be symmetric. This applies here as well as L207, L235, LL238-239, LL245-246, Tables 3&4, and the captions for Figs 3-6.

done

L209: "While tt-other" -> "While the tt-other"

done

L227: "can bias to the measurement" -> "can bias the measurement". (At least I think that is what is meant.)

done

L246: rewrite to be "and the t and tt rapidity at the particle and parton levels."

done

L247: "quark for parton and particle level." -> "quark at the parton and particle levels."

done

- MadGraph5_aMC@NLO is the ONLY official name for this event generator. Please use only this name everywhere.

At Sec.2, MadGraph5_aMC@NLO is defined as MG5_aMC@NLO. Then, MG5_aMC@NLO is used throughout the paper as a shorthand. (as Pedro mentioned)

- the description of the ME element scale uncertainty: please use the ‘official’ description from the top wiki as you did. The mention of both independent and the fact that the extreme opposite values are not used is important information. If the LE (or PubComm) does not like this description I propose they work together with the TOP conveners on a phrasing that passes language review. But this information should be in the paper as it is important.

Modified Sec.7 as followings: The uncertainty from the choice of $\mu_F$ and $\mu_R$ is estimated by varying the scales by a factor of two up and down in \POWHEG independently for the matrix element and parton shower steps. For the matrix element, all possible combinations are considered excluding the most extreme cases of $\mu_{F,R}$ = (0.5,2) or (2,0.5) where result could not be controlled in resummation~\cite{Cacciari:2003fi,Catani:2003zt}. The scale uncertainty in the parton shower is assessed by dedicated MC samples where the scale has been varied up and down. Then, the uncertainties in the factorisation and renormalisation scales at the matrix element and the parton shower are taken the envelope of the differences with respect to the nominal parameter choice.

LL23-27: I know that some people really like this kind of paragraph, but it really isn't needed, and does break the flow from the intro to the next section. This isn't such a long paper, so I would suggest to just drop it.

dropped

L35: "... can be found in Ref. [13]."

done

L40: I do think that it is "MadGraph5_aMC@NLO" and not "MG5..."

Top group commonly used MG5_aMC@NLO instead of MadGraph5_aMC@NLO. We indicated that “MadGraph5_aMC@NLO (MG5_aMC@NLO)”

LL47-48: I don't understand what "Additional partons are used in MG5_aMC@NLO to produce ttbar events" means.

In order to simulate ${\ttbar}$ events with additional partons, \aMCatNLO is used.

L49: "... decaying into ..." -> "... yielding ..."

done

LL56-59: This is should be re-written more along the lines of what TOP-12-039 used.

added one more line: The detector response to the final state particles is simulated using \GEANTfour~\cite{Allison:2006ve}. Additional pp collisions (pileup) are also simulated with \PYTHIA8 and superimposed on the hard scattering events using a pileup multiplicity distribution that reflects that of the analysed data. “The simulation is weighted to be in agreement with the distribution in the number of pileup vertices of data.” The events are reconstructed and analysed with the same software used to process the data.

L69: "... background. The isolation ..." (You don't need the "For muons", since the whole paragraph is on muons.

done

L74: "... track, matched to ..."

done

LL79-80: :The electron isolation considers the sum of the pt of hadrons (both neutral and charged) and the photon PF candidates around the electron."

done

LL85-86: repeated use of "residual". You should delete one of them, probably the first.

done

L89: Sorry, but jets don't come from B hadron decays. So say "Jets from b quarks"

done

L96: "... are required to have ..."

done

LL100-101: Repeated use of "additiion"; just say: "For the same-flavour lepton ..."

done

LL106-107: Repeated use of "difference". In addition, this sentence is pretty vague, and should be improved to describe exactly what you are trying to avoid.

modified to “This approach avoids theoretical uncertainties in the measurements due to different theoretical calculation of each generators, and leads to results that are expected to be independent of the generator implementation and tuning.”

L121: I don't think that Ref 37 is correct here ("The Catchment area of jets")

since we explained how we define b-jet, ref is removed. Slightly modified sentence. “To identify the flavour of the jet the ghost B hadron technique is used. B hadrons are included in the jet clustering after scaling down their momentum to be negligible.”

L124: I don't think that you've defined what a "pseudo-W" is. If this is the definition, then make it clear that in the following we'll call this a "pseudo-W boson"

modified as “Pseudo-W bosons are defined by combining a dressed lepton and a prompt neutrino in particle system.”

LL131-132: I'm not sure what the sentence "tt-others ... channel." is doing here.

Explained that ‘events not in the visible phase space from dilepton’ are also considered as background. modified as “Events that are not in the visible phase space from the dilepton channel are considered as background and combined with \ttbar-others (non dilepton \ttbar decay).”

LL133-134: Is one sentence enough to define what we mean by "parton level"? It's not clear to me.

one sentence is good enough. moved this sentence at the beginning of section as following: “The top quark and \ttbar system observables are defined at a level before the top quark decay and after QCD radiation which we refer to as the parton level. The particle level top quark is also defined by following the generator level prescriptions used…”

L153: The list of variables are not terribly different from each other. It is difficult to distinguish "l" from "j" for example, especially as a superscript. It might make more sense to use something like \pT^lep vs \pT^jet

changed everything at plot/table/text

L167: I don't understand "antagonistically"

Word "antagonistically" isn’t commonly used for physics analysis. So removed/modifed/re-arranged paragraph as following: “The pileup distribution used in the simulation is varied, by shifting the assumed total intelastic pp cross section by $\pm5\%$, in order to determine the associated systematic uncertainty. The systematic uncertainties in the lepton trigger, identification and isolation efficiencies are determined by varying the measured scale factors by their total uncertainties. Uncertainties in the jet energy scale (JES) and jet energy resolution (JER) are determined on a per-jet basis by shifting the energies of jets and propagated to the event level quantities~\cite{Chatrchyan:2011ds}.”

L182: What does "independently" but "not in opposite directions" mean?

Both words make readers confused. so, removed both words: “The uncertainty from the choice of $\mu_F$ and $\mu_R$ is estimated by varying the scales by a factor of two up and down, and taking the envelope of the differences with respect to the nominal parameter choice.”

LL196-199: You've quoted your uncertainties with at least one too many "significant" digits. Two digits is more than enough (and is still probably better than how well you really know them).

done

Table 2: ditto on precision.

done

LL223-226: rewrite this one sentence into two, one on purity and the other on stability. How do you get 50% purity and 50% stability? Can you control each independently?

we optimized bin width as both values(purity and stability) are around ~50% (as much as possible).

L235: "... found in Ref. [11]."

done

LL245-246: You can eliminate the sentence "We perform ... predictions." You've essentially said that in the preceding sentence.

done

L248: I believe that it should be "$p$-values, both here and in all of the tables (and captions).

done

LL250-251: "PowHeg+Phytia 8 describes better the pT ... parton level, while Powheg+Herwig++ ..."

done

LL254-268: This paragraph starts out OK, but then it just turns into a list, and it isn't even clear why. For example, the sentence starting at L260 is quite difficult to read. Someone who knows what is trying to be said here should help the authors.

The list is about QCD calculations. QCD calculations are listed as item.

L275: I'm not sure what "broadly" means here.

We can’t say they are in good agreement. It is acceptable agreement. Change “broadly” to “fairly”

Ref: [12]: I believe it is published now.

not yet. will be updated when it is published.

Also, previous comments about references are checked.

GEANT4 is different (TOP-12-039 uses NIM A503 (2003) 250)

They are different. We used same ref as TOP-16-008, l+jet channel.

MadGraph5aMC@NLO: both use the same reference, but TOP-16-007 version is v2.2.2 while TOP-12-039 is v5.1.3.30 (and is it MadGraph5 or MG5?)

fixed.

CUETP8M1: TOP-16-007 uses Skands et al EurPhysJ C74(2014)3024; TOP-12-039 uses CMS(!) EurPhysJ C76(2016)155 (is this a Pythia 8 vs Pythia 6 issue?)

yes it is Pythia 8

POWHEG: one reference in common; TOP-16-007 has a second ref; TOP-12-039 has 3 additional refs, two more recent than those in TOP-16-007

added more

TOP-16-007: no indication of what PDFs have been used.

it is mentioned at Systematics part.

- Around L92-93: Maybe specify what b-tagging efficiency this working point corresponds to? Added for v7

- Table 2: The table can be misleading since the uncertainties are different for different distributions, and the ones shown are the 'worst case scenario' (so the apparent combined uncertainty based on these are larger than for the given variables). Suggest to expand the caption for the table to explain clearly what these numbers correspond to. Modified for v7

- L204, 237: "standard model predictions" --> I think you mean compared to predictions from different generators (or similar)? Done for v7

Editorial comments:

- L87: which -> that Done for v7

- L89: decay -> decays Done for v7

- L97: trigger -> triggers Done for v7

- L97: threshold -> thresholds Done for v7

- L98: and high -> and a high Done for v7

- L156: fairly described -> fairly well described It isn't quite good agreement, so we used word "fairly".

- L159: systematic uncertainties -> systematic uncertainty Done for v7

- L160: as each systematic uncertainties -> as corresponding systematic uncertainty (?) Done for v7

- L191: computed by computing -> computed by taking Modified for v7

- L207: MC expectations -> MC predictions Done for v7

- L208: background contribution -> the background contribution Done for v7

• Remove jet and b-jet multiplicity plots; Those are not necessary to this paper; Added AN.

• Signal definition is modified; Added parton definition

• Here is more information for typical systematic uncertainty table; See the followings; https://twiki.cern.ch/twiki/pub/Sandbox/PaperTop16007/sys_med.pdf

• Differential cross section plots are updated

• Re-done Chi2-tests with full error covariance matrices; Updated tables

• Differential cross section tables on appendix are updated

L45: are you sure it is NNPDF30_lo_as_0130 for MG55_aMC@NLO? This is a LO set and it is unlikely that it has been used on top of a NLO calculation. Maybe that set was actually used for Madgraph (which is LO)? Fixed and rephrased for v4

L104:I think you should comment something on the disagreement in the jet multiplicity? Added for v4

L154: the comment "the data is slightly lower than MC simulation" is not generally applicable to all the distributions you show. Some show a scale effect, other show a slope in the ratio. I think you should expand the discussion a bit. Added more about pt_l, j, t

L185: Are you using 30% flat for all background, or only for non top ones, and how is the 30% determined. A short explanation is needed. This is old recipe, so we added reference

Table 3: I'm confused by the message that emerges from the chi2. The chi2 tends to be good (and the p-value high) for distributions with large experimental systematics. What I mean is that the table might give the message that some distributions are described better than other, while instead some distribution show a better chi2 just because the uncertainty is larger. Also, why pTjet chi2 is so bad? The agreement is not so bad looking at Fig 4 top right. We considered full errors while we performed chi2 test. New tables are updated on v5. I think this is because of log-scale plot for pt_j

General: Did you evaluate the covariance matrix for the systematic uncertainties (needed for hepdata and the chi2 tests)? We evaluated covariance matrix with different ttbar MC samples when we estimated generator systematics.

Line-by-line comments (some type A, some type B):

L13: “have been performed” done for v4!

L45: Were the NNPDF30_lo_as_0130 PDFs used for all MG5_aMC@NLO samples, or just the LO one? Fixed and rephrased for v4

L48: Is FxFx merging used for powheg (also one of the NLO generators), or just MG5_aMC@NLO? Fixed and rephrased for v4

L50: Suggest to say “calculated inclusive cross section”. done for v4!

L72: I think “from pileup” should be removed here as it does not apply to “charged hadrons from the primary vertex”? done for v4!

L79: 1.5660 (consistent number of decimal places) done for v4!

L97: passing (instead of passed) done for v4!

L102: shows done for v4!

L102: the Z boson mass M_Z done for v4!

L102: make some comment on the data-MC disagreement of Figure 1. done for v4!

L141: Does this mean you’re using slightly different top reconstruction for the particle and parton measurements (different Mlb distributions for weights)? Is the parton-level reconstruction identical to that used in Ref [8]? yes. we used different input plots for mlb and W for particle/parton level. But techincal method are same as Ref [8]

L148: “The final solution of the tt system is the solution with the maximum weight”: I thought the weighted average over the 100 trial solutions was used? (I think I have asked about this before, but I don’t remember the answer and I think it could be explained more clearly). Rephrased for v4

L154: “The data is slightly lower than MC simulation”: I think a more accurate description is required. You could link it back to the description added on L102. Rephrased for v4

L170: Remove “ME”. (I think the scales aren’t just varied at ME level, as reported on L179). Done for v4

L175: Does the new recommendation not to quote the powheg vs mc@nlo uncertainty apply to this analysis? https://twiki.cern.ch/twiki/bin/view/CMS/TopSystematics#Generator_uncertainty TOP-16-008(semi-lepton channel) also quoted the powheg vs mc@nlo uncertainty. Need to be discussed

L180: Why take the max deviation instead of the average deviation for the top mass systematic? The max will be more affected by MC statistical uncertainty (which I guess may be dominating the systematic). What do you get if you use samples separated in top mass by +/-3 GeV instead? Top mass are well-measured, so we varied only +- 1 GeV. Also, similar measurement of TOP-16-008(semi-lepton channel) used +- 1 GeV variation

L191: It's odd that the modelling systematic is bigger for particle level than parton level, as this is supposed to be one of the benefits of using particle level. See also my comment on L175. Need to be checked

L192: Why is the background systematic so much larger for particle level? I would expect it to be comparable to parton level. Signal definitions are different . For parton we didn't consider lepton and jet from top, but we checked visible phase sapce for leptons and jets for particle level. There are large background contribution from dilepton channel that leptons and jets are flying to out of phase space.

L214: Remove “efficiencies” since this is not related to migration. done for v4!

L226: Detailed done for v4!

L246: I don’t think you can say “good agreement”, because several are not really in agreement. Removed "good" for v4!

Table 3: Were these chi2 calculated using the full statistical+systematic covariance matrix? %Blue%We considered full errors while we performed chi2 test. New tables are updated on v5.

Table 5: <0.35 should be 0.35? done for v4!

I also have two old comments on the figures that we didn’t have time to change for the PAS:

1. DeltaPhi figures: could use units of pi on the x axis (making the bin edge align with axis ticks and meaning you don't use lots of decimal places in the y-axis label to improve the presentation for the paper) done for v4!

2. You could try to fix the overlap between some of the bottom y-axis labels in the main figures and the top y-axis labels in the ratio panels (e.g. in Fig 4). The easiest way is to adjust the axis ranges so that a label does not appear at the very top or bottom, e.g. change 1.5 -> 1.49. This is the more general solution: https://root.cern.ch/phpBB3/viewtopic.php?f=3&t=7080&sid=25fb8d1474795f4ad574734d0e687a92&start=15#p77091 Also, I think you can zoom in a little bit in Figs 4,5, e.g. I think 0.61 - 1.39 still covers all the points and bands. We use 0.61-1.39. We checked plots and adjust the axis range.

- General: nothing is said about trigger selection, I would include some minimal definition of the selection applied at online level done for v3!

- Abstract and L11: the year of data taking is not mentioned, I believe it is worthwhile to point it out done for v3! Added at Introduction

- L39-40: Which PDF used Powheg? mentioned! also at systematics

- L81: Reference 39 is given for Lepton and trigger SFs, but that PAS uses as reference for SFs (references 9 and 53 in PAS-TOP-16-011): CMS Collaboration, “Measurement of the top quark pair production cross section in proton-proton collisions at sqrt(s) = 13 TeV with the CMS detector”, Phys. Rev. Lett. 116 (2016) 052002, doi:10.1103

CMS Collaboration, “Measurement of the Drell-Yan cross sections in pp collisions at sqrt(s) =7 TeV with the CMS experiment”, JHEP 10 (2011) 007, doi:10.1007/JHEP10(2011)007, arXiv:1108.0566.

Better to use those, or the former done for v3! Former one

-L118: Is reference 48, "Pileup subtraction using jet areas",adecuated to ilustrate ghost B hadron technique? done for v3!

- L144 and L153: Not sure if factorization scale is varied at PS, it was at Run1, but not in Run2 with Pythia8. done for v3!

-L167: space missing between JES,JER done for v3!

- L194-195: I would clarifiy further (here or in captions) that Figs1 and 2 are showing reconstructed observables, not particle nor parton level distributions. Signal definition at particle level is different from parton level. Mentioned at background part.

-L219: Reference 61, l+jets differential analysis is given in the justification of the number of iterations, was this same approach used in that analysis? If so, please clarify Yes, it is. cite l+jet reference. done for v3!

- L232: "POWHEG +PYTHIA 8 is better described pT of tt system for particle and parton level." It is the MC which describes better the observable trend seen with data, right? yes

- L238: "is better" -> "has better" oe "seems to perform better" done for v3!

- L240-251: Figures 7 and 8 are not referenced in the text. done for v3!

- L250-251: found to be IN good agreement done for v3!

- L257: space, predictions repeated done for v3!

L235 for the high done for v3!

L250/251 In general...agreement : One has to be careful because in practice only NNLO predictions are available to describe all the variables. Maybe rephrase as: The rapidity of the top and the \pt of the \ttbar system are found to be in fair agreement with the different predictions considered. We observe some tension between data and the NNLO predictions for other variables such as the top quark \pt, M_\ttbar and y^\ttbar. done for v3!

Picking one comment from Louise and Maria, could be good to include the tables with the measurements performed. done for v3!

* Section 1

- L11-12: The sentence is ambiguous. Proposal (for simplicity): "The normalised tt differential cross section is measured in the dilepton channel as a function of the kinematic properties of the tt system, the top quarks, and the leptons and jets associated with bottom (b) quarks (b jets) from top quark decays." You can specify later in the paper which distributions you measure for each object. The current text suggests that you also measure, for example, invariant mass of leptons and b jets. done!

- L19: "compare with theory predictions." --> "compare with perturbative QCD calculations beyond next-to-leading-order (NLO) accuracy." done!

* Section 3

- L43: "with the dilepton invariant mass [...]" --> remove this part of the sentence, as it is not necessary. done!

* Section 4

- Please remove the formula for the relative isolation (it is not necessary for the paper) and just explain it in a sentence. done!

- L90-91: Please use the definition of MET provided in the CMS Publication Guidelines (https://twiki.cern.ch/twiki/bin/viewauth/CMS/Internal/PubGuidelines). done!

- L92-93: Please remove (not necessary). done!

- L98: It would be good to add some control plots after the full event selection but before the kinematic reconstruction. done for v3! jet and b-jet multiplicity plots are added.

* Section 5

- L107: Please explain what "dressed" means here. Also, write "dressed" in quotation marks. done!

* Section 6

- Please explain clearly at the beginning of the section how the systematic uncertainties are determined. Are the differential cross sections recalculated for each variation of each source of systematic uncertainty, and then the difference of the changed result wrt the nominal in each bin is taken as systematic uncertainty? done!

- L154-157 and Table 2: - It is not clear why you choose to calculate typical values of systematic uncertainties for 3 distributions only, or why you choose these 3 in particular. Please explain/justify in the text. done!

- The table corresponds only to particle-level observables. A similar table should be provided also for parton-level observables. done!

* Section 7

- General: - The kinematic reconstruction and the corresponding control distributions, namely L166-L185 and figures 1-2, should be described in a new section that should go before Section 6. done for v3!

- It would be useful to add an event yield table after full event selection and kinematic reconstruction. An event yield is added as text instead of table.

- The discussion of the results and the summary have to be very much extended, based on the results of the chi2/ndof obtained for each prediction per distribution. done!

- L171-180: - As reference for the kinematic reconstruction, please cite Ref.4 instead of Ref.26 (which is a PAS, and therefore cannot be cited). What are the differences between the kinematic reconstruction you use and the one used in Refs.4 and 26? For parton level it should be exactly the same, right? Otherwise, please explain. - Please elaborate more on how the kinematic reconstruction at parton and particle level differ. done! They are all same except lb mass and W plots.

- Fig 1-2: please add in the ratio an uncertainty band containing the statistical uncertainty and all shape systematic uncertainties. done!

- L193-198: I am not sure how necessary it is to describe in such detail the subtraction of tt background, since the procedure is neither new (it was done in the 8 TeV differential cross section paper in Ref.4) nor special. I would propose to remove Eq.1 and just describe how it is done in the text. What is actually missing is a justification on why you do it (to avoid the dependence on the total inclusive tt cross section that is used for normalizing the signal MC sample, I understand). Mentioned. Left equation instead of adding reference

- L207: Please add a few words on the performance of the unfolding procedure and the tests that have been performed to check for possible biases from the choice of the ttbar model. Answers in Progress

- L221-223: Please extend the description on how the chi2 tests are performed. Also, the chi2 tests should be performed for the theoretical predictions. done!

- L224: Please add a paragraph with some description of the different theoretical calculations you are comparing to. You can take the description from TOP-16-011 (pages 9-10), as it has been "blessed" by the different authors of the predictions. done!

- Legends of Figures 7,8: Please refer to the predictions by their corresponding publications. For example: "NNLO (A. Mitov et al.)" --> "NNLO (arXiv:1606.03350)". done!

- Please add tables for all distributions with the normalized differential cross section and the statistical, systematic, and total uncertainties per bin. done for v3!

Abstract:

L4 remove "collected in 2015" done!

L5 cross section is -> cross sections are done!

L7 models of perturbative QCD -> perturbative QCD models done!

L8 and later L231-232 in the text "in agreement with the standard model predictions" this is not accurate as summary. Tables 3 and 4 clearly show that none of the predictions is able to describe all the variables studied simultaneously. Please rephrase these sentences done!

Text L3 remove "explaining" done!

L5 Thus, the done!

L7 provides -> provide / remove "good" / Ref. [1] seems a bit arbitrary to me - please remove it. Suggest to add instead the following to the end of the sentence. "Some recent examples can be found in~\cite{Czakon:2015owf, Czakon:2016vfr}." where the references can be taken from http://inspirehep.net/record/1478979/export/hx and http://inspirehep.net/record/1402357/export/hx done!

L8 The -> Several / sections for \ttbar production have / the LHC References [4,5] should be expanded to include the following 8 TeV papers done!

CMS http://inspirehep.net/record/1473674/export/hx http://inspirehep.net/record/1454211/export/hx http://inspirehep.net/record/1394164/export/hx http://inspirehep.net/record/1397174/export/hx done!

ATLAS http://inspirehep.net/record/1473191/export/hx http://inspirehep.net/record/1477814/export/hx done!

L9-10 The...detector. -> In this paper a new measurement, at a different center-of-mass energy, $\sqrt{s}=13 TeV$, is reported, using data which correspond to an integrated luminosity of 2.2\fbinv. The measurement complements other recent measurements which have performed in different decay channels~\cite{13TeV papers}. For the 13TeV papers the list should be done!

CMS http://inspirehep.net/record/1491950/export/hx done!

ATLAS http://inspirehep.net/record/1495243/export/hx done!

L18-19 Also...predictions. -> These measurements are complemented by a measurement of the normalised \ttbar differential cross sections in the full phase space, at parton level . done!

After L19 please add a paragraph describing the organization of the paper. done!

L21-28 - I see some typos in the current description. please use the latest version of the CMS detector description available @ https://twiki.cern.ch/twiki/bin/viewauth/CMS/Internal/PubDetector done!

L31 in addition to [8] quote the hvq model paper https://inspirehep.net/record/756360/export/hx done!

L32 and assuming done!

L33 of -> obtained with done!

L35 spell-out LO done!

L37 is modelled using -> simulation is performed with done!

L38-40 This pertains only the Madgraph-based samples as Powheg uses a different prescription - please clarify Answers in Progress

L42/L44 remove "+jets" done!

L45 The single -> Single done!

L46 scheme~\cite{Re:2010bp}. where the reference is http://inspirehep.net/record/867473/export/hx done!

L51 the luminosity profile -> that done!

L57 Particle Flow -> particle flow done!

L78 data-driven -> data-based done!

L111 Sec. 4, \ie done!

L112 change one of the "with" to "using" done!

L113 0.4 . The clustering is applied / on -> to done!

L116 Ref. [35] seems suspicious to me as the one to be given for B hadrons. Is this correct? done!

L119 Pseudo-W...reconstructed -> We denominate as pseudo-W bosons two particle systems reconstructed done!

A -> In each event a done!

L122/123 with...GeV -> requiring that the invariant mass is close to 172.5 GeV. done!

Table 1 decay*s* done!

L130 you never defined barrel and endcap - make sure such description is in the CMS detector section done!

L132-134 is based...to determine -> is varied, by shifting the assumed total intelastic pp cross section by \pm 5%, in order to determine done!

L135 tag and probe for JES and JER ? can't be... please rephrase appropriately done!

L139 NNPDF is not based on eigvenvector but on MCMC replicas, please rephrase done!

L142 in addition to these you should consider the two alphaS variations of the PDF and add to form the PDF+alphaS uncertainty. Did you do it? If so, please state it. If not please do it and state it. done!

L151 please give the reference for HERWIG++ since its the first time and state the UE tune used in that case done!

L152 move [39] to the end of the sentence and quote instead https://inspirehep.net/record/1497736/export/hx which corresponds to the same dataset done!

L154-156 Typical...level-> Table 2 illustrates typical values for the magnitude of the systematic uncertainties in the measurement of several variables which will be explained later in Section 7. The values are computed by computing the median value of the distribution of the uncertainties in each bin of the measurement of the corresponding differential cross sections at particle level. done!

L157 Please add "Other experimental uncertainties such as JES,JER and b-tagging efficiency are found to be relevant for the measurement of most variables." done!

Table 2 please include, for comparison a line with the statistical uncertainty and in the text add a sentence on which uncertainty is the dominating one at this point. done!

L164-165 - remove this last part "which is...physics". This is true for most of the variables you have listed. done!

L172 [26] can't be quoted. As TOP-16-007 is the paper with 2015 data there is no need to quote other papers for this. done!

L185 please comment in more detail about the level of (dis)agreement of the variables Answers in Progress

Figure 1/2 remove preliminary and "Dilepton". Move CMS to inside the box. Which uncertainties are represented here? Only statistical? Please refer in the caption Would be interesting to include in the ratio panels other curves as well. E.g. to Powheg+Herwig++ or aMC@NLO FxFx. Would it be possible? Sys & Stat uncertainties are included as band.Other curves hasn't included yet.

L191-198 move this paragraph entirely to appear before the paragraph in L186-190 which is already discussing a different subject done!

L208 Start with "After subtracting the background contribution and unfolding to particle level, Figs. 3 and 4..." done!

L213-215 Remove "The main...efficiency" as this was previously discussed in Section 6. done!

L221 We ... predictions. -> The compatibility between the measurements and the predictions has been quantified by means of a $\chi^2$ which includes the bin-by-bin difference between data and predictions and the full covariance matrix. done!

L221/222 Tab. 3 and Tab. 4 are lists of -> Tables 3 and 4 report the values obtained for the done!

L223 This sentence is not accurate enough. Table 3 shows first of all that none of the considered MCs is able to describe simultaneously all the observables. Some variables such as \pt of jets, leptons,\pt(\ttbar), y(\ttbar) are poorly described by all of them. Top \pt is better described by Powheg+PYTHIA8 while M(ttbar) and the \ttbar decorrelation is described better by Powheg wrt to Madgraph setups. Please rephrase this to reflect somehow the main conclusions. done!

Figs 3/4 remove preliminary and "Dilepton". Move CMS to inside the box. enlarge font in the bottom panels done!

Table 3/4 - please enlarge considerably these table done! - calcuated -> calculated done!

L211 Please expand here: - how is parton level defined? done! - how is the extrapolation to full space performed (how large are the corrections and the associated uncertainties)? done!

L224 Figures 7...predictions. -> The parton level results have been furthermore compared to different fixed-order perturbative calculations~\cite{all_papers_for_predictions_used}. done! In addition please add a table, similar to Table 4 where the chi^2 is compared to these theory predictions and refer the main conclusions in the text.

References [30,37] Journal of Instrumentation -> JINST done!

[24,25,26,29,30,31,32,33] can't be used as a reference in a paper, please use papers wherever available. If techniques are similar to those in Run I please use Run I papers referring in the text s.th. like "similarly to the procedure described in~\cite{runI}" done!

[34] this is a special case but can't be quoted because it's a PAS - maybe can use TOP-16-008 paper as a reference for the pseudo-top prescription - or use the same reference it has used done!

[42] Technical Report ? removed

[45] update reference to the paper done!