Measurement of double parton scattering in proton-proton collision events with four jets with low transverse momentum at 13 TeV

This page collects review information for the analysis SMP-20-007



Pre-approval date: 09 September 2020

Approval data: 20 October 2020

Contact: Maxim Pieters

ARC Members: Klaus Rabbertz (chair), Philip Chang, Hwidong Yoo, Mate Csanad, Pierre Van Mechelen (CCLE)

Authors: M. Pieters, H. Van Haevermaet, P. Van Mechelen

CADI: SMP-20-007

Hypernews: SMP-20-007

Documentation: Latest AN-2019/068 version: v5 - Latest Paper draft: v12

Paper Review: draft v17 -- CWR Comments (March 1, 2021)

Color code:

Green: comment accepted and implemented

Blue: comment accepted, but text has been modified in a different way than proposed

Orange: response given, but no change has been implemented

All comments from Vassili Kachanov

1) Page 1, the lines 25-26
"For fully uncorrelated production of A and B, sigma_eff tends to the
total inelastic cross section, while ..."
The total inelastic section of what?

Changed line 25-26 to: "... the total inelastic cross section of proton-proton interactions..."

2) Page 1, the lines 28-30
The phrase "Indeed, the jets resulting from DPS are more often than not
produced in two independent pairs in a back-to-back configuration."
looks too cumbersome.
We propose to simplify it:
"Indeed, the jets resulting from DPS are more often occur in a back-to-back
configuration than SPS jets."

The reformulation looks double to us, "the jets resulting from DPS" are the four jets of interest of which only the jets created in the same interaction are more likely to be in a back-to-back configuration.

3) Page 1, the lines 42-43
"The CMS collaboration additionally obtained a measurement for ..."
"The CMS collaboration additionally performed a measurement for ..."

Line 42-43 had been changed accordingly.

4) Page 3, "Observables"
It is well known, that "2- > 4" process is described by 7 independent
variables. However, the authors provide the detailed description
of only SIX ones.

In the paper six observables are used that are not necessarily completely independent of each other. The main goal is to look at certain DPS sensitive observables by studying the balance between the jets. Our onset is not to give a complete description of the process.

5) Pages 5 and 6, Multijet models section.
The authors have used MadGraph5 and POWHEGBOX generators.
However, it is not clear why they need to consider KATIE predictions.
We recommend to provide additional motivation for inclusion KATIE model.

KaTie allows for off-shell production in the kT factorization scheme, which allows the initial partons to be off their respective mass-shell. Therefore, the created jet pairs can deviate from the typical back-to-back configurations obtained in on-shell production. A second argument is that collinear factorization is expected to fail at low pT, whereas kT factorzation is expected to show an improvement in this region of the phase space.

When introducing KaTie we have added the following to line 182: "... based on kT-factorization [61-63], allowing for on-shell and off-shell production. In the case of the latter, the initial partons are generated with a non-zero intrinsic $k_\mathrm{T}$ which can alter the momentum balance of the jets, yielding different topologies compared to on-shell production."

6) Page 8, the lines 269-271
"A combined event is discarded whenever two or more jet axes spatially
coincide. This veto condition is ..., where the indices i and j indicate
jets belonging to an event from the first and second data sample,
It is not clear should we apply this criteria for jets belonging
to one sample ?

The criteria is only between jets coming from event A compared the jets coming from event B, per definition no overlapping jets within the same event are found as they are reconstructed as one. The veto condition aims to remove coincidentally overlapping jets coming from the two seperate interactions, will clarify in the text.

We have reformulated it to: "..., where the indices $i$ and $j$ indicate jets of an event originating from the datasets of the processes A and B, respectively."

7) Page 8, line 274
The value epsilon_{4j} = 0.324... has used in several places.
It make sense to put it makes sense to put it on a separate line

Has been implemented in the paper. The four jet efficiency has been replaced with references to that equation.

8) Page 11.
It would be useful to briefly consider the nature of the very large
difference between cross section values sigma_A and sigma_B.

Added "A large increase in cross section is observed when lowering the $p_\mathrm{T}$ threshold from 50 GeV to 30 GeV, as is expected." right after the cross sections sigma_A and sigma_B.

9) Page 11, the line 362
"... labelled P8, ..." ==> "... labelled as P8, ..."

Changed accordingly.

10) Page 24, the 499
two commas are missed
"... from data as described in Sec. 6 is used ..."
==> "... from data, as described in Sec. 6, is used ..."

Changed accordingly.

11) Page 24, the 521
The value of sigma_eff = 32 +- 1 ... is appeared at this place only.
One can not find this value in Fig. 14 or Tab. 5

The effective cross section that we give here is not extracted from from the data through the template fit. It is rather a separate calculation: it is observed that the CDPSTP8S1 -4j tune predicts an excess of DPS events given the effective cross section of 21.3 fm that it uses, therefore, we tried to calculate the optimal effective cross section for the tune and do not think the number belongs with the extraction results.

All comments from Hyon San Seo

Type A
L88+1: jet pair -> jet pairs

Implimented accordingly.

L140: dipole antenna shower -> dipole-antenna shower

Implimented accordingly.

L193: For two the off-shell samples -> For two off-shell samples

Implimented accordingly.

L194: transverse momentum dependent (TMD) PDFs -> transverse-momentum-dependent (TMD) PDFs

Implimented accordingly.

L326: between 1 and 16% -> between 1% and 16%

Implimented accordingly.

Table1: What is eta_A and eta_B? Is delta S_DPS referring to data-driven template? But not defined

DS_DPS is now defined right above the four-jet efficiency in section 6 and eta_A and eta_B defined in the text right after eq.(2) in section 6.

L358: As I understand, sigma A and B is not used for DPS cross section, but to extract sigma_eff with it

Yes, this is true.

L370 : low p_T jets -> low-p_T jets (to be consistent with the expression in L368) Implimented accordingly.

Table3: NNPDF2.1 LO -> NNPDF2.3 LO according to the main text

Implimented accordingly.

L444: Angular ordered parton showers -> Angular-ordered parton showers

Implimented accordingly.

L513: in 15 -> in Fig. 15

Implimented accordingly.

L561: four jet -> four jets

Implimented accordingly.

L572: angular ordered/dipole antenna -> angular-ordered/dipole-antenna

Implimented accordingly.

Figure15: The expression "the fitted MC model" is confusing. Maybe my misunderstand, but isn't it just bin-normalized PW sample?

It is the bin-normalized PW sample, however, it has been scaled in the fitting procedure, we have changed it to "the scaled MC model".

Figures: delta phi_Soft -> delta phi soft (to be consistent with the main text)
Figures: delta pt_Soft -> delta pt soft (to be consistent with the main text)

Both have been made consistent.

L97: Since your analysis uses jets with high pseudo-rapidity, it would be good if you specify more information about forward calorimeter.

In order to be consistent we have added a description of all calorimeters (ECAL, HCAL and HF).

L119: Can we say the parton shower is a next-to-leading-log precision? It is between LL and NLL depending on an observable, and it would be safe just saying LL.

We checked the Pythia 8 manual, you are correct, the scentence has been changed accordingly.

L221: Why do you use 4 single-jet triggers with different pt threshold? Are some of them prescaled or having additional filter, or cut? It would be better describe some details.

The triggers are prescaled indeed, using them in disjunct pT ranges yield more statistics, we have added the following scentence: "... a \PT above 30, 50, 80, or 100 GeV, and within $|\eta| < 4.7$. As the triggers have been prescaled, they are used in disjunct \PT ranges. Offline cuts ..."

L267: How the events are divided into two disjunct datasets, is it randomly divided? I think it would be good to briefly mention how it is done.

Changed "Events that meet the online and offline analysis criteria are stored in two disjunct datasets, respectively."
to "Events that meet the online and offline analysis criteria for the processes A and B are stored in two disjunct datasets, respectively."

L337: Is luminosity uncertainty 2.5% like other analysis although your analysis is based on low PU run? And for further information, LUM-17-003 (ReadyForFR) shows lumi uncertainty in 2016 is 1.3%.

We had taken the number of LUM-17-001, however, if the paper is published before ours, we will make the necessary changes.

Figure12: Unlike other observable, the effect of adding DPS contribution for phi_ij is not noticeable in DPS-sensitive region. Rather, it looks like there are more DPS contribution in high delta phi_ij in KT offshell sample when comparing with Figure8. It would be great if there is some comment on it

In the first section of the results we show the large (!) dependence of the observable on the type of parton shower, we conclude there that the obserable is more apropriate to study parton shower effects rather than DPS.

L549: Is sigma_eff universal even for different sqrt(s)? Maybe other 13 TeV results are more suitable for comparison.

It is still open if the sigma_eff deducted from the pocket formula is a propber observable and whats its dependencies exactly are, naivley one does expect it to rise with sqrt(s), however, we think it is indicateve to compare it with older measurements to see the evolution of the measurements.

All comments from Daniel Bloch

- abstract: 1 line before the end : what do you mean by earlier measurements ? do you mean measurements already published in a previous paper ? or do you mean the measurements you discussed previously in the abstract ?

Replaced by "previous measurements" as we want to compare by measurements performed in the same topology but at different center-of-mass energies.

- section 1:

- line 30 : The DPS cross section rises with increasing center-of-mass energy at fixed pT : Isn’t it something true for all processes ?

Yes, this is true.

- line 55-57 : For a reader external to SMP, this part is not sufficient to understand what is done… why do you need to mix the pure DPS signal with inclusive single-jet events ?
After reading section 6, maybe you wanted to say something like this :
a pure DPS signal sample is created from data by randomly mixing together two inclusive single-jet events.

Altered the scentence to "a pure DPS signal sample is reconstructed from data by randomly mixing two inclusive single-jet events into an inclusive DPS four-jet event, whereafter it is fitted along with several SPS-only background MC models to the distributions obtained from four-jet data." in order to make the statement more clear

- section 2: Do I understand correctly that the hard jet pair is produced by one interaction (leading to the jets a and b on Fig 1 right), while the soft jet part is produced by the other (= c and d on Fig 1 right) ? If yes, could you write it somewhere in the paper. If not, how can I see that the two softest jets are more often back-to-back in DPS if they don’t come from the same interaction ?

Fig. 1 is the ideal case, in reality the pT of the jets of the different interactions can mix whenever the two created jet pairs are close in pT to each other. On average the two softest jets come from the same interaction and will flatten the curve, we have checked the different jet configuraions in a MC study.

- section 4:

- line 141 : what is the default tune ? same comment in Table 2.

The default parameter values of Pythia 8.3 are used, we have changed "default tune" to "default parameter values of \PYTHIA~8.301"

- section 4.3:

- line 205-208 : For the second PYTHIA8 sample, you write that it has already been mentioned before, what I understand as lines 135-138. But the first sample with CP5 seems also to have been mentioned before, as in lines 133-134. Is it correct ? If yes, please rephrase it saying that it too has been mentioned before. If not, then explain briefly why it is not the same sample as before…

The SPS part has been mentioned before, however, the sample now includes an explicit DPS part to the original part described at 133-134, we changed it to: "A \PYTHIA~8.240 sample is generated with the \texttt{CP5} tune and the \texttt{NNPDF3.1 nnlo} PDF. It is the aforementioned \PYTHIA~8 sample to which a DPS contribution is added."

In fact, when reading the full section 4.3, it is very confusing that some sample of PYTHIA8 was already mentioned before, while at least for the KATIE samples it seems that the samples with DPS contribution are different from the multijets one. For a non-expert reader, could you specify what is the difference between multijets and SPS+DPS ? Do the multijets include only the SPS case ? And what about the PYTHIA samples ? Do they include both SPS+DPS in all cases ? or sometimes only SPS ?

All samples in the first two paragraphs (Pythia and Herwig samples, multijet models) obtain a small DPS contribution from the tune they are interfaced with, however, this DPS contribution is much softer as what is needed for these DPS sensitive models. Therefore, models with an explicit DPS contribution (in addition to the much softer MPI from the tune) is added in order to obtain the SPS+DPS models.

- section 6: It is not clear which data you are using to extract the DPS cross section.

What I understand is the following :
- sigma_A (sigma_B) is extracted from events with at least 1 jet with pT > 50 (30).
- you construct a combined sample : mixing one event with at least 1 jet with pT > 50 GeV and one event with at least 1 jet with pT > 30 GeV.
- you want to extract sigma_eff which is computed knowing sigma_A, sigma_B, sigma_DPS, efficiency_4jets.

The fact that you mention that ∆S observable is chosen for the extraction of the DPS cross section in this part of the description (line 282) is very confusing for later. Because at this step of the explanation, I am still within the combined sample.

-> why do you need the Delta S observable to compute sigma_eff (line 282) ?

We need to calculate the fraction of DPS events which gives us sigma_DPS by integrating it. This is done by examining the discrepancy between the data and SPS models for the DeltaS observable. The difference in shape is a measure for the amount of DPS events.

-> In which sample is determined the efficiency_4jets ? In simulation ? or in the combined sample ?

We have changed "data" to "combined sample" in order to clarify, we now write "... ($\epsilon_{\rm 4j}$) has been obtained from combined sample as detailed below." in what was line 265.

-> which events are used to extract sigma_DPS ? After the full reading of the section, I suppose it is from the real four-jet data events (as indicated on line 290). But I am not sure as you spend a big part of section 6 to describe the combined sample without introducing any change of samples in the text…

The combined sample is used to obtain a signal template in order to estimate the amount of DPS events (needed for the calculation of the sigma_DPS). Together with the SPS models, the DPS template is fitted to the data in this regard.

-> on line 284, you mention the DPS fraction. In which sample do you extract it ? As you derive the signal template from the combined sample, I suppose that the DPS fraction is equal to 1 in that sample. So I suppose you are back in the four-jet data sample for the estimate of fDPS… If yes, it should be specified on line 284.

It has been stated explicitly now, what was line 284 now reads "The fraction of DPS events, $f_\mathrm{DPS}$, is extracted by performing a template fit to the unfolded \DS distribution, obtained from the original inclusive four-jet sample."

-> on line 287, why do you write “which is used to extract the ∆S distribution in exactly the same manner as before” ?

We have stated it more clearly, line 287 now reads "The signal template is taken from the combined data sample, which is used to extract the \DS distribution. The \DS signal template is corrected to generator level in exactly the same manner as before, including the correction for detector effects by means of unfolding."

- section 8

- lines 358 and 359 : I don’t understand this sentence. Why do we need the cross sections (5) to (8) to determine the DPS cross section ? I understood that the the DPS cross section is determined from the fit of the Delta S observable (fit of eq (3)) which allows to evaluate f_DPS, then used in eq (4)) ?

Cross section (5) is of interest because it allows for the comparison of the cross section for the other observable appart from the DeltaS observable. Cross section (6) is needed in order to obtain the DPS cross section, as from the template method only a fraction of DPS events is obtained, e.g. 3.5% and thus the DPS cross section = 3.5% x (6). The cross sections (7) and (8) are needed in the pocket formula as they are the two independent processes A and B in the pocket formula.

- Fig 15 : I don’t understand what are the two ratios in the bottom panel : one is PW/data and the other is (PW+DPS)/data ? If so, the color code of the hatched zones should be better chosen. With a yellow hatched zone, it provides the feeling that we look at DPS/data (and anyhow yellow is usually hardly visible...)

We have added a legend to the ratio plots in order to clarify what is represented by which curves.

- lines 514-521 : I miss why you need to know sigma_eff to derive f_DPS. This probably related to the fact that I don’t understand in which order you extract which information (in section 6).
I find very bizarre to obtain sigma_eff with a procedure where you impose an input value of sigma_eff. Why don’t you then perform a second iteration,
i.e. estimate a new value of f_true with sigma_eff = 32 mb as input ? Would in that case the value of sigma_eff = 32 mb be stable ?

The model with the CDPSTP8S1 -4j tune already has an explicit DPS contribution, however, it has been derived for 7 TeV data. With its fraction of DPS events, a fraction of DPS events is related. We also see that the model overestimates the data, thus the input value of sigma_eff of the model is wrong in the 13 TeV case with our cuts, therefore we invert the calculation in order to obtain the value of the sigma_eff that would be optimal for the model for our phase space.

Comments on the text :

-->> Text will be adapted accordingly.

- line 29 : Indeed, the jets resulting from DPS are more often than not produced in two independent pairs in a back-to-back configuration. —> Indeed, the jets resulting from DPS are more often than from SPS, produced in two independent pairs in a back-to-back configuration.

Implemented accordingly.

- line 119: give a reference for DGLAP

Included the necessary references.

- line 229-230: defined by cuts -> defined by selection cuts

Implemented accordingly.

- blank line after 263: can be removed

Implemented accordingly.

- line 274: write in parentheses that the systematics on eps_4j are explained in the next section

The 4-jet efficiency has been given its own equation, followed by "Where the statistical uncertainty was found to be negligible and the systematic error is detailed in the next section."

- blank line after 357: can be removed

Implemented accordingly.

- line 358 what is a: DPS pocket formula ?

Has been replaced by Eq.(1), a reference to the formula.

- line 359 : A dot is missing at the end of the sentence.

Implemented accordingly.

- section 8.2 : missing reference to Table 3 in the text of this section

Reference included in the first paragraph "Table~\ref{tab:MultiJet} gives a complete overview of all models, tunes, and PDF sets, along with their respective cross sections."

- line 413 : Fig 6 shows only the pT distributions. Need to add Fig 7 in the sentence to cover also the eta distributions. Or remove the reference to eta in case you want to point only to Fig 6.

Removed the "eta spectra".

- line 513 : shown in 15 -> shown in Fig. 15

Implemented accordingly.

- line 516: attach mb to 21.3 (21.3~mb)

Implemented accordingly.

- Fig.15 caption line 2: In the ratio -> The ratio
Line 3: are show -> are shown

Implemented accordingly.

- line 586: used to the describe -> used to describe

Implemented accordingly.

All comments from Igor Volobouev

Type A comments:

L 33: extend --> extent.

Changed accordingly.

L 48: We suggest removing the word "such".

The text reads now reads "... allows not only to mitigate pileup effects, but also ..."

L 193: For two the --> For the two.

Changed accordingly.

L 428: and models --> models.

Changed accordingly.

L 463-464: These lines are easy to miss, please consider moving Fig. 10
to the top of the page.

L 559: While the --> The.

Changed accordingly.

Figures: In many figures in this paper, the yellow bands obscure the
vertical axis ticks. It would be better to draw the ticks
on top of these bands. It is often difficult to see the
point/band locations given that the ticks are hidden.
"RedrawAxis" would help.

All figures have been updated with the RedrawAxis " function.


All references have been updated accordingly.

L 598: R=0.5 --> R = 0.5

L 605: protonproton --> proton–proton

L 634: DOI link is broken

L 639: tevatron --> Tevatron

L 646: DOI link is broken

L 648: cern --> CERN

L 649: Physics Letters B --> Phys. Lett. B

L 651: four jet --> four-jet

L 670: sigma(eff) --> σeff

L 681: Production of four-jets --> Four-jet production

L 685: kt --> kT

L 688: cms --> CMS

L 713: Physics Letters B --> Phys. Lett. B

L 723: DOI not found

L 727: Nuclear Instruments and Methods in Physics Research Section A:
Accelerators, Spectrometers, Detectors and Associated Equipment
--> Nucl. Instrum. Meth. A

L 735: Vincia --> VINCIA

L 743: Accepted for publication. --> arXiv link?, this paper has been
submitted to the arXiv now as 2011.03422. Note that the paper
title has been changed.

L 768: small x --> small-x

L 782: TUnfold: --> TUnfold,

L 787: Svd --> SVD,
Nuclear Instruments and Methods in Physics Research Section A:
Accelerators, Spectrometers, Detectors and Associated Equipment
--> Nucl. Instrum. Meth. A

L 793: ROOT: --> ROOT

Type B comments:

L 30-31: It is not clear p_T of what is referred to in this statement.

Concerns the pT of the jets, has been altered to "... at fixed \PT of the jets. ..."

L 108: It is unclear what the words "in situ" mean in this context,
as the citation at the end of this statement is for a different
data sample.

"In situ" has been replaced, the scentence now reads "...Dedicated measurements of the momentum balance in dijet, photon+jet, Z+jet ..."

L 226: What type of reference trigger is used?

We use a minimum bias trigger, has been made clear in the text which now reads "... by comparing the performance of the jet trigger with that a minimum bias trigger serving as a reference trigger."

L 227: "...exactly one vertex...". Do you mean exactly one primary vertex?

Yes, exactly one primary vertex, text has been altered accordingly.

L 239-242: Naively, if you use the bin width which is twice larger than
the resolution, you expect that 68% of events do not migrate
outside the bin. This results in a diagonally dominant detector
response matrix which is easily invertible without any
additional regularization. Why, then, do you need to regularize
your unfolding procedure instead of simply inverting the
response matrix? As all regularization procedures are based on
some kind of a bias-variance trade-off, this leads to increase
in the bias and potentially underestimates statistical
uncertainties of the results.

The TUnfold algorithm requires a basic regularization matrix, almost all use a regularization on matrix based on the amplitude of the output distribution. However, the ScanLCurve function from the TUnfold package determines the optimal regularization strength through a cubic spline interpolation, which is in most cases small. It is the go-to unfolding package as it is much more reliable compared to bin-to-bin or d'Agostini unfolding methods.

Section 6: In general, it is not obvious why the jet reconstruction
inefficiencies (e.g., due to parton direction overlap) in
your signal template constructed from data would be similar
to those produced in a genuine DPS process which might
exhibit directional correlations. Perhaps, the relevant
assumptions should be spelled out explicitly.

We have added "In the derivation of the formula above, the partons originating from the same hadron are assumed to be fully uncorrelated. In Fig.~\ref{fig:SPSvsDPS} this would mean that the partons $i$ and $j$ as well as the partons $k$ and $l$ are fully uncorrelated." to the introduction, right below where the DPS pocket formula is introduced. As the incoming partons are assumed to be uncorrelated, the jets produced in the processes A and B will be uncorrelated. It seems more indicative to us to place the main assumption there.

All comments from Giuseppe Barbagli

Type B.
13-34 No references in this part ?

Included a reference to a general DPS paper where the pocket formula as well as many different aspects of DPS are detailed.

Eq.(1) this equation is referred later (page 8 line 358) to as
'DPS pocket formula'; if you do want to use this jargon language define eq.(1)
as 'DPS pocket formula' here.
See also comment at 358.

We have replaced "DPS pocket formula" with eq.(1).

29-30 back-to-back
Should we specify "in the transverse plane" ?

We have added "in the transverse plane" to the text. The text now reads "two independent pairs in a back-to-back configuration in the trnaverse plane. "

The pileup definition (or better description) looks unnecessarily
complicated... which is the reason to refer to 'nearby BX'?
Is nearby a proper term ? Nearby with respect to what?

It is supposed to differentiate between in-time and out-of-time pile up. In order to clarify, "nearby" has been replaced by "different".

48 but also to reach down to low jet pT
Why ?

Less pileup results in a less convoluted or complicated events. In addition, the offset corrections (to correct for pileup) are not needed, yielding a smaller uncertainty on the JEC factors (which is still the largest in our study).

49 albeit necessitating 49 a custom calibration of the jet energy scale....

No appropriated/correct JES factors were available for low pT low pileup data, if we were to apply the standard corrections provided by JETMET, our results would have been vastly different due to the systematic and wrongfull scaling of all jets. All corrections have been presented to and were approved by the JETMET group.

55-59 The DPS cross section is extracted with
a template method: a pure DPS signal sample is reconstructed from data by
randomly mixing inclusive single-jet events and is fitted along with several
SPS-only background MC models to the distributions obtained from four-jet
data. Finally, the effective cross section is computed
using Eq. (1), with σ A and σ B measured from data.

This part could be better explained.
It is absolutely not clear...

The lines have been rephrased as: "The DPS cross section is extracted with a template method: a pure DPS signal sample is reconstructed from data by randomly mixing two inclusive single-jet events into an inclusive DPS four-jet event, whereafter it is fitted along with several SPS-only background MC models to the distributions obtained from inclusive four-jet data."

Page 3 bullet after 82 The math expression does represent the variable only
if i, j are the jets giving the largest DeltaY; should it be re-written in a
better way ? is the notation φ_[ij] misleading ?; what
about φ_[Delta eta_[max]] or similar?

The notation has been used in multiple papers before, it was opted to use the same notation in order to stay conform with previous studies.

Section 3 The section contains the reference [35], used for the definition
of the coordinate system and the kinematic variables; should we
swap Section 2 (where eta, phi are used) and Section 3.

In our perspective it is better to discuss the observables after introducing the concept of DPS where we talk about the differences in correlations. Together with Fig.1 we think that the sections flow more naturelly in this way.

114-115 Could this sentence be moved immediately after 100 ?

We have added a description for ECAL,HCAL and HF and put the scentence with ref. [35] right after it, but before the discussion on the global event reconstruction algorithm.

119 DGLAP: this acronym (though standard in QCD) is not defined;
would be a reference or definition suitable ?

The approptiate references have been included.

127-141 Here you refer to PYHTHIA8, PYHTHIA8.240 and PYTHIA8.301...
we find this somehow confusing.

Pythia 8 is the main program, however, we use two different versions as PYTHIA 8.240 is the standard used in CMS, while only from PYTHIA 8.301 and newer versions allow for it to be interfaced with VINCIA.

152: “A second group of models” is too generic. Section 4.1
describes pure PS MC, while this section deals with ME+PS generators.
It would be useful to clarify for the reader how they are going to be used:
are they alternative models for the same SPS template? For different regions?
What is the aim of introducing such very different models?
We do not t expect in general that PS MC will describe correctly SPS 4
jets events.

"A second group of models uses higher..." has been altered to "A second group of models, hence on forth refered to as the multijet models, uses higher ..."

We include many models as different types of PS, the order of the calculation, the MEs used in the calculation, ... all have influence on the correlations between the four final state jets. Most studies only included a couple of models, however, in comparing the many different models we are able to learn much more about what works and what works not in this complex topology.

From 187 till the end of the section: can you elaborate a bit more on
the difference between on- and off-shell incoming partons? Why have off-shell
partons to be showered and hadronized with CASCADE and TMD PDFs? What are we
expected to learn from this alternative approach?

Off-shell partons can not be handled by the standard PDFs or MC event generators, CASCADE in combination with TMDs need to be used, this is general knowledge, ellaborating on this seems a bit redundant to us.

However, we did add a scentence to clarify the difference between on- and off-shell as to motivate the use of KaTie, we added "... allowing for on-shell and off-shell production. In the case of the latter, the initial partons are generated with a non-zero intrinsic $k_\mathrm{T}$ which can alter the momentum balance of the jets, yielding different topologies and correlations between the jets compared to on-shell production. A $2\to 4$ matrix element ..."

What is the size of these non-perturbative corrections?

The non-perturbative corrections range generally between 1-4%, only for the DeltaS observable they range from 1-11% with 11% in the first bin (the most left bin). They are now specified in the text, which reads "The nonperturbative corrections range from 1--4\% for all observables, except for the \DS observable for which corrections up to 11\% were found. "

224 "In this last case a correction as a function of the jet pT is
A correction to what?!?

"... as function of the jet pT is applied to the weight of the selected event." should clarify the statement.

230-231 'to meet PT cuts' We guess you mean that PT must exceed the cut
value; in this case it is better to write it esplicitely

"to meet the PT cuts" has been replaced with "to exceed the PT cuts".

231 'Asymmetric cuts have been chosen over symmetric'
What does (a)symmetric mean in this context?

Symmetrical cuts have the same values on the two leading/sub-leading jets, e.g. 50-50-30-30 GeV cuts. Asymmetric cuts are expected to boost the DPS signature.

267 "Events that meet the online and offline analysis criteria are
stored in two disjunct datasets"
How can 'online and offline' be disjunct?
If this was the case the veto criterion would not be needed,
isn't it?

Events that meet both online and offline analysis cuts for process A are stored in one set, and events that pass all criteria for process B are stored in the second disjunct data set. The scentence has been altered to "Events that meet the online and offline analysis criteria for the processes A and B are stored in two disjunct datasets, respectively." in order to clarify for any possible confusion.

Eq (4): should not the DPS contribution to the total cross-section in the data
be the integral of the first term on the right hand side of eq (3)?

Both methods are completely equivalent with only small differences in their statistical and systematical errors, we opted to use this method as the errors were slightly more stable compared to the alternative.

Sections 8 and 9: there are a lot of results but it is really difficult
to understand the strategy and to draw any conclusion out of them.
Besides the fitted values in tables 2-5 is there any conclusion about
which models fits the data better? What should the reader conclude
looking at figure 14? The spread of the results seems to be much larger
than the uncertainty on previous results.

From the tables it should become clear that collinear factorization, especially in the Pythia and Herwig case fails at low pT, improvements are found when mixing matrix elements or including NLO corrections, while the off-shell samples show the most promise of all samples. A clear need for a DPS contribution is also observed, however, it seems to be model dependent (fig 14). Indicating that either sigma_eff is rather a model dependent parameter due to the interplay between the ME and the PS and/or that the current models need to be improved (e.g. take correlations between different partons into account) in order to obtain a proper description of DPS events in the four jet topology.

358 "DPS pocket formula"; 'pocket' is jargon and never used
before in the paper; we understood this refers to Eq. (1); refer to that
explicitly. See also previous comment about Eq.(1).

Has been changed accordingly.

593-594 These results demonstrate the need for further development of
models in order to accurately describe final states with multiple jets and
the kinematic correlations between the jets.
Should this strong statement be included directly in the abstract ?

It is implicitly included in the abstract. The last scentence reads "... and of their dependence of the models used for the single-parton scattering background" which is also the main message of the paper.

Fig.2-3-4-5-6-7-8-9 Plots are quite confusing;
in each of them the two (three) bottom panels with the ratios show a total
of 7 (9) models, while the top panel shows only 4 models. Why? We
understand the plots would get too busy otherwise. But, if this is the case,
you need to have more plots. We do not see the point of having ratios for
profiles not shown.

The models shown in the Pythia and Herwig plots are the ones for the most recent tunes, while in the multijet case we have opted to use on of each of the four model groups (KaTie on-shell, KaTie off-shell, MadGraph leading order and Powheg). The ratios for profiles that we do not show allow for an indirect comparement which do yield important conclusions, eg the dependence on the PS of the phi_ij observable for instance. We would argue to keep the plots this way and if needed we could supply the additional curves as supplementary material as adding all curves would extend the paper with another 5-8 pages.

Type A.

2-3 provides a good de-
scription of jets with large transverse momentum (p T ) produced in
high-energy proton-proton (pp) collisions.
--> provides a good de-
scription of production of jets with large transverse momentum (p T )
in high-energy proton-proton (pp) collisions.

Changed accordingly.

11-12 However, final states with multiple jets, and their
kinematic correlations, call for further studies of the strong interaction.
--> However, final states with multiple jets, and their
kinematic correlations, require and suggest further studies of the strong
interaction properties.

Changed accordingly.

14-15 Radi-
ations before and/or after the interaction
--> Radi-
ation before and/or after the interaction

Changed accordingly.

23-24 one --> 1
two --> 2

Changed accordingly.

29 are more often than not produced
--> are often produced

Changed accordingly.

31 smaller values of the proton’s momentum fraction
--> smaller values of the proton momentum fraction

Changed accordingly

32 and accordingly a larger
--> and, accordingly, a larger

Changed accordingly.

33 extend --> extent

Changed accordingly.

141 and l149; in the former LO is uppercase; in the latter nnlo is
lowercase. See also 165, 175.

We have replaced nnlo with NNLO throughout the paper.

172-173 built is used twice

First "built" is replaced by "constructed".

188-189 Two on-shell and two-off
shell samples are produced.

--> Two on-shell and two off
-shell samples are produced.

Changed accordingly.

193 For two the off-shell samples
--> For the two off-shell samples

Changed accordingly.

221-222 "The online selection of events was based on four single-jet
triggers requiring the presence of at least one jet with a pT above 30, 50,
80, or 100 GeV " ==> "The online selection of events was based on four
single-jet triggers each requiring the presence of at least one jet with a
pT above 30, 50, 80, or 100 GeV respectively"

Changed accordingly.

277 the difference of the cross sections
--> the difference between the cross sections

Changed accordingly.

314 can not -->cannot

Changed accordingly.

341 a possible difference with the true efficiency
--> a possible difference with respect to the true efficiency

Changed accordingly.

Table 1 caption
between generator and detector level is estimated at 2%
--> between generator and detector level is estimated to be 2%

Changed accordingly.

In figure captions we see two statements about the way error bars
are displayed.
'The error bars represent the statistical
uncertainty, while the yellow band indicates the total (statistical+
systematic) uncertainty on the measurement.'
'The error bars and
bands are shown similarly to Fig. 2.'.
A uniform statement would be welcome...

We have been advised to use the reference to one of the earlier figures in order to shorten the captions for claritys sake.

428 and models, even the on shell K A T IE models,
--> and models, even the on-shell K A T IE models,

Changed accordingly.

465 the ∆S observables exhibits
--> the ∆S observables exhibit

Changed to "The DeltaS observable exhibits", only one DeltaS observable exists/is defined.

499 The mixed event sample obtained from data as described in Sec. 6
is used to obtain
--> The mixed event sample obtained from data as described in Sec. 6
is used to get

Changed accordingly.

522 The DPS fractions obtained for the other models ranges
--> The DPS fractions obtained for the other models range

Changed accordingly.

Figure 15 caption
systematical--> systematic

Changed accordingly.

Tab.5 In label of column #5 "(.syst.)" ==> "(syst.)"

Changed accordingly.

561 four jet
--> four-jet events ?

Changed accordingly.

Also in some other parts..

Has been changed where appropropriate.

Figure 15 caption
In the ratio of the fitted MC model
and of the total fitted result over the data are show in the bottom plot.
--> The ratio of the fitted MC model
and of the total fitted result over the data are shown in the bottom plot.

Changed accordingly.

586 A strong dependence of the extracted values of σ eff on the model used
to the describe the SPS
--> A strong dependence of the extracted values of σ eff on the model used
to describe the SPS

Changed accordingly.

All comments from Louise Skinnari

Type B (physics) comments

- It is not made clear why four different trigger thresholds are used. It would seem simpler to just use the 30 GeV trigger since it should include everything that is triggered by the higher thresholds as well. Are they lower pT triggers prescaled, or do they have tighter requirement in some other variable?

We clarified that the triggers are prescaled, the following scentence was added "As the triggers have been prescaled, they are used in disjunct p T ranges."

- It is not clear what is the motivation behind the inclusion of KATIE predictions. Perhaps the authors could provide additional information on why to include KATIE.

We extended the introduction of KaTie by including a scentence on the difference between oon-shell and off-shell production, it now reads "K A T IE version 23April2019 [64] is a LO parton level event generator, based on k T -factorization [65–67], allowing for on-shell and off-shell production. In the case ofthe latter, the initial partons are generated with a non-zero intrinsic k T which can al-
ter the momentum balance of the jets, yielding different topologies and correlations between the jets compared to on-shell production. ..."

- Line 11: missing citation to the referred QCD predictions?

The citation [1-3] is meant for this part, we moved the references to the end of the scentence "Measurements of the cross section for the production of inclusive high-p T jets have been performed by the CMS collaboration at various center-of-mass energies and show good agreement with perturbative QCD predictions at next-to-leading order (NLO) accuracy [1–3]."

Effective cross-section:
- Line 24: effective cross-section is not defined in the text. Perhaps a reference to some external paper with the relevant theoretical definitions would be sufficient. In Line 282 the variable DeltaS is used to extract Sigma_eff. This is not entirely clear to the reader. Perhaps the authors could provide a clear description of the definition of these quantities and their extraction.

Citation to an more extensive paper on DPS and the pocket formula is included.

We also added/ changed the explenation on the template method in the introduction to be more clear, so that line 282 should be better understood. In the itnroduction the text reads "... a pure DPS signal sample is reconstructed from data by randomly mixing two inclusive single-jet events into an inclusive DPS four-jet event, whereafter it is fitted along with several SPS-only background MC models to the distributions obtained from inclusive four-jet data." We do not yet mention DeltaS here yet but it should be clear from the text that an observable is used in the template method.

- L549-552: it is not clear why the same effective cross-section should be expected at a p-pbar vs pp collider, or at different center-of-mass energies?

The effective cross section is expected to lower with an increasing COM energy (more DPS therefore a lower sigma_eff), however, it is indicatve to compare measurements at different COM energies to see if there is an actual rise or not.

It has been shown that in the inclusive case, the effective cross section is expected to be process independent (, we will add a scentence on this in what was line 556 where we introduce the same sign WW and J/Psi measurements, it now reads "It has been shown that \sigmaeff is expected to be process independent for inclusive final states \cite{Seymour:2013sya}, therefore, it is noteworthy that the result from the same-sign WW analysis shows agreement with the results from the \MADGRAPH{}5 and NLO samples. The \cPJgy pair production measurement only shows agreement with the NLO models."

- Line 26: Is it not clear what the total inelastic cross-section refers to. Please rephrase for more clarity.

Changed to "total inelastic cross section of proton-proton interactions".

- Line 50-51: Data are corrected for detector effects by means of an unfolding procedure. Comment: It could be useful to elaborate on the unfolding process a bit or at least say "corrected for detector efficiency and resolution"

We changed the scentence to "Data are corrected for detector efficiency and resolution effects by means of an unfolding procedure.".

- L101-113: this seems to be a generic CMS description, which is not appropriate given that dedicated uncertainties have to be derived (systematics section) to account for the lower-pt jets. It would seem more appropriate to remove those generic statements from here (or if not, minimally comment on if these numbers are valid down to 20 GeV as probed in the analysis).

We have used the generic text, but we altered the numbers so that they do correspond the real uncertainties we derived, however, instead of giving the uncertainty for 30 GeV, we updated the text to go down to 20 GeV.

- L119: define or cite DGLAB

Added the appropriate references.

- Line 142-150: could you mention what ME provider did you use within the HERWIG matchbox?

We use the standard 2->2 LO ME which is already specified in what was line 118-119 of v17 of the paper, it reads "The PYTHIA 8 [36, 37], HERWIG ++ [38], and HERWIG 7 [39] MC event generators use 2 → 2 lead-
ing order (LO) matrix elements"

- Line 139, 165: It is surprising to use such an old PDF set (NNPDF 2.3) for Pythia8, is VINCIA with NNPDF3.1 not available?

It is not.

- Line 238: You state that TUnfold uses SVD, it is based on a least squares fit, as stated in the abstract of the TUnfold paper ( "TUnfold is a tool for correcting migration and background effects in high energy physics for multi-dimensional distributions. It is based on a least square fit with Tikhonov regularisation and an optional area constraint."

Yes, but at the heart lies the SVD decomposition, which is detailed in the paper you have linked on the mathematical formulation of the TUnfold algorithm. "The TUnfold algorithm [3] presented here compares best to algorithms based on matrix inversion or singular value decomposition, like [6, 10].", taken from the introduction of the paper.

- Line 267: disjunct -> disjoint? Does this mean all events are in both samples, and then random events from each sample are combined? Are all combinations of all events used for the final combined sample, or is each event used only once per sample. Please explain more fully how the events are combined.

Disjunct is proper english and indeed a synonym for disjoint.

All events that are in A are per definition in B, meaning that B is the larger of the two samples. In order to avoid any possible correlations, each events in A is randomly matched with one event in B, yielding sufficient statistics. We added the scentence "Each event is only used once in the combination procedure."

- L309-318: Dedicated JES scale factors had to be derived, is it correct that this is not needed for JER?

Yes, that is correct.

- L356: It is not clear what process A vs process B is here?

The processes are the inclusive single jet cross sections with pT cuts of 50 and 30 GeV for A and B respecively, as defined on page 7.

- Line 374-380: Could you be clear about when this normalization is occurring and over which bins it is happening? (With TUnfold it is necessary (?) to unfold before you normalize.)

We changed the text to "In order to make qualitative statements about the shape, the fully corrected distributions have been normalized to one or more bins where a much reduced DPS contribution is expected.", where "fully corrected" should be a clear indication that the normalization is performed after the unfolding procedure, ie it is performed on the generator level distributions.

Type A (editorial) comments:

L6: the probability to find a parton -> the probability to find a given parton

Changed accordingly.

L32: of the gluon density -> in gluon density

Changed accordingly.

L33: "extend" -> "extent"

Changed accordingly.

L36: published in [4-8]. -> published [4-8].

Changed accordingly.

L39: suggest “older” -> “earlier”

Changed accordingly.

L45: Suggest to combine this with L49-50 where the luminosity of this data is provided

Changed accordingly.

L47-48: "This allows not only to mitigate effects of such pileup but also to each down to low jet pT..." -> "This allows not only the mitigation of effects of such pileup but also access to low jet pT..."

Changed accordingly.

L70-72: split sentence into two parts. "The four leading jets are ordered with decreasing p_T. Based on the ...."

Changed accordingly.

L89-91: Perhaps there is a more mathematical word than "flatter" but either way it could be useful to switch the order, first describing the distribution peaking and then saying the other is "flatter".

Switched the order and used "less correlated"

L95: If you are going to refer to the pixel and strip as a single tracker then I think you should say " is a " rather than "are a".

The text is taken from the CMS pubcom twiki.

L111-113: The order should really be switched so that immediately following the jet energy scale you mention the jet energy resolution rather than discussing this additional selection criteria in the middle.

The text is taken from the CMS pubcom twiki.

L120: pT-ordered shower -> pT-ordered parton shower

Changed accordingly.

L121: suggest to add at the end of this sentence “(jointly referred to as HERWIG)”

Changed accordingly.

L133: nnlo not defined (furthermore, looks odd with the different usage “nnlo” vs “LO” throughout)

Defined NNLO and capitalized throughout the text.

L177: factorization and renormalization scale -> factorization and renormalization scales

Changed accordingly.

L193: “For two the…” => “For the two…” ?

Changed accordingly.

L214: "can not" -> "cannot"

Changed accordingly.

L224: Add a comma after "In this last case"

Changed accordingly

L246: "allows to calculate" -> "allows the calculation of"

Changed accordingly

Table 1: (i) what is “upper” vs “lower” here? (ii) add for each of the three parts of the table “(%)” to clarify that these are percentages.

Changed accordingly and explicitly stated what upper and lower are.

L358: what is "pocket"? Is this formula the same as eqn 1? (appears to be jargon, suggest to rephrase)

Changed to eq.(1).

L419: Add a comma after "In this case"

Changed accordingly.

L448: write out w.r.t

Changed accordingly.

L453: section => Sec.

Changed accordingly.

L461: “Fig. 10 and Fig. 11” => “Figures 10 and 11” (at start of sentence, should be written out)

Changed accordingly.

L513: “in 15” => do you mean “in Fig. 15” ?

Changed accordingly.

L516, 518, 520: must clarify what the different uncertainties are (stat vs syst)

Changed accordingly.

Figure 13: (i) for consistency should use “KT” rather than “K” in figure text. (ii) in caption add “(P8)” and “(KT)” similarly as for other figures.

Changed accordingly.

L553: “sqrt(s) equal to” => “of”

Changed accordingly.

Table 5: in the last column “(.syst.)” => “(syst.)” (i.e. there’s a random dot before syst)

Changed accordingly.

L561: "four jet" -> "four jets"

Changed accordingly.

L563: remove “sqrt(s) =”

Changed accordingly.

All comments from Greg Landsberg


- General: while CMS Style does recommend hyphenating double-parton scattering, it is not really correct. This becomes obvious when you use a hyphen in "single-parton scattering" (as you do), which simply makes no sense! It's not the parton that is single, but the scattering! (Any scattering requires two partons, so "single-parton scattering" is simply nonsense.) For consistency, you should not hyphenate double parton scattering either, as "double" modifies "scattering", not "parton". This affects the title, multiple lines in the abstract, and several occurrences in the main body of the text.

Removed the hyphenation throughout the text.

- Eq. (1): is there a need to use the combinatorial factor that is an integer divided by 2 (m/2 )? Why not simply call t C and define it as 1/2 for identical, and 1 for distinguishable processes?

Historically it is defined in this way, all older papers on DPS use the same definition.

- LL30-31: while the statement about the DPS cross section is correct, the SPS cross section at fixed pT
also rises with the c.o.m. energy. I guess what you want to say is that the DPS rises faster than the logarithmically growing SPS. Please, rephrase the sentence to specify exactly what you mean.

Altered accordingly, the text now reads "The DPS cross section rises faster compared to the SPS cross section with increasing center-of-mass energy at fixed \PT of the jets:"

- L37: vector bosons are no final-state particles; please rephrase as "Signatures involving one or two vector bosons ..."

Altered accordingly.

- L112: give a reference to the noise filtering algorithm [an old CMS PAS].

This is the recommended description provided by the CMS pubcom.

- L113: since you work with low-pT jets, the resolution at 1 TeV is all but irrelevant. Suggest saying: "The jet energy resolution (JER) ranges from 15-20\% at 30 GeV to 10\% at 100 GeV."

In our data we do encounter jets up to 800-1000 GeV, although its only a few instances we thought to give it just to be complete.

- Section 3: add a standard sentence about the triggers.

Added a standard paragraph on the L1 and HLT trigger systems.

- L127 and multiple places in the text: all the tunes should be spelled in normal font, e.g. CUETP8M1. The small-caps font is reserved for computer codes, which tunes are not! Please, go through the paper and replace all the dozens of occurrences.

Font of all tunes has been changed accordingly.

- L129 and multiple places in the text: similarly, PDFs should be spelled in normal font as well, e.g., NNPDF31\_nnlo - again there are a couple of dozens of occurrences throughout the paper, so please fix them all.

All occurences have been fixed.

- L156 and further in the text: standalone MadGraph5 has not been used in Run 2. The name of the program is \MGvATNLO; please, replace all MadGraph5 [and one occurrence of simple "MadGraph" on L437] with \MGv ATNO through the paper.

We have replaced all instances of MadGraph and MadGraph5 by the \MGvATNLO.

- LL165-166: ... the MLM scheme [54] is used to match jets produced via matrix element calculations with those from parton shower, using the matching pT scale of 18 GeV, which ...

Changed accordingly.

- L177: ... and factorization scales are set to the pT

Changed accordingly.

- LL225-226: ... with that of an unbiased reference trigger.

The scentence has been changed to "The trigger efficiency has been determined by comparing the performance of the jet trigger with a minimum bias trigger serving as an unbiased reference trigger."

- L274: the systematic uncertainty here comes out of the blue; please put a forward reference to the following section, where it's discussed.

The four-jet efficiency has been given its own line and the text now reads:

"... The four-jet efficiency has been determined to be equal to

\epsilon_\mathrm{4j} = 0.324^{+0.037}_{-0.065}{}\mathrm{(syst.)} \label{eq:four-jet-eff}

Where the statistical uncertainty was found to be negligible and the systematic error is detailed in the next section. ..."


Title: suggest dropping "the" on the first line;

Changed accordingly.


LL3-4: for the leading, second-, third-, and fourth-leading jets, respectively.

Changed accordingly.

L4: measured as functions of;

Changed accordingly.


L7: information about the proton;
L21: multiple parton interactions (MPI).
L24: nonidentical [CMS Style];
LL26-27: occurrence of processes A and B.
LL29-30: tow independent pairs, each in the back-to-back configuration.
L38: CMS Collaborations [CMS Style];
L42: CMS Collaboration;
Fig. 1 caption, L5: from an SPS event. [See CMS Style guide on an article before an acronym.]
L46: in 2016, during a special data taking period with a low probability ["run" is jargon!];
L52: studied via the;
LL65-66: of the results, followed by a summary of the paper in Section 9.

All changed accordingly.


General: ΔY and ΔS should have Y and S in italics, as variables. This affects multiple places in the paper, so please use search-and-replace to fix them all.
LL83-84: subprocesss, a decorrelation in the;

All changed accordingly.

The CMS detector:

LL96.97: ECAL, HCAL are never used in the paper, so delete the definition of the acronyms;

We have included a short description of ECAL and HCAL, now the acronyms need to be defined.

L93 The CMS detector and jet reconstruction;
LL111-112: to remove those potentially dominated;

All changed accordingly.

Monte Carlo event generators:

L119: at next-to-leading logarithmic level;
LL130,133,138,141,143,147,150,165,169,175,192,195,196,212: PDFs [45]. [You defined PDFs as plural on L6!];
L148: by the CMS Collaboration;
L152: higher-order matrix elements;
LL178-179: The generator-level minimal pT
requirement for the underlying ["cut" is bad jargon];
L181: parton-level event generator;
L183: The generator-level requirement on the pT;
L184: add a comma before "and";
L185: As the pT requirements are;
L191: with the CP5 and CH3 tunes, respectively.
L201: is adopted, as in Ref. [43]. Add a comma before "as it is determined";
L203: same kinematic requirements;
L215: nonperturbative;

All changed accordingly.

Data analysis:

LL219-220: collected in 2016 during a data taking period at low instantaneous luminosity, with;
LL222-223: Offline requirements are imposed to ensure;
LL229-230: defined by selections on jet pT;
LL230-231: to meet pT thresholds of 35,30,25, and 20 GeV. Asymmetric thresholds are chosen;
L233: higher-order calculations;
L235: set of selections is needed;

All changed accordingly.

Extraction of the effective cross section:

L252: laid out in Ref. [22].
L260: The lowest pT single-jet trigger [superlative compound modifiers are not hyphenated];
L274: add a space before "(syst)" and drop the period after "syst" [CMS Style];
L280: σ\rmA and σB [subscripts in Roman];
Eqs. (3-4): don't capitalize "data" in the superscripts;

All changed accordingly.

Systematic Uncertainties:

L295: Systematic uncertainties;
L299: dedicated JES calibration;
L300: in Ref. [34];
L305: the dominant contribution;
L307: of 39 (33)\%.
L309: Jet energy resolution uncertainty [JER is now defined earlier];
L337: Integrated luminosity uncertainty: The uncertainty in the integrated luminosity for data;
LL342-343: \PYTHIA 8 and \HERWIG ++ at both the detector and generator levels, after;
Table 1 caption, L2 asymmetric uncertainties, while;
LL2-3: put commas around "as well as the statistical uncertainty"
L4: between the generator- and detector-level selections is estimated to be 2\%.
Table 2 body, header line: since CMS Style prohibits the use of vertical dividers in the tables, please remove those and center "JES" and "Total" in between the "Upper" and "Lower" sub-headings. In the headings separating three parts of the table, capitalize just the first word, but not the subsequent ones. All the ranges in the table should be typeset with an en-dash, not a hyphen, e.g., 11--39. In the first column, the first four entries should be simply pT,1, etc., per earlier definition.

All changed accordingly.


L350: two phase space regions defined;
L352: leading jet pseudorapidity [compound modifiers of the type gerund-noun do not require hyphenation, as there is no ambiguity];
Eqs. (5-8): drop periods after "stat", "syst" [CMS Style];
L355: in Section 4;
L357: leading jet pseudorapidity;
L366: larger than the one observed in data.
L370: low-pT jets;
Table 2 body: remove all the vertical dividers;
Fig. 2-3 captions, L2: (upper left); (upper right); LL2-3: (lower left); (lower right); L4: uncertainty in the;
L382: uncertainty in the;
L385: and the MC-to-data ratio;
L389: decorrelated;
L400: Multijet models;
L411: larger than those from data;
Fig. 4 caption, LL4-5: uncertainty in the measurement.
Table 3 body: remove all the vertical dividers;
L443: (as shown, e.g., in the \MGvATNLO ... [don't tell a reader what to do!]
L461: Figures 10 and 11 show; add a comma before "respectively";
Figs. 6-7 captions, L2: add a comma before "and"; (upper left); (upper right); L3: (lower left); (lower right);
Table 4 body: remove all the vertical dividers;
L463: low-pT region;
Fig. 10-11 captions, L2: (upper left) (upper right); LL2-3: (lower left); (lower right) jet.
L472: add a comma before "and";
L476: (shown in Figs. 8 and 9);
L478: (taken from Ref. [43]);
LL481-482: pT-ordered parton shower show a too large decorrelation.
L490: Section 6.
L499: in Section 6 is;
L506: karger decorrelation;
L512: Table 5 and Fig. 14.
L513: in Fig. 15.
L536: ATLAS Collaboration;
L555: in Ref. [12]. Add spaces before the two opening parentheses and delete periods after "stat", "syst"; from the J/ψ meson pair;
L556: in Ref. [17].
L569: While the J$/\psi meson pair production;
Table 5 body: remove all the vertical dividers;

All changed accordingly.

L359: drop the period after "syst";

The four-jet efficiency has now itsequation-line in the text, we refere to the equation instead of giving the value again here.


L562: from proton-proton collisions;

Changed accordingly.


Add a standard acknowledgment section, appropriate for long papers.

Added the acknowledgment text for long papers.


Ref. [13]: typeset the addendum as a doi reference.
Ref. [15]: capitalize "Tevatron" in the title.
Ref. [18]: Phys. Lett. B;
Refs. [18,40,41,46,68]: delete "" from the doi reference.
Refs .[36,37]: replace the two reference to obsolete PYTHIA versions with a single standard PYTHIA 8.2 reference.
Ref. [39]: delete "Herwig is available from ...".
Ref. [40]: Phys. Rep. {\bf 97} (1983) 31.
Ref. [41]: Phys. Lett. B {\bf 83} (1979) 87.
Ref. [46]: Nucl. Instrumentavam. Meth. A {\bf 506} (2003) 250.
Ref. [51]: replace the PAS with the submitted arXiv version.
Ref. [68]: Nucl. Instrum. Meth. A {\bf 372} (1996) 469.

All changed accordingly.

Ref. [16]: the second doi reference must be an Erratum; please mention and typeset as such.

We assumne that you meant Ref.[17], we put the second doi refernce as an erratum here.

All comments from Sudeshna Banerjee (Statistics Committee)

L203 - In this case how do you take care of the scaling of the cross section ?

The scaling is handled by PYTHIA 8 internally. The effective cross section is calculated from the non-diffractive cross section sigma_ND = 55.51 mb and the impact parameter (which is in turn determined by tune specific parameters), ie sigma_eff = sigma_ND / <f_impact>. More details are found here

L222 - How do you take care of same event passing two or more different triggers ?

The triggers are used in succession, i.e. in seperate/disjunct regions of the phase space. (in different pT intervals) We have altered the text to be more clear on the subject, the paper now reads "... The online selection of events was based on four single-jet triggers each requiring the presence of at least one jet with a \PT above 30, 50, 80, or 100 GeV, and within $|\eta| < 4.7$. As the triggers have been prescaled, they are used in disjunct \PT ranges. ..."

L274 - The word "found" should be replaced by "estimated" or something similar.

The text reads "... The four-jet efficiency has been estimated to be equal to ... \begin{align} 4-jet efficiencty here \end{align} ..."

L322 - How is this fit performed ? How are the uncertainties on fit parameters determined ?

We have altered the text to "... These weights are obtained by fitting the trigger efficiency curve determined in data using a lest-squares minimization. ..."

Table 1 caption - How is the uncertainty due possible differences between generator and detector
level determined ? Has this been discussed in the systematic section ?

It is discussed on lines 339-348 and in more detail in the AN.

L358 - DPS pocket formula is described by equation 2 ? It is better to reference the equation
number here.

Reference to an equation has been used now isntead.

L367 - "...much larger..." which two values are being compared here ? If you mean MC versus data,
it will be good to mention it here.

The text now reads "... show that the much larger cross section of the MC models compared to the data is due to an abundance of low-\PT jets ..."

L389, L482 - How do you conclude about the de-correlation ? It should be clarified further.

The curves for the Pythia 8 2->2 models are significantly flatter compared to the Herwig models when comparing them to the shape of the data. We think that it is clear from the text as we explicitly mention that we perform a bin-normalization in order to compare the shapes of the different models in a more qualitative way.

Paper Review: draft v11 -- Approval presentation (Oct 20, 2020)

Color code:

Green: comment accepted and implemented

Blue: comment accepted, but text has been modified in a different way than proposed

Orange: response given, but no change has been implemented

All comments (Oct 20, 2020)

Paolo: Mention about the additional and supplementary plots. Supplementary plots don't need to be approved today.
A: All pt and eta spectra (shown in Sl. from 38 to 45) will be in the Appendix.

Q s30 Marco What do you mean with KT Onshell, is it without DPS?
A No all those models have DPS. On Sl. 25 and 26, the samples without DPS are multi jet samples.
Q On s30 do you conclude that there is not much DPS
A just a few %, but enough o measure it, the shape differences are important.

Q Marco s33 what do you conclude ? What is the measurement ?
Numbers don't agree. NLO-MC should be better
A Yes the conclusion is that sigma_eff is quite sensitive to the model.
Q Can we quote one single number / average with an overall uncertainty ?
Maybe NLO average with some syst could be one ?
There is no number or conclusion in the final paperIf you think you can conclude it would be good.
You give range of values. But not conclusive.
A Conclusions agreement towards lower values NLO, but still large model dependence
A Klaus (ARC chair) not so easy to give a single answer.
If you switch from LO to NLO prediction you need to full tune LO and full tune NLO.
Q Marco CP5 is a LO tune. Add caveat that the tuning is not fully consistent.

We decided not to take an average of all models as that would increase the error drastically and it would not show the clear model dependence which we think to be a central message of the paper. Rather we would like to keep all values in as they are.

Q Marco quoted uncertainties include full syst
A Yes Pierre comments that Model dependence on the unfolding is included,
but not on model itself used to extract sigmaeff.

Q: Paolo For model dependency two models are considered?
A: yes CUET tunes. The model unc. is not the dominant contributor to the x-section, see Sl.14.
JES is the dominant unc. source.

Q: Paolo How merging rate with vertex resolution is estimated? (1mm seems to be large )
A: Vertex resolution estimated from data then z-distribution of vertices used for merging rate.

Q Style comments by Guillelmo
- CMS/lumi/energy text shouldn't touch the frame and be consistent (Figs. 2, 3, 4, 5, 6, 7, 8, 11, 12, 14, 15, 16, 17)
-replace (XXX) by [XXX] both X and Y titles (Figs. 2, 4, 5, 6, 7, 8, 9, 11, 12, 14, 16)
- units should be visible, it is not on the right side of several of them or touching X titles (Figs. 2, 6, 14, 16)
- some of the legends are too large (Figs. 2, 3, 4, 5, 6, 14)

Updated all figures which will be added in the next paper draft.

- Don't we need references for Fig. 10?

References added in the caption of Fig. 10.

Additional comments:

Q Discussion on presentation of results sigmaeff
Pierre: Sl.33 It would be good to get to have opinion of group, whether to produce a combined value here.
Klaus: Given the uncertainties, don’t find the averaging NLO and LO as a good idea. you cannot have a unique CMS number, it’s not trivial to average these numbers.

Decision: Follow up offline.

Q AndreaR General comment. It’s hard to understand and to follow in the paper
where are the DPS contributions. Legends and captions can be made more clear.
ARC Chair Report by Klaus
* Preaproval was about three months ago
* A large amount of results and info is produced and presented
* Paper was reorganised and updated couple of times after pre-app.Authors very well responsive.
* Plotting style also improved a lot
* Most difficult part is to conclude from many MC models and predictions.
* If there are suggestions to improve the presentation of results, please do so.
* Agree that results can be presented in a better way for non-experts.

Paper Review: draft v9 -- ARC review (Sep 9, 2020)

Color code:

Green: comment accepted and implemented

Blue: comment accepted, but text has been modified in a different way than proposed

Orange: response given, but no change has been implemented

All ARC comments (Sep 9, 2020)

Comments to physics content: ========================

1. Introduction:

2nd paragraph: "An alternative approach" -> "A second approach" In the end SPS with higher-order pQCD AND DPS both contribute to multijet final states. So these two are not alternatives like one or the other. BTW the line numbering is missing here.

Text has been changed to "a second approach".

l 20: rises with increasing cms energy for fixed pT?

Yes at fixed pT, text has been altered accordingly.

l 46: Data events contain everything including potential DPS effects already. So how can mixed data events provide you with a "pure DPS" signal sample? Did you compare such mixed data events against mixed MC SPS events? You have to better explain why you can do this and that DPS already in data does not bias your templates.

Added more information to the text, along with a reference to a four jet DPS measurement performed by ATLAS where a similar approach has been used in order to construct a signal template.
We made a comparison of the mixed data on detector level with mixed MC samples (Pythia8 and Herwig++ interfaced with the CUET tunes) in the AN in Sec. 8.3. Such plots at generator level can be provided as well, although the general trend will be the same as at detector level.
The fraction of DPS events in the inclusive single jet event subsets is expected to be of percent level, therefore a possible bias will not have a significant impact. Additionally, we opted to use a data-driven method for the construction of the signal template. The alternative is to construct the signal template in a MC driven method which introduces an inevittable MC dependence. The approach has been used for other processes for which the calculations are more precise as for jet production, eg DPS in same sign WW pair boson production and DPS in W+jets.

2. Observables:

Figures 2-4: We have the impression that there are way too many plots already and find these ones the least important. We'll see later on anyway how these distributions look like at multiple occasions. We suggest to drop them. This would also avoid to introduce the KaTie model here with a forward reference to section 4.

We have moved the predictions from KaTie of the observables (the six DPS-sensitive observables, not the pT and eta spectra) to the appendices and put the proper reference there. We believe that it is important to show these plots for clarity and for a proper understanding of all observables. It is not only the shape of the distributions that is important here but the ratioplots that show the DPS sensitive areas. The lines 85-93 have been moved to the discussion on the data analysis, ie the paragraph on the event selection. The two phase space regions are now defined there, with the additional information on the asymmetric cuts

l 58: We advocate to use $region~I$ and $region~II$ whenever referring to the two phase space regions. Subscripts are quite unusual in this context and at first look like typos. For quantities in formulae or such subscripts are ok.

Adapted the $region~I$ and $region~II$ notations.

Moreover, the two regions here come out of the blue. Why using region II at all when it dampens (l.90) the DPS contribution? Maybe one could add here something that is explained only later: "To extract sigma_eff from Delta S ... for reasons of trigger efficiency a second set of pT thresholds is used defining region II."


"Due to the \PT thresholds of the available triggers, the second set of cuts is necessary in order to perform the extraction of the \sigmaeff from the \DS distribution, see Sec.~\ref{sec:extraction}."

4. MC event generators:

It is very good to see that the authors try to explain in detail which MC samples have been used and what characterises them. However, it could be presented in a better way. Maybe you could consider a table?

We have looked at the possibility of a table, however, there are to many varying parameters for the different models that in addition to the table additional specifications have to be made for different models.

At least, if numbers and letters instead of bullets would be used, some references would become easier and some duplication could be avoided. For example, labelling lines 161/2 as "LO I.b)" one could refer to this in lines 190-192 and 197-198 without the need to repeat the same description again.

In earlier stages of the paper (before the arc review) we used a structure with more bullets/numbers which we were advised against. We also think that even though the models are repeated often, it is more clear in this way and will allow for an easier reading of the paper. A lot of text is between the introduction of the models and the discussion of the results.

line 434 in 8.2 Multijet models and later "mixed MadGraph5 " is mentioned a lot, but it is never really defined! Also not in 4.2 and we do not aprreciate much the term "mixed".

The "mixed" has been dropped.

line 189: It should read "scalar sum", not "squared sum", right?

Yes, the error has been corrected.

ll 213-214: What does the phrase in () mean? Unclear, please explain.

The 35-30-25-20 GeV cuts have been placed on parton level, after hadronization the corresponding jets generally obtain a pT that is 5-10 GeV lower compared to their corresponding parton.
We rewrote the scentence more explicit:
"as the cuts are introduced at parton level, the effective thresholds on the resulting hadron level jets are typically 5 to 10 GeV lower"

l 237: It is unclear what "same parameters as before" refers to.

It is generated in the exact same manner as the multijet KaTie sample, ie same generation cuts, renormalization and factorization scales, ...
The text has been altered to:
"..., with the exact same generation parameters as the on-shell \KATIE LO sample from the multijet models."

5. Data analysis:

l 262 "to enable the extraction of sigma_eff" For the 2nd set of pT cuts, see also remark above, please clarify more elaborately the role of the two set of pT cuts and their connection to the inclusive single jet cut regions.

Added the following sentence to connect everything more:

"Inclusive single jet events will be used as the two independent processes A and B, of which the lowest possible \PT threshold is limited by the available triggers to 30~GeV."

ll 271ff: In the last sentences it is unclear what the various "rates" refer to. Please improve.

Changed to:
"..., as well as the probabilities of the occurence of migrations into and out of the phase space for all observables."
Is further explenation about these migrations into and out of the phase space needed? Can be extended to errors the reconstruction, ... if needed.

6. Extraction of sigma_eff:

ll 277ff: -> with "at least" 4 jets Here, for the first time you go beyond the DPS picture from Fig. 1 by including the cases, where the "inclusive jets" from PS #1 contribute 3 jets to your 4-jet selection and the "inclusive jets" from PS #2 only 1 (or vice versa). Please explain more elaborately. Otherwise it will be difficult to follow or to understand eqs. 10 & 11.

Extended the explenation beyond the ideal case where both A and B are dijet processes.

linie after 281; --> "lowest pT single jet trigger"

Added the missing pT.

l 301: "least affected by parton shower effects": Maybe bring this argument earlier to motivate Delta S?

Added a similar argument at the end of the definition of DeltaS:
"As the information of all four leading jets enters the defeinition of the DeltaS observable, it is expected to display the most resilient behavior to effects of the parton shower."

7. --> "Systematic effects" or "... uncertainties"

l 341 and Figs. 17/8 in the AN: Are you sure the uncertainties on the trigger efficiency curves are correct? Are they Wilson or Clopper-Pearson intervals or at least "efficiency" uncertainties? If not please correct and check impact on turn-on points if any.

We are certain that the trigger efficiencies are correct, due to the high prescales of the trigger statistical fluctuations in the trigger refernce method can result in bin above 1. for the triggers with the lowest pT thresholds a prescale of ~100 and larger are used and only a handfull of jets are selected at values larger than 50 GeV for the dijet triggers. The error bars on the bins still overlap with unity in all cases.

l 343ff: "by the models" or "As mentioned previously" is too unclear. Please be more specific!

Models used for the unfolding have been specified here for clarity.

l 361: "reweighting" to what exactly? Has second MC event generator been used to extract the efficiency? This is also a model dependence, no?

Samples are rewighted to the jet multiplicity and the leading jet pT spectrum, now specified in the text.
The same check has been done for the Herwig++ sample and the values of 0.4027 and 0.3921 at generator and detector level resp. were found.
As the differences were of the same order, we overestimated the uncertainty with two percent. To be completely formal we could use the average difference between detector and generator level of the two models.

l 368; "as they are in all calculations." ?? What does this first sentence want to state? Unclear.

It was a misconstrued scentence and was meant to indicate that the correlations are taken into account in all calculations of the cross sections, is now removed as it should be clear from the first part of the scentence.

Table 1 caption: Why is the 4jet eff. "defined" on detector level here? Why do you evaluate an uncertainty of 2% from the difference to Gen Level before, which is not even quoted here? What about the model dependence e.g. when using Herwig?

The 4-jet efficiency is on detector level as it is obtained from the data mixing before they are fully corrected.
The 2% uncertainty is added into the caption, along with the statement that it is included in the calculation of the total uncertainty on the 4-jet efficiency.
The same check has been done for the Herwig++ sample and the values of 0.4027 and 0.3921 at generator and detector level resp. were found. Therefore, we decided to work with a more conservative number of 2%. However I have added the numbers from Herwig to the text.
We also did not iclude this number in the table as here the model uncertainty is due to the unfolding of the observables and the 4-jet efficiency is not unfolded. We tought it would be more cofnusing when putting the number there.

8. Results:

How is result (14) obtained here? Trigger efficiency plays a role here in region I. Please explain.

Added that the cross sections are obtained by integrating the measured differential cross sections in function of the pseudorapidity spectrum of the leading jet and the DeltaS observable for the region I and region II respectively.

l 383/4 "in exactly the ... as above": The reference is unclear. Be more specific please.

Has been replaced by: "The measured differential cross sections have been integrated in order to obtain the inclusive single jet cross sections."

l 410 They are normalised to which bin precisely? Please be more specific.

Specified the bin: "... are normalized to their last single bin, as these..."

The design of all these figures improved dramatically. That is great! Nevertheless:

Figures 5, 6, 10, 11, 15, 16: Do we really need ALL pT_i and eta_i distributions although often they resemble each other?

We left Fig 5 and 6 in the text, the other pT and eta spectra have been placed in the appendix. We would like to argue that they all are of interest. Multiple single inclusive jet studies have been performed in the past, however, inclusive four jet spectra have not been studied this extensively in combination with all the models. It is not only the spectrum of the leading jet that is important but of all four of them.

Figures 7,8,9,12,13,14,17,18,19: Do we really need all the "unnormalised" plots? Isn't the difference in total cross section already visible just from the eta_1 distribution?

We have removed the plots in terms of the absolute cross sections and have adapted the text accrodingly, however, we would like to include these figures as supplementary material to the paper or in the appendices, with the possibility of making them too long, let us continue this discussion further of what you think of this idea.

We believe, an effort should be made by the authors to decide on figures to keep in the main part, to move to an appendix or similar, or to drop altogehter.

See above.

Figures and Tables 2,3,4: You discuss the results in the tables before you discuss the figures => suggest to move the tables to appear before the corresponding figures.

Position of tables and figures have been switched.

Comments to text: =============


l 9: data is -> data are next-to-last: -> centre-of-mass energies


3th -> 3rd.


pb -> pb^-1


The common × 10 factor is very odd.

Rewritten in mu barn instead of pb.

1. Introduction:

l 7: information on the -> information of the l 40: In the absctract it is 0.042 / pb. Please use one number with sufficient, but not more, digits throughout the text, tables and figures.

Corrected and 0.042 / pb will be used hence forth.

(missing line number): Radiation before -> Radiations before


l 43: Maybe better here "MC models" -> "MC event generators"?


l 43: remove quotations for DPS-sensitive


2. Observables:

1st and 4th bullet: "azimuthal angle difference" or "azimuthal angular difference"? A change would need to applied to the text as a whole, of course.

Used "azimuthal angular difference" throughout the text.

l 81: "in the tail" is unclear. "towards low Delta S"?


l 93: distributions --> predictions? (but maybe dropped anyway)


3. The CMS detector:

l 138: Is there a ref. for 13 TeV as well instead of only [34]?

Nothing published, there is an inactive cadi entry for the same study at 13 TeV but without any information:

5. Data analysis:

l 248: 0.04185 ... remove digits and superfluous "t" Also suggest: --> "... consisted of a series of four single-jet triggers ..."

Changes implemented.

l 253: drop "more robust" otherwise it could not be a reference ....

Changes implemented.

l 260ff: Here and throughout, also in figure captions: Please decide for one type of writing "first, second, third ..." or "1st, 2nd, 3rd, ...". Up to now it's a mixture.

Used "first", "second", ... everywehre.

6. Extraction of sigma_eff:

line after 281; --> "lowest pT single jet trigger"; --> "has a pT threshold of"

Changes implemented.

l 293: There is a superfluous "(stat)"


7. --> "Systematic effects" or "... uncertainties"

l 395 "uncertainty of is determined" ??

Removed the "of", was superfluous.

l 372 Missing () around Eq. ...

Parenthesis added to eq number.

Table 1 caption: --> "... uncertainties. The statistical uncertainty ..." "is given" --> "are given"

Rewrote scentence to:
"An overview of systematic uncertainties of all the observables, along with the statistical and the total uncertainties.
All uncertainties are given in percentages."

8. Results:

Table 2 caption: --> "... data and all ..." (similar in table 3, 4) 2nd row last col.: +- super/subscripts missing (same error in table 3, 4)

Changes implemented.

Suggest to have abbreviations "KT" and "PW" for KaTie and POWHEG. P too often means Pythia ...

Altered the legends in the plots to this notation.

Fig. 17 caption: Reference with [??]

Added correct label for the reference.

Fig. 21: lowest panel: the template band is missing

The template band has been scaled with a factor -10 in order to show it on the figure, the legend and caption have been altered accordingly.

l 389, 390: P8, H++, H7 abbreviations are not defined. This is the same for figures. Perhaps defined them in caption as well, similar to l 437, 438?

Defined P8, H++ and H7 in text, and added definition in caption of all figures.

Paper Review: draft v7 -- ARC review (July 27, 2020)

Color code:

Green: comment accepted and implemented

Blue: comment accepted, but text has been modified in a different way than proposed

Orange: response given, but no change has been implemented

Comments from Philip Chang

1. I don't quite understand the reason for vetoing events with R_ij < 0.4 when mixing two SPS data events to form DPS template. Supposedly, in a true DPS process, nature doesn't care that there is a jet overlapping. I feel like this veto condition will lead to a formation of template events that bias towards larger number of njets. And it would lead to eff_4j that is higher than the true value. What am I missing?

We implemented the veto condition to avoid the overplapping jets in the detector, we encountered the approach in an DPS measurement from ATLAS [1]. You are correct that in the data this does occer. However, we wanted to avoid a formal re-clustering procedure if this would be possible and if this would not affect the resulting DPS distributions, as such a procedure would have complicated things a lot more. In order to justify the veto condition, we did a naive clustering of the overlapping jets, i.e. recombination of the pT vectors of both jets by a vectorial sum and a recombination of the pseudorapidity of the overlapping jets. Afterwards we compared the shape of the DeltaS _DPS distributions with and without reclustering and found no deviations larger than 3%. Note that only the shape is important as the distribution is rescaled in the template fit. A variation of 3% in the shape of the DeltaS _DPS distribution would not affect the extracted values of f_DPS in a significant way. All of this is detailed, along with the shape comparison plot, in the analysis notes in Sec. 7 (pages 36-37). I hope this clarifies why we included the condition if not please let me know.


2. What percentage of events were thrown away due to this veto condition?

Approximately 10% of all events are lost (veto passing rate = 0.89408), however not the amount of events is important, but rather the change in shape of the DeltaS _DPS distribution due to the removal of these events. In the AN we have compared the DeltaS distribution with veto and with naive clustering and found a relative difference in shape of at most 3%, see the answer above.

3. What would happen if you don't require the veto condition and let the jet merge into a single jet (post re-clustering) and then recompute eff_4j? (i.e. njets will be slightly smaller, and some portion of events will have higher pt for the merged case) Does/would it change significantly? If so, should this be included as systematics?

The difference between the eps_4j values is of sub percent level. Below you can find the exact amount of events passing the veto and surviving selection, as well as the epsilon_4j values. Note that the events are weighted, the totals are therefore no whole numbers. The values also differ lightly from the one in the paper as I used a smaller sub-sample to check these numbers, as it is quite a long calculation when done on the full sample.

NO veto condition with clustering
Total events = 1545194.36937
Total events passing the veto = 1545194.36937, veto-passing rate = 1.0
Total events of selected DPS events = 498202.640429
epsilon_4j = 0.322420693671

veto condition with NO clustering
Total events = 1546323.65758
Total events passing the veto = 1383170.53586, veto-passing rate = 0.894489668501
Total events of selected DPS events = 440271.956622 ,
epsilon_4j = 0.318306344161

4. Is there some comparison between this data-merged template of DPS vs. simulated DPS samples somewhere?

In the result section 8.4 we compared the distributions for the DeltaS _DPS distribution for the pure DPS Pythia 8 + CP5 and onshell KaTie + CP5 models.

5. I am having a hard time understanding Equation 10. Equation 10 does not seem to reduce to DPS formula Equation 1 in Section 1. I was naively expecting the equation to reduce to Equation 1 with two processes being Eq 8 and Eq 9, but with the factor of eff_4j at the front. I feel like I am missing something there...

The reason for the two terms in eq. (10) is due to the symmetry factor m in eq. (1) and due to the cuts of the inclusive single jet processes A and B. Whenever both processes are identical, the symmetry factor takes a value equal to 1, and 2 otherwise. Therefore the symmetry factor is not unambiguously defined for the processes A and B as they become identical whenever the pT of the leading jet surpasses 50 GeV. Therefore we define two disjunct processes sigma_A and (sigma_B-sigma_A) that cover the whole phase space for which the symmetry factor is clearly defined. Eq.(1) is then rewritten as
sigma_eff = (m/2) * (eps_4j/sigma_eff) * (sigma_A* (Sigma_A + (Sigma_A-Simga_B)))
The m becomes equal to 1 for the first term and equal to 2 for the second term.

6. Why was ∆S chosen for the variable to extract f_DPS and not ∆phi_soft? Based on Figure 3, it seemed DPS is much more pronounced in ∆phi_soft. If I am reading this correctly it's 2:1 between SPS and DPS at high ∆phi_soft And it is 4:1 between SPS and DPS for ∆S.

Historically we used the same 35-30-25-20 GeV cuts for all observables. When studying the migrations for the unfolding procedure we noticed that the matrix of migrations was non-diagonal for the DeltaS observable. We had to increase the cuts to obtain a stable unfolding for the DeltaS observable due to the jet energy resolution at low pT. We then started comparing the data with different models and saw that the DeltaS observable, which has been used in many DPS studies before for the extraction of the sigma_eff, was the most stable. In the sense that it was least affected by effects from different parton showers, the order of the calculations ect. All models, except the KaTie models, undershoot the shape of the slope, leaving room for a DPS contribution. Therefore we decided to continue with the DeltaS observable as it showed to be the most robust variable with regards to DPS effects.

Lastly, I am curious what the answer was to one of the questions from other ARC member

- While double parton scattering would imply at least 4 jets, it may be > discussed in the paper why exactly 4-jet events were studied. In
> principle, one
> of the scatterings could even produce 3 jets probably, so multiple
> combinations
> would be possible altogether. -> DISCUSSED DURING MEETING ALREADY

To clarify: we do not select exactly four jet events, we perform a study on inclusive 4-jet events, which has been adapted in the title and throughout the paper for the next version. We also take into account that 3 or 1 jets instead of 2 can originate from a DPS events by using the inclusive single jets spectra to construct a DPS sample. Generally one is interested in the production of two dijet pairs in DPS as they can exhibit very different correlations compared to SPS. An inclusive 5-jet, ... state would unnecessarily complicate things even more.

Comments by Mate Csanad

General comments:

- From the data vs model comparision plots it seemed that basically all models are away by multiple sigmas from the data, and this is not mentioned in the abstract or in the summary. -

Most models are LO models, based on collinear factorization and the DGLAP equations for the parton evolution. It is expected/known that these models fail at low transverse momentum, where cross sections larger than the data are generally obtained. At high transverse momentum a better agreement is observed, as expected.

- Figure axis are not properly labeled in many cases, especiall w.r.t. units.

Figures will be updated.

- Introduction in general is a little bit short for such an extensive paper (where there are no length constraints).

Going into detail on the DPS cross section would take us to far we believe, however a figure depicting SPS vs DPS events will be included in the introduction with some explenation in the caption.

- Momentum is in GeV throughout the paper, probably its fine, but it could be mentioned that the "/c" is dropped.

We have never encountered this before and do not think it is necessary to explicitly state the use of natural units.

- Repetitive statements in captions could be shortened, e.g. "Error bars and bands are shown similarly to Fig. 4"

Updated the captions as suggested throughout the paper.

- In what sense are multijet models multijet compared to the non-multijet models (as all contain at least 4 jets in the end, I assume)? This is I guess trivial, maybe it is the 2->2 vs 2->3,4 matrix elements (also, later on, these are called multi-leg models, if I understand well), but it could be mentioned to educate some of the readers.

The multijet models have a combination of higher order MEs and/or NLO corrections, which allow for more than 2 jets to come from the ME and thus becoming less dependent on the parton shower in order to create an inclusive 4-jet event. I will add a line to the introduction of the multijet events in section 4.2.

Section 1
- Maybe some more details in paragraphs 1&2 would be useful to explain more about DPS? Also, a figure could be placed here (as from a non-expert point of view).

Going into detail on the DPS cross section would take us to far we believe, however a figure depicting SPS vs DPS events will be included in the introduction with some explenation in the caption.

- While double parton scattering would imply at least 4 jets, it may be discussed in the paper why exactly 4-jet events were studied. In principle, one of the scatterings could even produce 3 jets probably, so multiple combinations would be possible altogether.

It is an inclusive 4-jet measurement, the title has been changed. We take into account that 3 jets can be produced in one event (see the section on the extraction strategy). Generally one is interested in the production of two dijet pairs in DPS as they can exhibit very different correlations compared to SPS. An inclusive 5-jet, ... state would unnecessarily complicate things even more.

Section 2
- Why are [18,19] not referenced in the first paragraph?

Both use a different variable in the study of DPS and none of the observables we have used.

Section 3

- It may not be customary, but if CMS is discussed, a plot of the detector could be considered to be shown, maybe with how particles are detected in it. (Even if ref. [35] gives a lot of details.)

We do not think this is necessary, it is not customary to include a plot of the detector.

- Does jet reconstruction choice (e.g. radius) contribute to systematics?

The anti-kT reconstruction algorithm with a cone radius of 0.4 is the preference of CMS. A different choice would result in differences in the distributions, however this would be a phenomenological study that would take us too far. However, the performance of the different algorithms has been discussed in older papers which are cited.

Section 5
- The online selection of jets (pT thresholds) seems to be more restrictive than the offline cuts. It is not fully clear to me how this is reconciled, i.e. how are the events in the sample e.g. with the leading jet having a pT of 35 or 50 GeV, when the online cut was already 100 GeV?

The triggers are used in disjunct areas of the phase space determined by the leading jet pT (no overlap between the triggers, the details and the trigger ranges are given in the AN). The cuts are not applied individually but on the four jets at a time. and event with transverse momenta of eg 65-50-40-15 will fire the trigger with a pT threshold of 30 GeV, however the set of cuts of 35-30-25-20 GeV are not met.

- This section seems to be very short, one could consider unifying it with the next one?

The section outlines the details of the data analysis while the next gives an overview of the extraction procedure, I fear that merging them would make things more confusing and take the focus away of what is important in each section.

Section 6
- A very minor thing, but mentioning TFractionFitter seems a bit too technical here. I understand it is built upon or mentioned later many times, nevertheless, I would consider discribing it instead of mentioning the object (that may change in the future).

TFractionFitter has implemented a binned likelihood method, and has been described this way in the paper now.

Section 7
- Surely there are other sources of systematic uncertainty, these may be negligible, but this could be mentioned, such as "There are other sources of systematic uncertainty (e.g. jet reconstruction, thresholds, offline cuts), but these are negligible as compared to the listed sources."

We do not view the jet reconstruction as a systematic uncertainty, a different jet cone radius would yield different distributions but should yield the same physics, eg the same value for the sigma_eff parameter. The thresholds and offline cuts are rather phase space restrictions than systematic uncertainties, therefore they are also not mentioned in the section of the systematic uncertainties.

- Are all systematic uncertainties fully correlated in pT, eta, phi, etc? This does not seem to be mentioned.

Added "The different cross sections are extracted from the corresponding rapidity spectra taking correlations into account, as they are in all calculations." to the paper.
If it would be insufficient, I could ellaborate more.

Section 8
- Two missing references (Tab. ??) in 8.1 and 8.3

Proper references inserted.

Comments by Klaus Rabbertz

- space missing in "important. Models"

Space added between the words.

- "better description" has to be better than something in line 13

Added "... compared to standard leading-order models."

- centre-of-energies instead of center-of-mass energies

Change included.

- 4-jet production -->> Inclusive 4-jet production

Title updated.

1 Introduction
- page 1 par 2 line 2 : perhaps constant instead of factor

Change included.

- page 1 par 2 line 6 : remove however in An alternative approach, however, ...

Change included.

- page 1 bottem page : INCLUSIVE four jets

Change included.

- page 2 par 2 line 2 : "obtained from data" should be reformulated, not completely clear

Sentence has been reformulated.

2 Observables
- page 2 separation data driven and phenomenological -->> earlier measurements and phenomenological studies

Change included.

- page 2 eq 3 : drop last inequal to i or cross check with other papers

Other papers do not have the last inequality, will be dropped from hence on forth.

- page 3 DeltaS : more flat -->> flatter

Change included.

- page 3 pT selections : define a label for the different cuts to avoid captions shorter and make paper more readable readable

The two sets of cuts are hence on forth denoted as p_T = (35,30,25,20) GeV and p_T^DeltaS = (50,30,30,30) GeV.

- page 3 Mention explicitly the advantage of having asymmetric cuts + elaborate on different sets of cuts for DeltaS

More detailed explenation + references are now included justifying the choice of asymmetric over symmetric pT cuts.

- page 3 Figures : units on the axes, include GeV and pb instead of pb/eta

Change included.

4 Monte Carlo event generators
- page 8 : Pythia 8.240 first point, mention Geant somewhere

Mentionned GEANT 4 with reference.

- page 9 : par 2 : PowHeg factorization and renormalization scale is set at 10GeV is this the way of handling?

Checked in the publication: for PowHeg NLO 2->2 (dijet production) the transverse momentum of the underlying Born configuration is used and not set to 10 GeV. Paper has been updated.

- page 9 : Title 4.3 and title 8.3 not consistent, total is not best description perhaps SPS+DPS

Both titles have been updated and subsequentally all references to the "total" samples have been replaced with "SPS+DPS" samples.

5 data analysis
- page 10 par 2 : correction as function of the jet pT and PS, mention trigger efficiency and how it is derived

Mentionned reference trigger method, i.e. trigger in question is compared to a more robust reference trigger (minimum bias trigger in our study).

- page 10 par 4, line 3 : to avoid instead of combat

Change included.

- page 10 par 4 : rate of misses and fakes, is slang + should be defined, a better description of fake and miss rate is necessary

Now using "into and out of phase space migration". Explicitly mentionned the high rates in the lowest bins of the pT spectra of the four leading jets and that these contributions also are the main reason why the rates of the other observables are this large. The size can also be viewed from the fact that 4 jets are selected simultanously thus the chance of the event being viewed as a fake or miss increases as only one of the four jets needs to be a fake or miss to classify the event this way.

- In the AN : plot eta fig 41 : include only the higher pT part, remove lowest pT bins for AN for a measure of comparison

6 extraction of effective cross section
- page 11 par 2 : add to veto condition that it is for a spatial matching only and not for anti kT as it works with the rapidity instead of pseudorapidity

Added: "... whenever two or more jet axes SPATIALLY coincide.", can reformulate further if this is not clear.

- page 11 epsilon_4j : rewrite systematic error and mention that statistical is negligible

Updated systematical and statistical uncertainty. Also updated the epsilon_4j rates determined at generator and detector level in section 7 to the same amount of number as the efficiency from data.

- formula 10 : merge paragraph above formula together with "Taking into account ..."

Paragraphs merged for clarities sake.

7 Systematics
- page 12 vertex uncertainty : 0.1404 --> 0.14 too many digits

Change included.

- page 12 luminosity uncertainty : 2.5% valid for low PU runs?

Checked the 2016 luminosity measurement results (LUM-17-001), no explicit mention of low pile up conditions and wether or not an exception should/could be made. Can contact the Lumi POG conveners in this regard if needed.

- table : backslash not commenly used, longs dash or N.A., apply to all other tables

All tables updated.

- epsilon_4j : mention that stat unc is negligible, only three/four digits

Change icluded.

8 Results
- page 14 sentence 1 : phase domains should be phase space domains

change included.

- page 14 8.1, line 3 : table question marks

Reference updated.

- page 14 last paragraph : better wording for jet cone bump

A more descriptive formulation has been given: "the tail of the distributions, which is dominated by effects of the jet cone size".

- page 19 line 1 : either well described or not well described, not marginally well

Changed the wording

- table 2 : DeltaS cross section not an good label, should use something that indicates the different pT selection, see previous comment on the different labels for pT cuts

Updated the cross section according to the previously introduced labels for the different sets of pT cuts.

Fig 8 : Find better solution to presenting these results (too many lines), eg ATLAS publications : multiple ratios + only one line per model in the top plotor specify bands for groups of models

The Pythia+Herwig and MultiJet figures have been updated to double and triple ratio plots. Will be included in next version of the paper.

8.4 paragraph 2
instead of jet configuration remark a more general comment

Made the comment more general by stating that the differences originate from the differences in pT specta, which lie at the root of the different jet configuration rates.

Fig 18+19: caption: last word -->> plot instead of pot


Paper Review: draft v6

Color code:

Green: comment accepted and implemented

Blue: comment accepted, but text has been modified in a different way than proposed

Orange: response given, but no change has been implemented

Comments from pre-approval (10 July 2020)

Link to pre-approval presentation (can also be found on cadiline):

- Patrick: You are using TFractionFitter. Did you double-check with home-made implementation? TFractionFitter seemed to get uncertainties wrong

Yes also made alternative chi2 implementation and were consistent. TFractionFitter was more stable and seems better.

- Guillelmo: Default generator used to report the unfolding ?

Unfolding with Pythia and Herwig independently, and the average of the two is used. Then the half-difference is used as an uncertainty.

- Guillelmo: Sounds unusual, is that common practice?

yes, used in other papers

- Guillelmo: but this is not dominant uncertainty

No JES is mostly the dominant uncertainty

- Guillelmo: what does <21% syst mean ?

it is the maximum impact among all bins

- Guillelmo: would be better to have a range

Ok will update the syst table

- Gabor: final results plots s26 Could you explain again the model dependence ?

Sigma_eff extracted for different models. Have several different options for ME & PS.

- Gabor: Differences coming from the interplay of the data and the model fit


- Gabor: which was the model used for 7 TeV ?

Custom UE tune using Professor to achieve best Data/MC agreement not exactly used now, but consistent with the PY8 results.

Q Gabor s7 veto condition on overlapping jets (DR 0.4) do you create any complicated acceptance effect with that ?

Cross checked with a different reclustering condition and obtained stable results. Could check stability vs the veto condition requirement cut.

Q Gabor Syst table s11. Very large ranges. Would be better to explain better why the syst change so much and what they depend on

OK Will provide that on the AN

- Gabor: s15 right plot. What are the two deviating models ?

The two blue are PY8 SPS CP5 & SPS CUETP8M1; pt-ordered showers from PYTHIA don’t describe angular correlation between jets. DPS effecs hidden because of large Parton shower dependence

- Gabor: Does this show up in the final sigma_eff ?

No because those effects appear to cancel in DeltaS, more robust for extraction of sigmaeff, while the Phi_ij is more sensititive to parton shower.

- Henning: About KaTie prediction. Generation cuts for the jets pT. Are they the same as the offline cuts ? Is that not a problem ?

It seems to work fine with that setup. Compared parton level and particle level, can have migration effects. Explained a bit also in the paper.

- Ela: uncertainty of 2% of 4jet efficiency ?

Based on a gen/reco level study, difference was 1% but doubled it to be conservative and cover it well.

- Henning: vertex uncertainty ? Very small. It is based on a very old study.

Will check if a more recent study was done. Should not have much impact anyhow.

Paper Review: draft v2

Color code:

Green: comment accepted and implemented

Blue: comment accepted, but text has been modified in a different way than proposed

Orange: response given, but no change has been implemented

Comments by Salvatore Rappoccio (April 03, 2020)

Incorporation feedback ongoing as of April 07, 2020.

Paper draft:

- Please find a CCLE ASAP. The grammar needs considerable improvement, and there is a lot of jargon. It's hard to follow the paper for this reason.

Pierre Van Mechelen will be the CCLE

- Title: I suggest "Study of double parton scattering in proton-proton collision events with 4 jets with low transverse momentum at sqrt(s) = 13 TeV " or something similar.

Title has been updated.

- The organization of the paper is a bit disjointed, with material scattered throughout different parts in a nonuniform way. - Material is repeated often, which can be condensed, especially in the Results "section".

The overall order of the paper has been revised and as soon as Pierre (the CCLE) is done reviewing the text, we will upload the new version.

- The abstract has too much technical detail, and needs a better description of the overall procedure. Rules of thumb include that if you mention something in the abstract, the introduction should explain it in more detail (or the conclusions). For example, the "pocket formula" is mentioned and then not explained until page 24.

Abstract has been rewritten, new version of the paper wil be uploaded ASAP.

- The definitions and discussions of the various "sigmas" is inscrutable. They are mentioned in the abstract, so presumably important, but not even mentioned until page 6 in the MC description, and then only obliquely. The other sigmas (sigma_DeltaS, sigma^DS_A,B, sigma_A, sigma_B, etc) are not defined at all. They are defined only in the AN, which needs to be put into the paper.

The definitions have been clarified and are introduced gradually and in a more logic way as before.

- The Results section is 2/3 of the paper. Much of the information is repeated, or should be moved earlier in the paper. The descriptions were rather hard to read at some points, so a CCLE would help a lot also. The descriptions are too detailed. You don't need to describe every bin of every plot, and some more summary information is appropriate here. Try to condense this into a few specific, targeted physics conclusions. On that note, see the next point.

See previous answers.

- The figures could use some more guidance to the readers, like adding (either in the caption or on the figures themselves) where the DPS enhancement is. This is critically important especially since you are normalizing to non-DPS regions.

Added indication of where the DPS sensitive area is.

- Section 8.6 and onward are indecipherable to me. I don't know what you're doing here. The overall picture needs to be explained much better. This is not mentioned at all in the introduction, the methodologies are not discussed, and the connection between the various pieces are unclear. I'm sorry, I cannot review from 8.6 onward in any meaningful way, I'm going to need a clearer paper draft to understand what is done.

The section has been restructured together with the whole paper, will be updoalded in the latest version (v3).

Physics comments:

MC samples: Paper line 179-193: The generator cuts for KaTie are very close to the offline selection. Has this been checked to ensure there are no biases?

This was checked by Hans for KaTie and Madgraph (with a 2 to 4 ME, which we do not have in our final analysis).

Response matrices: AN: Overall Figures 28-34 look extremely non-diagonal. I'm a bit concerned about the matching criteria. If you look at Fig 28, there are several large populations along eta_det = 0 and eta_gen=0. These are almost 10% of the diagonal elements, which seems enormous to me. To me this suggests some mis-association in the matching procedure. Do you have 1-d plots of the delta R between the detector and generator level jets? Also, the "R_pt" in Equation X is set to 0.3, which seems extremely large (especially at high pt) given Figure 18, where the resolutions are much smaller. I'm not convinced the pt matching is appropriate here, and may be allowing in the wrong jets. My suggestion is to adjust the matching criterion to be eta-phi matching for all of the observables. Even if they are technically correlated with the unfolded distribution it may be necessary to ensure you're picking the right jets. The recommended matching is to use half of the jet radius. Incidentally, the color scheme makes it nearly impossible to understand this. You should use a monochromatic scheme so that differences in color are meaningful. As of now, you're seeing 10-40% off-diagonal elements, from what I can tell. The fraction of fakes and misses are also enormous at low pt. (Incidentally we also usually quote this in terms of purity and stability, not fakes and misses).

I will update the migration matrices to a monochromatic scheme for clarity.

In chapter 7 we do match jets with a R_pt of 0.3 in the determination of the resolution. The matching procedure here is not related to the one we perform during the unfolding. Here the mathing is done to determine the resolution and find an optimal binning.

We started the unfolding without any jet matching, the migration matrices in Fig. 28 are those of the rapidity spectra without any matching. The migration matrices show a large background due to the fact that the first leading jet on detector level not always corresponds to the first leading jet on generator level for example. Therefore we have implemented a spatial matching, detailed in section of the AN. The matching ensures that the same jets are compared to each other on detector and generator level and effectively erasing the background in Fig. 28, resulting in Fig. 30 for the rapidity spectra.

The pT spectra show non-diagonal behavior, however the bin values off the diagonal are multiple orders smaller as the diagonal elements. While the pT dependent observables, especially the Delta S observable, show non diagonal behavior due to the bad resolution of low pT jets. The only observable for which a background remains is the difference in azimuthal angle between the most remote jets, which is due to its definition, as explained in the AN.

I can provide purity and stability plots if needed. I have left them out, as they are originally defined for square migration matrices and as the TUnfold package uses different detector and generator level binning.

MC statistical uncertainties: AN Table 5: You're using the flat QCD samples for the measurement, which have relatively low statistics. Are you using the jackknife resampling technique for the MC uncertainties in the response matrices?

We do not use the jackknife resampling techique as these are not the standard flat QCD samples provided by CMS, they have a pTHat range of 15-7000 and a different way of oversampling (FlatP6). We requested these two samples for the derivation of the JEC originally, with a pTHat of 15-1000 and with the Flat oversampling with the goal of have more statistics in the low pT region. Roughly a third of all events (3.5 million) survive the analysis cuts of the samples.

PT distribution: Paper Figure 1: I'm not really understanding why the pt distribution is so far off, given that you reweight the samples via the AN Figure 25 histograms, which show almost perfect agreement, as do AN Figure 42. I'm confused what you're doing here. Also, are you applying nonperturbative corrections? I don't recall the inclusive jet measurement seeing such large differences in the pt spectrum but perhaps I'm wrong (it's hard to tell from their paper, the figures are small).

In the unfolding procedure we have reweighted the samples to obtain a more accurate description of the migrations. When discussing the results, we show the original samples, without the reweighing, compared to the unfolded data. In the inclusive jet measurement paper they indeed obtain smaller differences in the pT spectrum. However the discrepancies between the data and the models grows when going from an inclusive jet selection to 2 ,3 and eventually 4 jets, these distributions at detector level we have checked in the past.

Regularization: AN Table 7: The regularization conditions you're quoting seem enormous. Is this tau? If so, shouldn't this be a number much less than unity?

This is an error on my part, the title of the columns Reg.Conc. and cond(A) seem to have been switched. The numbers are the condition numbers of the migrations matrices, while the column with Size/Der./Curv. is the choice of the regularization condition in the unfolding procedure.

-- MaximPieters - 2020-04-07

Edit | Attach | Watch | Print version | History: r35 < r34 < r33 < r32 < r31 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r35 - 2021-03-05 - MaximPieters
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback