Analysis Checks

This twiki summarizes very useful information regarding to the HH4b resonant analysis fluctuations in HIG-17-009

Limits using the sideband prediction (Andrea Rizzi)

Limits

The following plots are the calculated observed limits using the side-band distribution instead of the signal region.

Observed Limits using side-band ( no bias uncertainty included ) Observed limits using side-band region ( bias uncertainty included )
UpperLimit_nobias_SB.png UpperLimit_bias_SB.png

Significances at mX = 330, 470, 560 GeV

The following table compares the calculated significances of the observed limit using the side-band distribution and the signal region distribution:

mass significance (local) in SB significance (local) in SR
330 -1.543 -3.609
470 +0.874 +2.584
560 -2.021 -2.571

Implementation of the bias

Bias estimate procedure

Make sure the bias is calculated properly. The bias is computed from toys using the nominal (Novosibirsk) and alternative (RevCrystal Ball) functions. The procedure is the following:

1. Throw 300 background-only toys from nominal function (mu_inj = 0)

2. Fit the toys with signal + alternative background model (Get signal strenght u_fit +/- sigma_fit)

3. Get distributions from toys: (u_fit - u_inj)/sigma_fit, sigma_fit, and u_fit - u_inj

4. Compute the calcuated bias = mean of the u_fit-u_inj distribution * Signal Normalization

5. Add the background uncertainty to the datacards. The uncertainty is included as an additional background process with the same shape as the signal (signal-as-backgroundshape), and its normalization allowed to vary up the measure bias. This normalization is centered at 0, and a Gaussian uncertainty of standard deviation equal to the calculated bias (Example mX = 300 GeV: datacard_330_novo_285_625.txt).

In the case of LMR, the Signal Normalization is the production cross-section (1 pb). Below, the mention distributions for the masses 300 and 350 GeV, and a table with the calculated biases:

(u_fit - u_inj)/sigma_fit, Mx=300 GeV sigma_fit, Mx=300 GeV (u_fit - u_inj)/sigma_fit, Mx=300 GeV
pull300.png sigma300.png bias300.png
(u_fit - u_inj)/sigma_fit, Mx=350 GeV sigma_fit, Mx=350 GeV (u_fit - u_inj)/sigma_fit, Mx=350 GeV
pull350.png sigma350.png bias350.png

Mass (GeV) Mean of (u_fit - u_inj)/sigma_fit Mean of sigma_fit Mean of (u_fit - u_inj) Calculated bias (pb)
300 0.12 0.3114 0.038 0.038
350 0.047 0.1344 0.005 0.005

After the application of the Spline interpolation (3rd order polynomial), we get the bias for the intermediate points:

Mass (GeV) Calculated bias (pb)
300 0.038
310 0.03834
320 0.03269
330 0.02358
340 0.01351
350 0.005

Background only fit for 330 GeV

Plot the fits (prefit / background only / signal + background) of the distribution of the 330 GeV Mass used for limits, in order to see that normalizations are correct.

Prefit background only signal+backgroud
330_prefit.png 330_fit_b.png 330_fit_s.png

Impact on the expected limits

Plot Paper Figure 9 including and not including the bias (PDFs are attached to the twiki):

Observed Limits ( no bias ) Observed limits ( bias )
UpperLimit_nobias.png UpperLimit_bias.png

Significance and limits not including the bias:

Mass (GeV) Observed Local Significance Observed limit (fb) Expected limit (fb) - 1 s. d. band width (fb) +1 s.d. band width (fb)
320 -2.69617 188.4 431.6 122.5 177.3
330 -3.64088 129.3 362.3 102.4 151.6
340 -2.78042 135.2 311.5 88.4 132.9

Significance and Limits including the bias:

Mass (GeV) Observed Local Significance Observed limit (fb) Expected limit (fb) - 1 s. d. band width (fb) +1 s.d. band width (fb)
320 -2.66357 192 435.5 124.1 178.9
330 -3.60928 131.4 366.2 103.9 153.3
340 -2.76961 136.1 313.5 89.4 131.2

Andrea Marini (Stat Com)

Validity Asymptotic Assumption

Calculate the observed/expected limits using two methods: Asymptotic (default) and HybridNew (alternative using toys). The results show that both methods estimate similar limits for Mx=330 GeV negative fluctuation.

  Asymptotic HybridNew
Observed Limit: r < 0.1314 r < 0.125847 +/- 0.078125
Expected 2.5%: r < 0.1967 r < 0.178337 +/- 0.0314367
Expected 16.0%: r < 0.2623 r < 0.248815 +/- 0.0163743
Expected 50.0%: r < 0.3662 r < 0.337338 +/- 0.0718463
Expected 84.0%: r < 0.5195 r < 0.496998 +/- 0.0383968
Expected 97.5%: r < 0.7121 r < 0.703139 +/- 0.00304321

LEE effect

Compute the looking elsewhere effect (LEE) with toys. The procedure is the following: a. Throw a toy (assuming no-signal) for the all mass spectrum and b. Get the observed significance as function of mass. The p-value is the number of toys passing a target local significance (e. g. 1 sigma, 2 sigma, etc) over the total number of toys. From 10k toys, the results of the study of the three fluctuations (1 positive, 2 negatives) in LMR2 (285,625) are the following:

mass p-value (local) p-value (global) significance (local) significance (global)
330 0.0002 0.0033 -3.609 -2.716
470 0.0049 0.0890 +2.584 +1.347
560 0.0051 0.0775 -2.571 -1.422

Probabilities of 1/2/3 fluctuations

2 sigma significance

Probabilities of one, two and three fluctuations with 2 sigma significance (either positive (+) or negative (-) )

Case Probability
+ 0.1531
- 0.1443
+ / - 0.2974
++ 0.0685
-- 0.0592
+- / -+ 0.0654
++ / -- / +- / -+ 0.1931
+++ 0.0279
--- 0.0193
+-+ / ++- / -++ 0.0265
-+- / --+ / +-- 0.0254
+++ / --- / +-+ / -+- / --+ / ++- / -++ / +-- 0.0991

2.5 sigma significance

Probabilities of one, two and three fluctuations with 2.5 sigma significance (either positive (+) or negative (-) )

Type of fluctuation Probability
+ 0.1042
- 0.0901
+ / - 0.1943
++ 0.0233
-- 0.0167
+- / -+ 0.0120
++ / -- / +- / -+ 0.0520
+++ 0.0063
--- 0.0038
+-+ / ++- / -++ 0.0018
-+- / --+ / +-- 0.0012
+++ / --- / +-+ / -+- / --+ / ++- / -++ / +-- 0.0131

2.571 sigma significance

Probabilities of one, two and three fluctuations with 2.571 sigma significance (either positive (+) or negative (-) )

Type of fluctuation Probability
+ 0.0906
- 0.0775
+ / - 0.1681
++ 0.0196
-- 0.0127
+- / -+ 0.0081
++ / -- / +- / -+ 0.0404
+++ 0.0048
--- 0.0029
+-+ / ++- / -++ 0.0010
-+- / --+ / +-- 0.0008
+++ / --- / +-+ / -+- / --+ / ++- / -++ / +-- 0.0095

2.584 sigma significance

Probabilities of one, two and three fluctuations with 2.584 sigma significance (either positive (+) or negative (-) )

Type of fluctuation Probability
+ 0.0890
- 0.0753
+ / - 0.1643
++ 0.0188
-- 0.0122
+- / -+ 0.0079
++ / -- / +- / -+ 0.0389
+++ 0.0044
--- 0.0029
+-+ / ++- / -++ 0.0008
-+- / --+ / +-- 0.0008
+++ / --- / +-+ / -+- / --+ / ++- / -++ / +-- 0.0089
-- DanielGuerrero - 2018-03-17
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng 330_fit_b.png r1 manage 14.8 K 2018-03-17 - 21:21 DanielGuerrero  
PNGpng 330_fit_s.png r1 manage 14.8 K 2018-03-17 - 21:21 DanielGuerrero  
PNGpng 330_prefit.png r1 manage 15.5 K 2018-03-17 - 21:21 DanielGuerrero  
PDFpdf UpperLimit_bias.pdf r1 manage 20.6 K 2018-03-17 - 20:40 DanielGuerrero  
PNGpng UpperLimit_bias.png r1 manage 33.8 K 2018-03-17 - 20:40 DanielGuerrero  
PNGpng UpperLimit_bias_SB.png r1 manage 33.4 K 2018-03-28 - 20:38 DanielGuerrero  
PDFpdf UpperLimit_nobias.pdf r1 manage 20.6 K 2018-03-17 - 20:40 DanielGuerrero  
PNGpng UpperLimit_nobias.png r1 manage 33.8 K 2018-03-17 - 20:40 DanielGuerrero  
PNGpng UpperLimit_nobias_SB.png r1 manage 33.2 K 2018-03-28 - 20:38 DanielGuerrero  
PNGpng bias300.png r1 manage 15.4 K 2018-03-17 - 21:08 DanielGuerrero  
PNGpng bias350.png r1 manage 14.4 K 2018-03-17 - 21:08 DanielGuerrero  
Texttxt datacard_330_novo_285_625.txt r1 manage 1.1 K 2018-03-17 - 22:08 DanielGuerrero  
PNGpng pull300.png r1 manage 21.5 K 2018-03-17 - 21:08 DanielGuerrero  
PNGpng pull350.png r1 manage 22.0 K 2018-03-17 - 21:08 DanielGuerrero  
PNGpng sigma300.png r1 manage 13.9 K 2018-03-17 - 21:08 DanielGuerrero  
PNGpng sigma350.png r1 manage 13.7 K 2018-03-17 - 21:08 DanielGuerrero  
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2018-03-28 - DanielGuerrero
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback