# Analysis Checks

This twiki summarizes very useful information regarding to the HH4b resonant analysis fluctuations in HIG-17-009

## Limits using the sideband prediction (Andrea Rizzi)

### Limits

The following plots are the calculated observed limits using the side-band distribution instead of the signal region.

 Observed Limits using side-band ( no bias uncertainty included ) Observed limits using side-band region ( bias uncertainty included )

### Significances at mX = 330, 470, 560 GeV

The following table compares the calculated significances of the observed limit using the side-band distribution and the signal region distribution:

 mass significance (local) in SB significance (local) in SR 330 -1.543 -3.609 470 +0.874 +2.584 560 -2.021 -2.571

## Implementation of the bias

### Bias estimate procedure

Make sure the bias is calculated properly. The bias is computed from toys using the nominal (Novosibirsk) and alternative (RevCrystal Ball) functions. The procedure is the following:

1. Throw 300 background-only toys from nominal function (mu_inj = 0)

2. Fit the toys with signal + alternative background model (Get signal strenght u_fit +/- sigma_fit)

3. Get distributions from toys: (u_fit - u_inj)/sigma_fit, sigma_fit, and u_fit - u_inj

4. Compute the calcuated bias = mean of the u_fit-u_inj distribution * Signal Normalization

5. Add the background uncertainty to the datacards. The uncertainty is included as an additional background process with the same shape as the signal (signal-as-backgroundshape), and its normalization allowed to vary up the measure bias. This normalization is centered at 0, and a Gaussian uncertainty of standard deviation equal to the calculated bias (Example mX = 300 GeV: datacard_330_novo_285_625.txt).

In the case of LMR, the Signal Normalization is the production cross-section (1 pb). Below, the mention distributions for the masses 300 and 350 GeV, and a table with the calculated biases:

 (u_fit - u_inj)/sigma_fit, Mx=300 GeV sigma_fit, Mx=300 GeV (u_fit - u_inj)/sigma_fit, Mx=300 GeV (u_fit - u_inj)/sigma_fit, Mx=350 GeV sigma_fit, Mx=350 GeV (u_fit - u_inj)/sigma_fit, Mx=350 GeV

 Mass (GeV) Mean of (u_fit - u_inj)/sigma_fit Mean of sigma_fit Mean of (u_fit - u_inj) Calculated bias (pb) 300 0.12 0.3114 0.038 0.038 350 0.047 0.1344 0.005 0.005

After the application of the Spline interpolation (3rd order polynomial), we get the bias for the intermediate points:

 Mass (GeV) Calculated bias (pb) 300 0.038 310 0.03834 320 0.03269 330 0.02358 340 0.01351 350 0.005

### Background only fit for 330 GeV

Plot the fits (prefit / background only / signal + background) of the distribution of the 330 GeV Mass used for limits, in order to see that normalizations are correct.

 Prefit background only signal+backgroud

### Impact on the expected limits

Plot Paper Figure 9 including and not including the bias (PDFs are attached to the twiki):

 Observed Limits ( no bias ) Observed limits ( bias )

Significance and limits not including the bias:

 Mass (GeV) Observed Local Significance Observed limit (fb) Expected limit (fb) - 1 s. d. band width (fb) +1 s.d. band width (fb) 320 -2.69617 188.4 431.6 122.5 177.3 330 -3.64088 129.3 362.3 102.4 151.6 340 -2.78042 135.2 311.5 88.4 132.9

Significance and Limits including the bias:

 Mass (GeV) Observed Local Significance Observed limit (fb) Expected limit (fb) - 1 s. d. band width (fb) +1 s.d. band width (fb) 320 -2.66357 192 435.5 124.1 178.9 330 -3.60928 131.4 366.2 103.9 153.3 340 -2.76961 136.1 313.5 89.4 131.2

## Andrea Marini (Stat Com)

### Validity Asymptotic Assumption

Calculate the observed/expected limits using two methods: Asymptotic (default) and HybridNew (alternative using toys). The results show that both methods estimate similar limits for Mx=330 GeV negative fluctuation.

 Asymptotic HybridNew Observed Limit: r < 0.1314 r < 0.125847 +/- 0.078125 Expected 2.5%: r < 0.1967 r < 0.178337 +/- 0.0314367 Expected 16.0%: r < 0.2623 r < 0.248815 +/- 0.0163743 Expected 50.0%: r < 0.3662 r < 0.337338 +/- 0.0718463 Expected 84.0%: r < 0.5195 r < 0.496998 +/- 0.0383968 Expected 97.5%: r < 0.7121 r < 0.703139 +/- 0.00304321

### LEE effect

Compute the looking elsewhere effect (LEE) with toys. The procedure is the following: a. Throw a toy (assuming no-signal) for the all mass spectrum and b. Get the observed significance as function of mass. The p-value is the number of toys passing a target local significance (e. g. 1 sigma, 2 sigma, etc) over the total number of toys. From 10k toys, the results of the study of the three fluctuations (1 positive, 2 negatives) in LMR2 (285,625) are the following:

 mass p-value (local) p-value (global) significance (local) significance (global) 330 0.0002 0.0033 -3.609 -2.716 470 0.0049 0.0890 +2.584 +1.347 560 0.0051 0.0775 -2.571 -1.422

### Probabilities of 1/2/3 fluctuations

#### 2 sigma significance

Probabilities of one, two and three fluctuations with 2 sigma significance (either positive (+) or negative (-) )

 Case Probability + 0.1531 - 0.1443 + / - 0.2974 ++ 0.0685 -- 0.0592 +- / -+ 0.0654 ++ / -- / +- / -+ 0.1931 +++ 0.0279 --- 0.0193 +-+ / ++- / -++ 0.0265 -+- / --+ / +-- 0.0254 +++ / --- / +-+ / -+- / --+ / ++- / -++ / +-- 0.0991

#### 2.5 sigma significance

Probabilities of one, two and three fluctuations with 2.5 sigma significance (either positive (+) or negative (-) )

 Type of fluctuation Probability + 0.1042 - 0.0901 + / - 0.1943 ++ 0.0233 -- 0.0167 +- / -+ 0.0120 ++ / -- / +- / -+ 0.0520 +++ 0.0063 --- 0.0038 +-+ / ++- / -++ 0.0018 -+- / --+ / +-- 0.0012 +++ / --- / +-+ / -+- / --+ / ++- / -++ / +-- 0.0131

#### 2.571 sigma significance

Probabilities of one, two and three fluctuations with 2.571 sigma significance (either positive (+) or negative (-) )

 Type of fluctuation Probability + 0.0906 - 0.0775 + / - 0.1681 ++ 0.0196 -- 0.0127 +- / -+ 0.0081 ++ / -- / +- / -+ 0.0404 +++ 0.0048 --- 0.0029 +-+ / ++- / -++ 0.0010 -+- / --+ / +-- 0.0008 +++ / --- / +-+ / -+- / --+ / ++- / -++ / +-- 0.0095

#### 2.584 sigma significance

Probabilities of one, two and three fluctuations with 2.584 sigma significance (either positive (+) or negative (-) )

 Type of fluctuation Probability + 0.0890 - 0.0753 + / - 0.1643 ++ 0.0188 -- 0.0122 +- / -+ 0.0079 ++ / -- / +- / -+ 0.0389 +++ 0.0044 --- 0.0029 +-+ / ++- / -++ 0.0008 -+- / --+ / +-- 0.0008 +++ / --- / +-+ / -+- / --+ / ++- / -++ / +-- 0.0089
-- DanielGuerrero - 2018-03-17
Topic attachments
I Attachment History Action Size Date Who Comment
png 330_fit_b.png r1 manage 14.8 K 2018-03-17 - 21:21 DanielGuerrero
png 330_fit_s.png r1 manage 14.8 K 2018-03-17 - 21:21 DanielGuerrero
png 330_prefit.png r1 manage 15.5 K 2018-03-17 - 21:21 DanielGuerrero
pdf UpperLimit_bias.pdf r1 manage 20.6 K 2018-03-17 - 20:40 DanielGuerrero
png UpperLimit_bias.png r1 manage 33.8 K 2018-03-17 - 20:40 DanielGuerrero
png UpperLimit_bias_SB.png r1 manage 33.4 K 2018-03-28 - 20:38 DanielGuerrero
pdf UpperLimit_nobias.pdf r1 manage 20.6 K 2018-03-17 - 20:40 DanielGuerrero
png UpperLimit_nobias.png r1 manage 33.8 K 2018-03-17 - 20:40 DanielGuerrero
png UpperLimit_nobias_SB.png r1 manage 33.2 K 2018-03-28 - 20:38 DanielGuerrero
png bias300.png r1 manage 15.4 K 2018-03-17 - 21:08 DanielGuerrero
png bias350.png r1 manage 14.4 K 2018-03-17 - 21:08 DanielGuerrero
txt datacard_330_novo_285_625.txt r1 manage 1.1 K 2018-03-17 - 22:08 DanielGuerrero
png pull300.png r1 manage 21.5 K 2018-03-17 - 21:08 DanielGuerrero
png pull350.png r1 manage 22.0 K 2018-03-17 - 21:08 DanielGuerrero
png sigma300.png r1 manage 13.9 K 2018-03-17 - 21:08 DanielGuerrero
png sigma350.png r1 manage 13.7 K 2018-03-17 - 21:08 DanielGuerrero
Topic revision: r6 - 2018-03-28 - DanielGuerrero

 Home Sandbox Web P View Edit Account
 Cern Search TWiki Search Google Search Sandbox All webs
Copyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback