-- AmnonHarel - 07-Jul-2010
## Introduction

### Current degradation

### how does the current degradation come about?

### Plots of the wrong things....

Here is a plot of the exclusion probabilities for those same ensembles as a function of JES for four critical values.

Due to the sampling of the JES nuisance parameter, the exclusion probability is a weighted mean of these probabilities. If these plots were linear, this uncertainty would not degrade the limits!

During the approval meeting, Gigi raised the question of how does the 10% JES uncertainty translate to a degradation of the contact interaction limits. Naively, since the SM is almost independent of the JES, and the contact interaction models depend mostly on lambda/JES, one would expect the limits to degrade by 10% as well.

This begs several questions:

- what is the current degradation?
- how does the current degradation come about?
- does it matter that these are 95% limits, not 68% limits?
- in some ideal scenarios this does not matter, regardless of whether we're setting 1sigma or 5sigma limits, the shifting along the x axis is the same. Can this hold for CLs + Cousins-Highland?

- do CLs limits behave the same as frequentist limits?
- for exclusion probabilities above 50% the two are close, but below that they diverge rapidly (CLs hits 0 when frequentist plateaus into 5%, but the difference is bigger before the plateau)

- in the ideal case (detailed below) should the limits degrade by 10% according to the Cousins-Highland logic?
- nominal hypothesis is independent of S (e.g. a constant)
- alternate hypothesis is to add a term f(x,S,L)=g(x*S/L), e.g., g(y)=0 below 1 and 1-y above it
- set limits on L, treating S as nuisance parameter (e.g. Gaussain(1, 0.1)).
- frequentist limits?
- 68% limits?
- I don't see a trivial answer - interesting statistics study?

Attempts to start answering these questions are below

The PAS analysis used ensembles of 4000 pseudo datasets (PDSs). What the ensemble statistics uncertainty on the resulting limits? Repeated the analysis a few times with different seeds - the predicted 95% CLs limits were: 556, 550, 558, 548 --> 553 +/- 5 This is for ensembles without systematic uncertainties (much quicker to generate).

For ensembles with only the JES uncertainty we get a predicted 95% CLs limits of 470. For a degradation of **(15 +/- 1)%**.
Which at least naively, is on the conservative side.

The degradation for a particular dataset, e.g. the real data, is much less predictable and thus doesn't teach us as much about this.

It's also worth noting that in Jim's predictions table, he saw that the total uncertainties degrade the predicted limits by somewhat less than 10%. Which isn't very consistent with the degradation for the 11.4 nb dataset.

As we showed in the approval talk: * the JES nuisance parameter changes the predictions used in creating the PDSs (but not the LLR) * per bin the change in the ratio are 5-30%. The change is basically DeltaJES * Mjj * slope_of_ratio, the latter two parameters scale the same as lambda increases, so that estimate holds for all lambdas

As the JES nuisance parameter is sampled (it is distributed as Gaus( 1, 0.1 ) ), it effects the LLR. This is demonstrated with NP ensembles of lambda=600GeV and only JES systematics:

LLR for 3 JES bins | LLR profiled as a function of JES |
---|---|

- the deviation from a linear dependence at very high JES is surprising. May be related to the non-Gaussian nature of the ratio distribution.

How does this effect the probability to exclude this model?

- I haven't found the right things to plot here. The following are wrong:
- the exclusion probability as a function of lambda in normal ensembles without systematics. Irrelevant because LLR changes between points so that it matches the simulated lambda.
- probability of the new physics LLR to be above some threshold, e.g., the 68% / 95%, frequentist / CLs limits for that lambda. Irrelevant because the critical values come from the NP distributions themselves.
- probability of the SM LLR to be above the threshold which depends on JES?
- might be the right plot. Haven't made that one yet.

- Note that the critical values use the entire distribution, and thus depend on the JES uncertainties assumed

lambda=450 | lambda=500 | lambda=600 |
---|---|---|

- Limits with 11.4nb data and only JES systematics:

I | Attachment | History | Action | Size | Date | Who | Comment |
---|---|---|---|---|---|---|---|

png | clim1_4KI0_01.png | r1 | manage | 16.3 K | 2010-07-07 - 18:10 | AmnonHarel | Limits with 11.4nb data and only JES systematics |

png | uju_excl_450.png | r1 | manage | 17.8 K | 2010-07-07 - 18:01 | AmnonHarel | Exclusion probabilities for lambda=450GeV |

png | uju_excl_prob.png | r1 | manage | 17.9 K | 2010-07-07 - 17:00 | AmnonHarel | Exclusion probabilities for lambda=600GeV |

png | uju_excl_prob_500.png | r1 | manage | 18.2 K | 2010-07-07 - 18:01 | AmnonHarel | Exclusion probabilities for lambda=500GeV |

png | uju_llr_by_jes.png | r1 | manage | 12.4 K | 2010-07-07 - 16:59 | AmnonHarel | LLR for NP ensembles @ lambda=600 with only JES systematics |

png | uju_llr_jes_profile.png | r1 | manage | 10.5 K | 2010-07-07 - 17:01 | AmnonHarel | LLR dependence on the JES nuisance parameter (JES only systematic) |

This topic: Main > TWikiUsers > AmnonHarel > AmnonHarelJetRatio > AmnonHarelUnderstandingJESuncertainties

Topic revision: r1 - 2010-07-07 - AmnonHarel

Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.

or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback

or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback