-- AmnonHarel - 07-Jul-2010


During the approval meeting, Gigi raised the question of how does the 10% JES uncertainty translate to a degradation of the contact interaction limits. Naively, since the SM is almost independent of the JES, and the contact interaction models depend mostly on lambda/JES, one would expect the limits to degrade by 10% as well.

This begs several questions:

  • what is the current degradation?
  • how does the current degradation come about?
  • does it matter that these are 95% limits, not 68% limits?
    • in some ideal scenarios this does not matter, regardless of whether we're setting 1sigma or 5sigma limits, the shifting along the x axis is the same. Can this hold for CLs + Cousins-Highland?
  • do CLs limits behave the same as frequentist limits?
    • for exclusion probabilities above 50% the two are close, but below that they diverge rapidly (CLs hits 0 when frequentist plateaus into 5%, but the difference is bigger before the plateau)
  • in the ideal case (detailed below) should the limits degrade by 10% according to the Cousins-Highland logic?
    • nominal hypothesis is independent of S (e.g. a constant)
    • alternate hypothesis is to add a term f(x,S,L)=g(x*S/L), e.g., g(y)=0 below 1 and 1-y above it
    • set limits on L, treating S as nuisance parameter (e.g. Gaussain(1, 0.1)).
    • frequentist limits?
    • 68% limits?
    • I don't see a trivial answer - interesting statistics study?

Attempts to start answering these questions are below

Current degradation

The PAS analysis used ensembles of 4000 pseudo datasets (PDSs). What the ensemble statistics uncertainty on the resulting limits? Repeated the analysis a few times with different seeds - the predicted 95% CLs limits were: 556, 550, 558, 548 --> 553 +/- 5 This is for ensembles without systematic uncertainties (much quicker to generate).

For ensembles with only the JES uncertainty we get a predicted 95% CLs limits of 470. For a degradation of (15 +/- 1)%. Which at least naively, is on the conservative side.

The degradation for a particular dataset, e.g. the real data, is much less predictable and thus doesn't teach us as much about this.

It's also worth noting that in Jim's predictions table, he saw that the total uncertainties degrade the predicted limits by somewhat less than 10%. Which isn't very consistent with the degradation for the 11.4 nb dataset.

how does the current degradation come about?

As we showed in the approval talk: * the JES nuisance parameter changes the predictions used in creating the PDSs (but not the LLR) * per bin the change in the ratio are 5-30%. The change is basically DeltaJES * Mjj * slope_of_ratio, the latter two parameters scale the same as lambda increases, so that estimate holds for all lambdas

As the JES nuisance parameter is sampled (it is distributed as Gaus( 1, 0.1 ) ), it effects the LLR. This is demonstrated with NP ensembles of lambda=600GeV and only JES systematics:

LLR for 3 JES bins LLR profiled as a function of JES
uju_llr_by_jes.png uju_llr_jes_profile.png
  • the deviation from a linear dependence at very high JES is surprising. May be related to the non-Gaussian nature of the ratio distribution.

How does this effect the probability to exclude this model?

  • I haven't found the right things to plot here. The following are wrong:
    • the exclusion probability as a function of lambda in normal ensembles without systematics. Irrelevant because LLR changes between points so that it matches the simulated lambda.
    • probability of the new physics LLR to be above some threshold, e.g., the 68% / 95%, frequentist / CLs limits for that lambda. Irrelevant because the critical values come from the NP distributions themselves.
    • probability of the SM LLR to be above the threshold which depends on JES?
      • might be the right plot. Haven't made that one yet.

Plots of the wrong things....

Here is a plot of the exclusion probabilities for those same ensembles as a function of JES for four critical values.
  • Note that the critical values use the entire distribution, and thus depend on the JES uncertainties assumed
lambda=450 lambda=500 lambda=600
uju_excl_450.png uju_excl_prob_500.png uju_excl_prob.png
Due to the sampling of the JES nuisance parameter, the exclusion probability is a weighted mean of these probabilities. If these plots were linear, this uncertainty would not degrade the limits!

  • Limits with 11.4nb data and only JES systematics:
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng clim1_4KI0_01.png r1 manage 16.3 K 2010-07-07 - 18:10 AmnonHarel Limits with 11.4nb data and only JES systematics
PNGpng uju_excl_450.png r1 manage 17.8 K 2010-07-07 - 18:01 AmnonHarel Exclusion probabilities for lambda=450GeV
PNGpng uju_excl_prob.png r1 manage 17.9 K 2010-07-07 - 17:00 AmnonHarel Exclusion probabilities for lambda=600GeV
PNGpng uju_excl_prob_500.png r1 manage 18.2 K 2010-07-07 - 18:01 AmnonHarel Exclusion probabilities for lambda=500GeV
PNGpng uju_llr_by_jes.png r1 manage 12.4 K 2010-07-07 - 16:59 AmnonHarel LLR for NP ensembles @ lambda=600 with only JES systematics
PNGpng uju_llr_jes_profile.png r1 manage 10.5 K 2010-07-07 - 17:01 AmnonHarel LLR dependence on the JES nuisance parameter (JES only systematic)
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2010-07-07 - AmnonHarel
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback