-- AmnonHarel - 12-Sep-2010 How should we interpret the "disappearance" of the CLs limits at high lambda?

CLs basics

The "modified frequentist" CLs approach is to exclude regions of phase space where $CLs=\frac{CL_{s+b}}{CL_b} < 1-\alpha$, where $\alpha$ is the desired confidence level. Here, $\alpha=0.95$.

Fine print

  • When the number of pseudo datasets (PDSs) that underlie either $CL_{s+b}$ or $CL_b$ at the limit is below 10, we consider the determination of the limit unreliable and do not quote it.
    • For the final results, we generate enough PDFs at the crucial points in phase space to prevent this requirement from effecting the results, except where it clearly removes statistical noise (see $\lambda=5$TeV example below)
  • Since the test statistic is the log likelihood ratio, with or without systematics, when we define an excluded region of the LLR, it will normally be simply connected and include $-\infty$ (the sign convention is such that this is the value most unlike the new physics scenario). As we'll see, when the LLR does not include systematics, this may fail to hold. Currently, we quote only the uppermost excluded LLR in the lowest exclusion region.
  • The plots on this page use systematics which are slightly lower than the final ones. There are not qualitative differences, in particular, the sensitivity runs out at 4TeV either way, and quantitative differences are likely to be tiny.

The simple cases

To understand how the CLs limits go away, we'll look at plots of CLs as a function of LLR (for a given new physics model, here contact interactions).

Easy exclusion

LLR distributions CLs plot
cpsdn20001I2_88.png ex1.png

No exclusion

LLR distributions CLs plot
cpsdn50001I2_88.png ex2.png
The leftmost "0" is real, but due to a single point from the SM ensemble. The 0,0 points (quite a few, actually) are an artifact that does not effect the code (these plots were meant to be internal to the code...).

CLs with borderline sensitivity

More than one behavior near the edge of the experimental sensitivity is possible. Here are two simple scenarios. In both scenarios, under both hypotheses (i.e. in both ensembles), the LLR is distributed as a Gaussian and the two distributions are displaced by an amount that descreases as we run out of experimental sensitivity.
  • 1 - the Gaussians have the same width
    • in this scenario, the lower the LLR (below the SM peak), the more the SM is prefered.
    • thus, as $\lambda$ increases and we run out of experimental sensitivity, the CLs limit exists, and will rapidly drop to $-\infty$. However, our ability to determine its correct value will detriorate quickly - brute force ensemble testing is not a suitable tool for learning about the tails of distributions.
    • this is the behavior that was observed in the ICHEP results, where increasing the ensemble size by an order of magnitude extended the limit
  • 2 - the new physics distribution is wider
    • in this scenario, for very low LLR values (well below the SM peak), the SM is no longer prefered.
    • thus, as $\lambda$ increases and we run out of experimental sensitivity, the CLs limit no longer exists as the plot is always above 0.05
    • this is the behavior observed in the current results - increasing the ensemble size does not extended the limit

The current borderline sensitivity region

LLR distributions CLs plot
from 0 to 1 zoomed in
cpsdn40001I2_88.png ex3.png ex6.png
LLR distributions CLs plot
from 0 to 1 zoomed in
cpsdn40501I2_88.png ex4.png ex5.png

The table

Lambda At the CLs point Frequentist limiting Data
CLb CLsb LLR value LLR value LLR value
1.80 0.0498 0.9963 -1.01 -0.99 -27.75
1.85 0.0497 0.9939 -0.82 -0.78 -25.50
1.90 0.0494 0.9875 -1.49 -1.41 -24.14
1.95 0.0495 0.9906 -0.78 -0.73 -23.32
2.00 0.0489 0.9787 -2.04 -1.92 -21.23
2.20 0.0462 0.9238 -2.24 -2.03 -15.19
2.40 0.0443 0.8856 -2.17 -1.83 -13.98
2.60 0.0460 0.9194 -0.29 -0.12 -10.90
2.80 0.0414 0.8276 -0.94 -0.71 -8.84
3.00 0.0209 0.4188 -3.10 -1.95 -7.02
3.20 0.0157 0.3132 -2.99 -1.81 -5.99
3.40 0.0047 0.0933 -3.46 -1.67 -4.86
3.60 0.0058 0.1153 -2.74 -1.46 -4.00
3.80 0.0023 0.0466 -2.90 -1.38 -3.66
4.00 0.0020 0.0395 -3.02 -1.47 -3.57
4.05 --- --- --- -1.66 -3.41
4.10 --- --- --- -1.57 -3.27
4.15 --- --- --- -1.60 -3.18
4.20 --- --- --- -1.61 -3.03
5.00 --- --- --- -1.07 -1.49

Assorted attachments

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng cpsdn20001I2_88.png r1 manage 16.2 K 2010-09-12 - 21:23 AmnonHarel  
PNGpng cpsdn40001I2_88.png r1 manage 18.0 K 2010-09-12 - 21:23 AmnonHarel  
PNGpng cpsdn40501I2_88.png r1 manage 18.5 K 2010-09-12 - 21:24 AmnonHarel  
PNGpng cpsdn50001I2_88.png r1 manage 17.7 K 2010-09-12 - 21:24 AmnonHarel  
PNGpng ex1.png r1 manage 12.7 K 2010-09-12 - 21:19 AmnonHarel  
PNGpng ex2.png r1 manage 12.3 K 2010-09-12 - 21:19 AmnonHarel  
PNGpng ex3.png r1 manage 12.2 K 2010-09-12 - 21:20 AmnonHarel  
PNGpng ex4.png r1 manage 13.0 K 2010-09-12 - 21:20 AmnonHarel  
PNGpng ex5.png r1 manage 13.4 K 2010-09-12 - 21:20 AmnonHarel  
PNGpng ex6.png r1 manage 10.9 K 2010-09-12 - 21:20 AmnonHarel  
Edit | Attach | Watch | Print version | History: r13 | r6 < r5 < r4 < r3 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r4 - 2010-09-24 - AmnonHarel
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback