(r7) AmnonHarelCLsInDCR < Main

Main Web>TWikiUsers>AmnonHarel>AmnonHarelJetRatio>AmnonHarelDijetCentralityRatio>AmnonHarelCLsInDCR (revision 7) (raw view)~~EditAttachPDF~~

-- Main.AmnonHarel - 12-Sep-2010

*How should we interpret the "disappearance" of the CLs limits at high lambda?*

%TOC{title="Contents:"}%

---++ CLs basics
The "modified frequentist" CLs approach is to exclude regions of phase space where %$CLs=\frac{CL_{s+b}}{CL_b} < 1-\alpha$%, where %$\alpha$% is the desired confidence level. Here, %$\alpha=0.95$%. 

---+++ Fine print
   * When the number of pseudo datasets (PDSs) that underlie either %$CL_{s+b}$% or %$CL_b$% at the limit is below 10, we consider the determination of the limit unreliable and do not quote it.
      * For the final results, we generate enough PDFs at the crucial points in phase space to prevent this requirement from effecting the results, except where it clearly removes statistical noise (see %$\lambda=5$%TeV example below)
   * Since the test statistic is the log likelihood ratio, with or without systematics, when we define an excluded region of the LLR, it will normally be simply connected and include %$-\infty$% (the sign convention is such that this is the value most unlike the new physics scenario). As we'll see, when the LLR does not include systematics, this may fail to hold. Currently, we quote only the uppermost excluded LLR in the lowest exclusion region.
   * The plots on this page use systematics which are slightly lower than the final ones. There are not qualitative differences, in particular, the sensitivity runs out at 4TeV either way, and quantitative differences are likely to be tiny.

---++ Updated results
Generated an obscene amount of additional PDSs...

---+++ Tools
   * Added statistical uncertainties on the CLs values.
      * Since the # of PDS (%$N$%) is predetermined, the uncertainties on CLb and CLsb are (independent) binomial efficiency problems. I use the usual approximation of putting in the observed efficiency (%$p=A/N$%, where %$A$% is the number of PDSs below the LLR value). This gives an uncertainty on the relevant fraction (%$f$%, either CLb or CLsb) of %$\sigma_f = f \sqrt{ \frac{1}{A}-\frac{1}{N}}$%.
      * Then the two numbers are combined with standard error propagation, which implies the Gaussian approximation.
   * Added more diagnostic columns to the tables

---+++ Plots
|  *Lambda*  |  *LLR distributions*  |  *CLs plot*  |
|  *2TeV*  |  <img src="%ATTACHURLPATH%/cpsdn20001I2_88.png" alt="cpsdn20001I2_88.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/n8ex1.png" alt="n8ex1.png" width='298' height='286' />  |
|  *3.6TeV*  |  <img src="%ATTACHURLPATH%/cpsdn36001I2_88.png" alt="cpsdn36001I2_88.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/n8ex8.png" alt="n8ex8.png" width='298' height='286' />  |
|  *4.0TeV*  |  <img src="%ATTACHURLPATH%/cpsdn40001I2_88.png" alt="cpsdn40001I2_88.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/n8ex6.png" alt="n8ex6.png" width='298' height='286' />  |
|  *4.05TeV*  |  <img src="%ATTACHURLPATH%/cpsdn40501I2_88.png" alt="cpsdn40501I2_88.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/n8ex5.png" alt="n8ex5.png" width='298' height='286' />  |
|  *4.1TeV*  |  <img src="%ATTACHURLPATH%/cpsdn41001I2_88.png" alt="cpsdn41001I2_88.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/n8ex7.png" alt="n8ex7.png" width='298' height='286' />  |
|  *4.15TeV*  |  <img src="%ATTACHURLPATH%/cpsdn41501I2_88.png" alt="cpsdn41501I2_88.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/n8ex9.png" alt="n8ex9.png" width='298' height='286' />  |
|  *5.0TeV*  |  <img src="%ATTACHURLPATH%/cpsdn50001I2_88.png" alt="cpsdn50001I2_88.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/n8ex2.png" alt="n8ex2.png" width='298' height='286' />  |


---+++ The table
|  *Lambda*  |  *N_{PDS}*  |  *At the CLs point*  ||||  *Frequentist limiting*  |  *Data*  ||||| 
| ^ |  ^ |  *N_{S+B}*  |   *CLb*  |  *CLsb*  |  *LLR value*  |  *LLR value*  |  *LLR value*  |  *N_{S+B}*  |  *CLs*  |  *MC stat. err. on CLs*  |
 |  1.80  |  4000  |  199  |  0.0498  |  0.9963  |  -1.01  |  -0.99  |  -27.75  |  0  |  0.0000  |  0.0000  |
 |  1.85  |  4000  |  199  |  0.0497  |  0.9939  |  -0.82  |  -0.78  |  -25.50  |  0  |  0.0000  |  0.0000  |
 |  1.90  |  4000  |  197  |  0.0494  |  0.9875  |  -1.49  |  -1.41  |  -24.14  |  0  |  0.0005  |  0.0005  |
 |  1.95  |  4000  |  198  |  0.0495  |  0.9906  |  -0.78  |  -0.73  |  -23.32  |  1  |  0.0015  |  0.0015  |
 |  2.00  |  4000  |  196  |  0.0489  |  0.9787  |  -2.04  |  -1.92  |  -21.23  |  0  |  0.0002  |  0.0002  |
 |  2.20  |  4000  |  185  |  0.0462  |  0.9238  |  -2.24  |  -2.03  |  -15.19  |  0  |  0.0000  |  0.0000  |
 |  2.40  |  4000  |  177  |  0.0443  |  0.8856  |  -2.17  |  -1.83  |  -13.98  |  0  |  0.0000  |  0.0000  |
 |  2.60  |  4000  |  184  |  0.0460  |  0.9194  |  -0.29  |  -0.12  |  -10.90  |  0  |  0.0000  |  0.0000  |
 |  2.80  |  4000  |  166  |  0.0414  |  0.8276  |  -0.94  |  -0.71  |  -8.84  |  0  |  0.0000  |  0.0000  |
 |  3.00  |  4000  |  84  |  0.0209  |  0.4188  |  -3.10  |  -1.95  |  -7.02  |  2  |  0.0211  |  0.0162  |
 |  3.20  |  4000  |  63  |  0.0157  |  0.3132  |  -2.99  |  -1.81  |  -5.99  |  2  |  0.0181  |  0.0148  |
 |  3.40  |  4000  |  19  |  0.0047  |  0.0933  |  -3.46  |  -1.67  |  -4.86  |  1  |  0.0265  |  0.0268  |
 |  3.60  |  21000  |  121  |  0.0058  |  0.1153  |  -2.74  |  -1.46  |  -4.00  |  10  |  0.0389  |  0.0123  |
 |  3.80  |  12000  |  28  |  0.0023  |  0.0466  |  -2.90  |  -1.38  |  -3.66  |  2  |  0.0261  |  0.0187  |
 |  4.00  |  16000  |  32  |  0.0020  |  0.0395  |  -3.02  |  -1.47  |  -3.57  |  1  |  0.0057  |  0.0057  |
 |  4.05  |  181000  |  346  |  0.0019  |  0.0382  |  -2.89  |  -1.47  |  -3.41  |  44  |  0.0403  |  0.0063  |
 |  4.10  |  146000  |  178  |  0.0012  |  0.0244  |  -2.91  |  -1.46  |  -3.27  |  35  |  0.0434  |  0.0076  |
 |  4.15  |  35000  |  ---  |  ---  |  ---  |  ---  |  -1.46  |  -3.18  |  15  |  0.0793  |  0.0216  |
 |  4.20  |  4000  |  ---  |  ---  |  ---  |  ---  |  -1.61  |  -3.03  |  4  |  0.1469  |  0.0859  |
 |  5.00  |  4000  |  ---  |  ---  |  ---  |  ---  |  -1.07  |  -1.49  |  11  |  0.4184  |  0.1493  |

---+++ Conclusions
---++++ Why did we see fake rises?
   * the main problem is that we evaluated the significance using the number of PDS in the low-end rise region. The right way to think of it is by analogy to a bump hunt - is there a signficant excess over background? Just using the number of PDS is looking at S+B. In particular, a high B is a reason to distrust an excess, not to trust it :-(
      * it also hurt that the relevant numbers weren't readily available. They are in the tables now.
   * also a mild "look elsewhere" effect
---++++ Which CLs limits should we use?
It is not safe to use CLs limits which are based on small tail probabilities. How small?
   * From these results, anything below ~0.001 is suspect
   * Our description of systematic uncertainties wasn't meant to cover such extreme cases. It's probably good enough till at least 3 sigma --> ~0.0015. It's certainly not good enough at 4 sigma (truncated there) --> ~0.00003. Also the choice of the prior for the JES wasn't checked below ~0.001 (see AmnonHarelStatusReportForStatisticsBoard#New_prior_shapes_for_nuisance_pa)

Going over the crucial lambda values one by one:
   * Though the separation at lambda=3.6TeV is low enough that it is almost power-constrained away, it is certain that a CLs limit exists and the data lies below it, so this value is excluded.
   * 4.0TeV is excluded
   * A CLs limit exists at 4.05TeV. It is probably above the data, so that 4.05 is excluded. I'd guestimate the significance of the previous statement, in terms of MC statistics, at around 2 sigma.
   * A CLs limit exists at 4.1TeV. There is a hint that it is above the data, so that 4.1 is excluded. I'd guestimate the significance of the previous statement, in terms of MC statistics, at around 0.5 sigma.
   * There is a hint that no CLs limit exists at 4.15TeV. If a crucial CLs value exists, the data value can easily (~50%?) lie above it.

So the best choice is to stick with the limits presented in the CWR, and add one more item to the discussion of the difficulties with CLs: 

"CLs sometimes requires estimating extreme tails which stresses the validity of the systematic variations and is difficult using brute force ensemble testing."

Need to decide on the right presentation though...

---++ Improved plots with ensembles as of a bit before CWR (early 21st of September)
---+++ The simple cases
To understand how the CLs limits go away, we'll look at plots of CLs as a function of LLR (for a given new physics model, here contact interactions).

---++++ Easy exclusion
|  *LLR distributions*  |  *CLs plot*  |
|   <img src="%ATTACHURLPATH%/cpsdn20001I2_88_v2.png" alt="cpsdn20001I2_88_v2.png" width='298' height='286' />   |  <img src="%ATTACHURLPATH%/ex1.png" alt="ex1.png" width='298' height='286' />  |
---++++ No exclusion
|  *With systematics (as usual)*  ||  *Only statistical variations*  ||  
|  *LLR distributions*  |  *CLs plot*  |  *LLR distributions*  |  *CLs plot*  |
|   <img src="%ATTACHURLPATH%/cpsdn50001I2_88_v2.png" alt="cpsdn50001I2_88_v2.png" width='298' height='286' />   |  <img src="%ATTACHURLPATH%/ex2.png" alt="ex2.png" width='298' height='286' />  |   <img src="%ATTACHURLPATH%/cpsdn50002I2_88.png" alt="cpsdn50002I2_88.png" width='298' height='286' />   |  <img src="%ATTACHURLPATH%/cls_wo_syst_5TeV.png" alt="cls_wo_syst_5TeV.png" width='298' height='286' />  |
The leftmost "0" is real, but due to a single point from the SM ensemble. The 0,0 points (quite a few, actually) are an artifact that does not effect the code (these plots were meant to be internal to the code...).
---+++ CLs with borderline sensitivity
More than one behavior near the edge of the experimental sensitivity is possible. Here are two simple scenarios. In both scenarios, under both hypotheses (i.e. in both ensembles), the LLR is distributed as a Gaussian and the two distributions are displaced by an amount that descreases as we run out of experimental sensitivity.
   * *1* - the Gaussians have the same width
      * in this scenario, the lower the LLR (below the SM peak), the more the SM is prefered.
      * thus, as %$\lambda$% increases and we run out of experimental sensitivity, the CLs limit exists, and will rapidly drop to %$-\infty$%. However, our ability to determine its correct value will detriorate quickly - brute force ensemble testing is not a suitable tool for learning about the tails of distributions.
      * this is the behavior that was observed in the ICHEP results, where increasing the ensemble size by an order of magnitude extended the limit
   * *2* - the new physics distribution is wider
      * in this scenario, for very low LLR values (well below the SM peak), the SM is no longer prefered.
      * thus, as %$\lambda$% increases and we run out of experimental sensitivity, the CLs limit no longer exists as the plot is always above 0.05
      * this is the behavior observed in the current results - increasing the ensemble size does not extended the limit
---+++ The current borderline sensitivity region
|  *LLR distributions*  |  *CLs plot*  || |
| ^ |  *from 0 to 1*  |  *zoomed in*  | |
|   <img src="%ATTACHURLPATH%/cpsdn40001I2_88_v2.png" alt="cpsdn40001I2_88_v2.png" width='298' height='286' />   |  <img src="%ATTACHURLPATH%/ex3.png" alt="ex3.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/ex6.png" alt="ex6.png" width='298' height='286' />  |
|  *LLR distributions*  |  *CLs plot*  ||  *LLR breakdown*  |
| ^ |  *from 0 to 1*  |  *zoomed in*  |  *by highest non-zero inner bin*  |
|   <img src="%ATTACHURLPATH%/cpsdn40501I2_88_v2.png" alt="cpsdn40501I2_88_v2.png" width='298' height='286' />   |  <img src="%ATTACHURLPATH%/ex4.png" alt="ex4.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/ex5.png" alt="ex5.png" width='298' height='286' />  |  <img src="%ATTACHURLPATH%/more_ex1.png" alt="more_ex1.png" width='298' height='286' />  |
|  *LLR distributions*  |  *CLs plot*  || |
| ^ |  *from 0 to 1*  |  *zoomed in*  |  |
|   <img src="%ATTACHURLPATH%/cpsdn41001I2_88_v2.png" alt="cpsdn41001I2_88_v2.png" width='298' height='286' />   |  ---  |  <img src="%ATTACHURLPATH%/ex7.png" alt="ex7.png" width='298' height='286' />  |  |

---+++ The table
|  *Lambda*  |  *N_{PDS}*  |  *At the CLs point*  ||||  *Frequentist limiting*  |  *Data*  ||||| 
| ^ |  ^ |  *N_{S+B}*  |   *CLb*  |  *CLsb*  |  *LLR value*  |  *LLR value*  |  *LLR value*  |  *N_{S+B}*  |  *CLs*  |  *MC stat. err. on CLs*  |
 |  1.80  |  4000  |  199  |  0.0498  |  0.9963  |  -1.01  |  -0.99  |  -27.75  |  0  |  0.0000  |  0.0000  |
 |  1.85  |  4000  |  199  |  0.0497  |  0.9939  |  -0.82  |  -0.78  |  -25.50  |  0  |  0.0000  |  0.0000  |
 |  1.90  |  4000  |  197  |  0.0494  |  0.9875  |  -1.49  |  -1.41  |  -24.14  |  0  |  0.0005  |  0.0005  |
 |  1.95  |  4000  |  198  |  0.0495  |  0.9906  |  -0.78  |  -0.73  |  -23.32  |  1  |  0.0015  |  0.0015  |
 |  2.00  |  4000  |  196  |  0.0489  |  0.9787  |  -2.04  |  -1.92  |  -21.23  |  0  |  0.0002  |  0.0002  |
 |  2.20  |  4000  |  185  |  0.0462  |  0.9238  |  -2.24  |  -2.03  |  -15.19  |  0  |  0.0000  |  0.0000  |
 |  2.40  |  4000  |  177  |  0.0443  |  0.8856  |  -2.17  |  -1.83  |  -13.98  |  0  |  0.0000  |  0.0000  |
 |  2.60  |  4000  |  184  |  0.0460  |  0.9194  |  -0.29  |  -0.12  |  -10.90  |  0  |  0.0000  |  0.0000  |
 |  2.80  |  4000  |  166  |  0.0414  |  0.8276  |  -0.94  |  -0.71  |  -8.84  |  0  |  0.0000  |  0.0000  |
 |  3.00  |  4000  |  84  |  0.0209  |  0.4188  |  -3.10  |  -1.95  |  -7.02  |  2  |  0.0211  |  0.0162  |
 |  3.20  |  4000  |  63  |  0.0157  |  0.3132  |  -2.99  |  -1.81  |  -5.99  |  2  |  0.0181  |  0.0148  |
 |  3.40  |  4000  |  19  |  0.0047  |  0.0933  |  -3.46  |  -1.67  |  -4.86  |  1  |  0.0265  |  0.0268  |
 |  3.60  |  8000  |  ---  |  ---  |  ---  |  ---  |  -1.59  |  -4.00  |  7  |  0.0803  |  0.0307  |
 |  3.80  |  12000  |  28  |  0.0023  |  0.0466  |  -2.90  |  -1.38  |  -3.66  |  2  |  0.0261  |  0.0187  |
 |  4.00  |  16000  |  32  |  0.0020  |  0.0395  |  -3.02  |  -1.47  |  -3.57  |  1  |  0.0057  |  0.0057  |
 |  4.05  |  12000  |  ---  |  ---  |  ---  |  ---  |  -1.66  |  -3.41  |  6  |  0.0992  |  0.0440  |
 |  4.10  |  8000  |  ---  |  ---  |  ---  |  ---  |  -1.57  |  -3.27  |  5  |  0.1350  |  0.0641  |
 |  4.15  |  4000  |  ---  |  ---  |  ---  |  ---  |  -1.60  |  -3.18  |  4  |  0.2368  |  0.1259  |
 |  4.20  |  4000  |  ---  |  ---  |  ---  |  ---  |  -1.61  |  -3.03  |  4  |  0.1469  |  0.0859  |
 |  5.00  |  4000  |  ---  |  ---  |  ---  |  ---  |  -1.07  |  -1.49  |  11  |  0.4184  |  0.1493  |
---+++ Assorted attachments

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who
png	cls_wo_syst_5TeV.png	r1	manage	8.2 K	2010-09-27 - 21:08	AmnonHarel
png	cpsdn20001I2_88.png	r2 r1	manage	16.2 K	2010-09-30 - 20:42	AmnonHarel
png	cpsdn20001I2_88_v2.png	r1	manage	16.2 K	2010-09-30 - 20:30	AmnonHarel
png	cpsdn36001I2_88.png	r1	manage	17.5 K	2010-09-30 - 20:38	AmnonHarel
png	cpsdn40001I2_88.png	r2 r1	manage	16.7 K	2010-09-30 - 20:43	AmnonHarel
png	cpsdn40001I2_88_v2.png	r1	manage	16.7 K	2010-09-30 - 20:31	AmnonHarel
png	cpsdn40501I2_88.png	r3 r2 r1	manage	15.5 K	2010-09-30 - 20:47	AmnonHarel
png	cpsdn40501I2_88_v2.png	r1	manage	18.0 K	2010-09-30 - 20:33	AmnonHarel
png	cpsdn41001I2_88.png	r2 r1	manage	14.8 K	2010-09-30 - 20:45	AmnonHarel
png	cpsdn41001I2_88_v2.png	r1	manage	17.9 K	2010-09-30 - 20:31	AmnonHarel
png	cpsdn41501I2_88.png	r1	manage	17.7 K	2010-09-30 - 20:50	AmnonHarel
png	cpsdn50001I2_88.png	r2 r1	manage	17.1 K	2010-09-30 - 20:44	AmnonHarel
png	cpsdn50001I2_88_v2.png	r1	manage	17.1 K	2010-09-30 - 20:30	AmnonHarel
png	cpsdn50002I2_88.png	r1	manage	16.6 K	2010-09-27 - 21:03	AmnonHarel
png	ex1.png	r3 r2 r1	manage	10.9 K	2010-09-30 - 19:00	AmnonHarel
png	ex2.png	r3 r2 r1	manage	12.9 K	2010-09-30 - 19:00	AmnonHarel
png	ex3.png	r3 r2 r1	manage	12.1 K	2010-09-30 - 19:03	AmnonHarel
png	ex4.png	r2 r1	manage	12.7 K	2010-09-30 - 19:01	AmnonHarel
png	ex5.png	r2 r1	manage	14.1 K	2010-09-30 - 19:01	AmnonHarel
png	ex6.png	r2 r1	manage	11.4 K	2010-09-30 - 19:01	AmnonHarel
png	ex7.png	r1	manage	11.3 K	2010-09-30 - 19:02	AmnonHarel
png	more_ex1.png	r1	manage	20.9 K	2010-09-28 - 17:21	AmnonHarel
png	n8ex1.png	r1	manage	10.9 K	2010-09-30 - 19:20	AmnonHarel
png	n8ex2.png	r1	manage	12.9 K	2010-09-30 - 19:20	AmnonHarel
png	n8ex3.png	r1	manage	12.1 K	2010-09-30 - 19:20	AmnonHarel
png	n8ex4.png	r1	manage	12.4 K	2010-09-30 - 19:20	AmnonHarel
png	n8ex5.png	r1	manage	13.9 K	2010-09-30 - 19:21	AmnonHarel
png	n8ex6.png	r1	manage	11.4 K	2010-09-30 - 19:21	AmnonHarel
png	n8ex7.png	r1	manage	10.6 K	2010-09-30 - 19:21	AmnonHarel
png	n8ex8.png	r1	manage	10.6 K	2010-09-30 - 19:35	AmnonHarel
png	n8ex9.png	r1	manage	13.8 K	2010-09-30 - 20:50	AmnonHarel

Topic revision: r7 - 2010-09-30 - AmnonHarel

Main

Webs

Public webs

Welcome Guest

- Cern Search
- TWiki Search
- Google Search
Main All webs

Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback