Difference: AmnonHarelCLsInDCR (11 vs. 12)

Revision 122010-10-07 - AmnonHarel

Line: 1 to 1
 
META TOPICPARENT name="AmnonHarelDijetCentralityRatio"
-- AmnonHarel - 12-Sep-2010
Line: 13 to 13
 None of these really matters at the end, but for completeness:
  • When the number of pseudo datasets (PDSs) that underlie either $CL_{s+b}$ or $CL_b$ at the limit is below 10, we consider the determination of the limit unreliable and do not quote it.
    • For the final results, we generate enough PDFs at the crucial points in phase space to prevent this requirement from effecting the results, except where it clearly removes statistical noise (see $\lambda=5$TeV example below)
Changed:
<
<
  • Since the test statistic is the log likelihood ratio, with or without systematics, when we define an excluded region of the LLR, it will normally be simply connected and include $-\infty$ (the sign convention is such that this is the value most unlike the new physics scenario). We considered the possibility that when the LLR does not include systematics, this may fail to hold, in which case we would quote only the uppermost excluded LLR in the lowest exclusion region. Currently, we have no reason to believe this abnormal situation arises in our measurement.
>
>
  • Since the test statistic is the log likelihood ratio, with or without systematics, when we define an excluded region of the LLR, it will normally be simply connected and include $-\infty$ (the sign convention is such that this is the value most unlike the new physics scenario). We considered the possibility that when the LLR does not include systematics, this may fail to hold, in which case we would quote only the uppermost excluded LLR in the lowest exclusion region. Currently, we have no reason to believe this abnormal situation arises in our measurement except as a statistical fluctuations (from finite ensemble size).
 

Final results

What changed

  • Found a bug that greatly reduced the shift uncertainty in all results, and another that reduced the absolute JES in the results sent to the statistic board during CWR. The three leading uncertainties were unaffected by these bugs, so the results barely change. It is only due to the finicky nature of CLs limits in the regime we're at, that these tiny effects must be resolved.
Line: 22 to 22
 
  • Decided on the stopping conditions.
    • lambda value excluded / allowed at 2 sigma level
    • CLs value (i.e. the confidence level of the exclusion) is known at 0.5% accuracy
Changed:
<
<
      • note that this condition is neccesary - without it at the key lambda value, where the exclusion truly is 95%, only a large statistical fluctuation will yield 2 sigma separation between the true CLs level and the desired 5%. Furthermore, closure tests of frequentist methods often show inaccuracies at the 1% level.
>
>
      • note that this condition is neccesary - without it at the key lambda value, where the exclusion truly is 95%, only a large statistical fluctuation will yield 2 sigma separation between the true CLs level and the desired 5%.
      • as a benchmark, closure tests of common frequentist methods often show inaccuracies at the 1% level.
 

What drives the low LLR tails?

Changed:
<
<
The earlier results:
>
>
The following earlier results hinted that statistical effects drive the low tails:
 
  • the nuisance parameters in the tails are as one would expect from their correlations with LLR in the bulk. They are only mildly effected.
  • the plot of the ensemble broken down by the last non-empty inner bin:
    more_ex1.png
Deleted:
<
<
both hinted that statistical effects drive the low tails.
  To check this further, we took a key ensemble and filtered out PDSs where any of the nuisance parameters is more than 2 sigma away from its nominal value, and checked how this effects the LLR tails.
plot_syst_tail.png
Line: 50 to 50
 
4.05TeV cpsdn40501I2_88.png v7ex5.png
4.1TeV cpsdn41001I2_88.png v7ex7.png
4.15TeV cpsdn41501I2_88.png v7ex9.png
Added:
>
>
4.2TeV cpsdn42001I2_88.png v7ex10.png
 
5.0TeV cpsdn50001I2_88.png v7ex2.png

  • The two peak structure visible at 5.0TeV is due to having one very high mass event, which can be either inner or outer. See also the "old" plots of 4.05TeV below, with a breakdown by the last non-empty inner bin.
Line: 74 to 75
 
3.80 92098 187 0.0020 0.0406 -3.01 -1.43 -3.66 23 0.0321 0.0068
4.00 472238 164 0.0003 0.0069 -3.56 -1.69 -3.57 151 0.0489 0.0041
4.05 413663 12 0.0000 0.0006 -3.83 -1.66 -3.41 156 0.0566 0.0047
Changed:
<
<
4.10 209056 49 0.0002 0.0047 -3.32 -1.62 -3.27 71 0.0551 0.0068
4.15 218000 30 0.0001 0.0028 -3.32 -1.64 -3.18 77 0.0572 0.0068
>
>
4.10 389236 46 0.0001 0.0024 -3.44 -1.62 -3.27 151 0.0636 0.0054
4.15 420161 --- --- --- --- -1.63 -3.18 171 0.0670 0.0054
 
4.20 181056 4 0.0000 0.0005 -3.34 -1.55 -3.03 71 0.0701 0.0088
5.00 25000 --- --- --- --- -1.08 -1.49 48 0.2426 0.0418

Conclusions

Going over the crucial lambda values one by one:
Changed:
<
<
  • Though the separation at lambda=3.6TeV is low enough that it is almost power-constrained away, this lambda value is clearly excluded.
  • 3.8TeV is excluded
  • The CLs at 4.0TeV is known to an accuracy of 0.5%, and at that accuracy it is below 5%, so that 4.0TeV is excluded
  • The CLs at 4.05TeV is known to an accuracy of 0.5%, and at that accuracy it is above 5%, so that 4.05TeV is not excluded
  • A CLs critical value exists at 4.1TeV. excluded?
  • 4.15TeV?
  • A critical CLs value probably exists for 4.2TeV, but it is above the data (at 2 sigma of MC statistics), so 4.2TeV is not excluded.
  • If a critical CLs value exists at 5TeV, the data value is well above it and 5TeV can not be excluded.
>
>
  • 3.6TeV: Though the separation is low enough that it is almost power-constrained away, this lambda value is clearly excluded.
  • 3.8TeV: is excluded
  • 4.0TeV:The data CLs is known to an accuracy of 0.5%, and at that accuracy it is below 5%. So that 4.0TeV is excluded
  • 4.05TeV:The data CLs at 4.05TeV is known to an accuracy of 0.5%, and at that accuracy it is above 5%, so that 4.05TeV is not excluded
  • 4.1TeV: a critical CLs value probably exists at 4.1TeV. Our data lies above the possible critical value (at >2 sigma of MC statistics), so 4.1TeV is not excluded
  • 4.15TeV:If a critical CLs value exists, the data value is above it (at 3 sigma of MC statistics) and 4.15TeV can not be excluded.
  • 4.2TeV:Some indication that a critical CLs value exists, but the data value is above it (at >2 sigma of MC statistics), so 4.2TeV is not excluded.
  • 5.0TeV:If a critical CLs value exists, the data value is well above it and 5TeV can not be excluded.

So, as usual, we exclude lambda values less than equal to 4.0TeV.

What is new, is that CLs exclusion value are available at 4.05, 4.1, and 4.2 TeV.

Presentation

Continue CLs line out to 4.1TeV. This shows the crossover, and should satisfy the reader looking for physics. For the statistics aficionado, the presence of an exclusion at 4.2 is interesting, but it would be difficult to communicate this visually in the graph, and it would be distracting from the main results of the paper. Presumably this information will be made public in a table format for such needs. Given such a presentation, it is not clear that any additional text is required. In short - as far as can be shown with our tools this is a plain-vanilla CLs situation, and it should be shown as such.
 

Updated results during CWR

Generated an obscene amount of additional PDSs...
Line: 226 to 232
 
META FILEATTACHMENT attachment="ex5.png" attr="" comment="" date="1285866096" name="ex5.png" path="ex5.png" size="14468" stream="ex5.png" tmpFilename="/usr/tmp/CGItemp40072" user="aharel" version="2"
META FILEATTACHMENT attachment="ex6.png" attr="" comment="" date="1285866113" name="ex6.png" path="ex6.png" size="11674" stream="ex6.png" tmpFilename="/usr/tmp/CGItemp39869" user="aharel" version="2"
META FILEATTACHMENT attachment="cpsdn20001I2_88.png" attr="" comment="" date="1286463800" name="cpsdn20001I2_88.png" path="cpsdn20001I2_88.png" size="16263" stream="cpsdn20001I2_88.png" tmpFilename="/usr/tmp/CGItemp34118" user="aharel" version="3"
Changed:
<
<
META FILEATTACHMENT attachment="cpsdn40001I2_88.png" attr="" comment="" date="1286463840" name="cpsdn40001I2_88.png" path="cpsdn40001I2_88.png" size="16198" stream="cpsdn40001I2_88.png" tmpFilename="/usr/tmp/CGItemp34084" user="aharel" version="3"
META FILEATTACHMENT attachment="cpsdn40501I2_88.png" attr="" comment="" date="1286463865" name="cpsdn40501I2_88.png" path="cpsdn40501I2_88.png" size="15475" stream="cpsdn40501I2_88.png" tmpFilename="/usr/tmp/CGItemp34160" user="aharel" version="4"
META FILEATTACHMENT attachment="cpsdn50001I2_88.png" attr="" comment="" date="1285872244" name="cpsdn50001I2_88.png" path="cpsdn50001I2_88.png" size="17545" stream="cpsdn50001I2_88.png" tmpFilename="/usr/tmp/CGItemp43868" user="aharel" version="2"
>
>
META FILEATTACHMENT attachment="cpsdn40001I2_88.png" attr="" comment="" date="1286486561" name="cpsdn40001I2_88.png" path="cpsdn40001I2_88.png" size="16652" stream="cpsdn40001I2_88.png" tmpFilename="/usr/tmp/CGItemp54047" user="aharel" version="4"
META FILEATTACHMENT attachment="cpsdn40501I2_88.png" attr="" comment="" date="1286486458" name="cpsdn40501I2_88.png" path="cpsdn40501I2_88.png" size="16112" stream="cpsdn40501I2_88.png" tmpFilename="/usr/tmp/CGItemp53881" user="aharel" version="5"
META FILEATTACHMENT attachment="cpsdn50001I2_88.png" attr="" comment="" date="1286486309" name="cpsdn50001I2_88.png" path="cpsdn50001I2_88.png" size="18409" stream="cpsdn50001I2_88.png" tmpFilename="/usr/tmp/CGItemp54083" user="aharel" version="3"
 
META FILEATTACHMENT attachment="cpsdn50002I2_88.png" attr="" comment="" date="1286463961" name="cpsdn50002I2_88.png" path="cpsdn50002I2_88.png" size="17029" stream="cpsdn50002I2_88.png" tmpFilename="/usr/tmp/CGItemp34026" user="aharel" version="1"
META FILEATTACHMENT attachment="cls_wo_syst_5TeV.png" attr="" comment="" date="1285614486" name="cls_wo_syst_5TeV.png" path="cls_wo_syst_5TeV.png" size="8392" stream="cls_wo_syst_5TeV.png" tmpFilename="/usr/tmp/CGItemp36201" user="aharel" version="1"
META FILEATTACHMENT attachment="more_ex1.png" attr="" comment="" date="1285687304" name="more_ex1.png" path="more_ex1.png" size="21436" stream="more_ex1.png" tmpFilename="/usr/tmp/CGItemp37903" user="aharel" version="1"
META FILEATTACHMENT attachment="ex7.png" attr="" comment="" date="1285866171" name="ex7.png" path="ex7.png" size="11592" stream="ex7.png" tmpFilename="/usr/tmp/CGItemp37481" user="aharel" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="cpsdn41001I2_88.png" attr="" comment="" date="1286463899" name="cpsdn41001I2_88.png" path="cpsdn41001I2_88.png" size="14939" stream="cpsdn41001I2_88.png" tmpFilename="/usr/tmp/CGItemp34087" user="aharel" version="3"
>
>
META FILEATTACHMENT attachment="cpsdn41001I2_88.png" attr="" comment="" date="1286486417" name="cpsdn41001I2_88.png" path="cpsdn41001I2_88.png" size="15877" stream="cpsdn41001I2_88.png" tmpFilename="/usr/tmp/CGItemp53798" user="aharel" version="4"
 
META FILEATTACHMENT attachment="n8ex1.png" attr="" comment="" date="1285867208" name="n8ex1.png" path="n8ex1.png" size="11118" stream="n8ex1.png" tmpFilename="/usr/tmp/CGItemp40548" user="aharel" version="1"
META FILEATTACHMENT attachment="n8ex2.png" attr="" comment="" date="1285867227" name="n8ex2.png" path="n8ex2.png" size="13214" stream="n8ex2.png" tmpFilename="/usr/tmp/CGItemp40763" user="aharel" version="1"
META FILEATTACHMENT attachment="n8ex3.png" attr="" comment="" date="1285867240" name="n8ex3.png" path="n8ex3.png" size="12374" stream="n8ex3.png" tmpFilename="/usr/tmp/CGItemp40481" user="aharel" version="1"
Line: 249 to 255
 
META FILEATTACHMENT attachment="cpsdn40501I2_88_v2.png" attr="" comment="" date="1285871632" name="cpsdn40501I2_88_v2.png" path="cpsdn40501I2_88_v2.png" size="18469" stream="cpsdn40501I2_88_v2.png" tmpFilename="/usr/tmp/CGItemp40463" user="aharel" version="1"
META FILEATTACHMENT attachment="cpsdn36001I2_88.png" attr="" comment="" date="1286463821" name="cpsdn36001I2_88.png" path="cpsdn36001I2_88.png" size="15174" stream="cpsdn36001I2_88.png" tmpFilename="/usr/tmp/CGItemp34172" user="aharel" version="2"
META FILEATTACHMENT attachment="n8ex9.png" attr="" comment="" date="1285872624" name="n8ex9.png" path="n8ex9.png" size="14122" stream="n8ex9.png" tmpFilename="/usr/tmp/CGItemp40491" user="aharel" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="cpsdn41501I2_88.png" attr="" comment="" date="1286463931" name="cpsdn41501I2_88.png" path="cpsdn41501I2_88.png" size="15096" stream="cpsdn41501I2_88.png" tmpFilename="/usr/tmp/CGItemp34158" user="aharel" version="2"
>
>
META FILEATTACHMENT attachment="cpsdn41501I2_88.png" attr="" comment="" date="1286486381" name="cpsdn41501I2_88.png" path="cpsdn41501I2_88.png" size="15701" stream="cpsdn41501I2_88.png" tmpFilename="/usr/tmp/CGItemp53782" user="aharel" version="3"
 
META FILEATTACHMENT attachment="plot_syst_tail.png" attr="" comment="" date="1286449190" name="plot_syst_tail.png" path="plot_syst_tail.png" size="21593" stream="plot_syst_tail.png" tmpFilename="/usr/tmp/CGItemp30707" user="aharel" version="1"
META FILEATTACHMENT attachment="cpsdn41501I2_88_v4.png" attr="" comment="" date="1286461758" name="cpsdn41501I2_88_v4.png" path="cpsdn41501I2_88_v4.png" size="18166" stream="cpsdn41501I2_88_v4.png" tmpFilename="/usr/tmp/CGItemp30718" user="aharel" version="1"
META FILEATTACHMENT attachment="cpsdn40501I2_88_v4.png" attr="" comment="" date="1286461780" name="cpsdn40501I2_88_v4.png" path="cpsdn40501I2_88_v4.png" size="15826" stream="cpsdn40501I2_88_v4.png" tmpFilename="/usr/tmp/CGItemp30752" user="aharel" version="1"
Line: 258 to 264
 
META FILEATTACHMENT attachment="cpsdn36001I2_88_v4.png" attr="" comment="" date="1286461877" name="cpsdn36001I2_88_v4.png" path="cpsdn36001I2_88_v4.png" size="17962" stream="cpsdn36001I2_88_v4.png" tmpFilename="/usr/tmp/CGItemp30800" user="aharel" version="1"
META FILEATTACHMENT attachment="cpsdn20001I2_88_v4.png" attr="" comment="" date="1286461906" name="cpsdn20001I2_88_v4.png" path="cpsdn20001I2_88_v4.png" size="16612" stream="cpsdn20001I2_88_v4.png" tmpFilename="/usr/tmp/CGItemp30663" user="aharel" version="1"
META FILEATTACHMENT attachment="cpsdn41001I2_88_v4.png" attr="" comment="" date="1286461938" name="cpsdn41001I2_88_v4.png" path="cpsdn41001I2_88_v4.png" size="15198" stream="cpsdn41001I2_88_v4.png" tmpFilename="/usr/tmp/CGItemp30812" user="aharel" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="v7ex9.png" attr="" comment="" date="1286463017" name="v7ex9.png" path="v7ex9.png" size="13863" stream="v7ex9.png" tmpFilename="/usr/tmp/CGItemp30903" user="aharel" version="1"
META FILEATTACHMENT attachment="v7ex8.png" attr="" comment="" date="1286464529" name="v7ex8.png" path="v7ex8.png" size="11797" stream="v7ex8.png" tmpFilename="/usr/tmp/CGItemp33724" user="aharel" version="2"
META FILEATTACHMENT attachment="v7ex7.png" attr="" comment="" date="1286464463" name="v7ex7.png" path="v7ex7.png" size="12234" stream="v7ex7.png" tmpFilename="/usr/tmp/CGItemp33367" user="aharel" version="2"
META FILEATTACHMENT attachment="v7ex6.png" attr="" comment="" date="1286464556" name="v7ex6.png" path="v7ex6.png" size="13792" stream="v7ex6.png" tmpFilename="/usr/tmp/CGItemp33471" user="aharel" version="2"
META FILEATTACHMENT attachment="v7ex5.png" attr="" comment="" date="1286463226" name="v7ex5.png" path="v7ex5.png" size="13914" stream="v7ex5.png" tmpFilename="/usr/tmp/CGItemp30885" user="aharel" version="1"
META FILEATTACHMENT attachment="v7ex1.png" attr="" comment="" date="1286463249" name="v7ex1.png" path="v7ex1.png" size="11121" stream="v7ex1.png" tmpFilename="/usr/tmp/CGItemp30720" user="aharel" version="1"
META FILEATTACHMENT attachment="v7ex2.png" attr="" comment="" date="1286463265" name="v7ex2.png" path="v7ex2.png" size="12771" stream="v7ex2.png" tmpFilename="/usr/tmp/CGItemp30658" user="aharel" version="1"
>
>
META FILEATTACHMENT attachment="v7ex9.png" attr="" comment="" date="1286483920" name="v7ex9.png" path="v7ex9.png" size="12251" stream="v7ex9.png" tmpFilename="/usr/tmp/CGItemp53213" user="aharel" version="2"
META FILEATTACHMENT attachment="v7ex8.png" attr="" comment="" date="1286483967" name="v7ex8.png" path="v7ex8.png" size="11685" stream="v7ex8.png" tmpFilename="/usr/tmp/CGItemp53312" user="aharel" version="3"
META FILEATTACHMENT attachment="v7ex7.png" attr="" comment="" date="1286483998" name="v7ex7.png" path="v7ex7.png" size="11882" stream="v7ex7.png" tmpFilename="/usr/tmp/CGItemp53030" user="aharel" version="3"
META FILEATTACHMENT attachment="v7ex6.png" attr="" comment="" date="1286484029" name="v7ex6.png" path="v7ex6.png" size="13949" stream="v7ex6.png" tmpFilename="/usr/tmp/CGItemp53372" user="aharel" version="3"
META FILEATTACHMENT attachment="v7ex5.png" attr="" comment="" date="1286484053" name="v7ex5.png" path="v7ex5.png" size="13807" stream="v7ex5.png" tmpFilename="/usr/tmp/CGItemp53446" user="aharel" version="2"
META FILEATTACHMENT attachment="v7ex1.png" attr="" comment="" date="1286484097" name="v7ex1.png" path="v7ex1.png" size="11121" stream="v7ex1.png" tmpFilename="/usr/tmp/CGItemp53277" user="aharel" version="1"
META FILEATTACHMENT attachment="v7ex2.png" attr="" comment="" date="1286484074" name="v7ex2.png" path="v7ex2.png" size="12771" stream="v7ex2.png" tmpFilename="/usr/tmp/CGItemp53256" user="aharel" version="1"
META FILEATTACHMENT attachment="v7ex10.png" attr="" comment="" date="1286483946" name="v7ex10.png" path="v7ex10.png" size="12312" stream="v7ex10.png" tmpFilename="/usr/tmp/CGItemp50469" user="aharel" version="1"
META FILEATTACHMENT attachment="cpsdn42001I2_88.png" attr="" comment="" date="1286486118" name="cpsdn42001I2_88.png" path="cpsdn42001I2_88.png" size="15250" stream="cpsdn42001I2_88.png" tmpFilename="/usr/tmp/CGItemp50599" user="aharel" version="1"
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback