TWiki> Main Web>TWikiUsers>AmnonHarel>AmnonHarelJetRatio>AmnonHarelStatusReportForStatisticsBoard (2010-04-28, AmnonHarel) EditAttachPDF

-- AmnonHarel - 17-Mar-2010
## New prior shapes for nuisance parameters

## 19-Apr-2010 Information

### Introduction and references

Some material in preparation of the April 20th meeting.
#### Additional presentations

### The preferred technique

(with a few questions to the SB interleaved)

### Additional statistical analysis of high mass region?

We would appreciate your input on whether any additional test statistics that answer the question:
"Is the data above the Tevatron limits consistent with our SM predictions?" ### Additional statistical analysis of low mass region?

There are two more, closely-related questions that can be asked, and we would appreciate your input on them. ### Additional information

#### Input models

## 17-Mar-2010 Information

### Selected documentation:

### Now what?

I see two main options:

The statistics board asked for gamma and lognormal priors. These distributions have 3 degrees of freedom, two of which are fixed by the requirements of given mean and RMS. From a literature scan I found that the common practice is to assume mu=0, where I use the variable names used in ROOT's TMath.

- lognormal - as in top_statistics (D0 note 5817, formula 5.8), Collie, and arXiv 0202013
- gamma - as in http://itl.nist.gov/div898/handbook/apr/section1/apr165.htm (contrasting with http://www.itl.nist.gov/div898/handbook/eda/section3/eda366b.htm).

To see what these look like I used the attached priors.c (may require ROOT5.26 to run). Some output examples are:

mean: | 1 | 1 | 10 |
---|---|---|---|

width: | 0.01 | 0.1 | 1 |

plot: |

The numbers in parenthesis are the errors on the last digits. The first number is the mean of the histogram, and the 2nd is the RMS. The Gauss and LogNormal histogram is filled by transforming a normal variable, rather than from the plotted function.

This supplements Robert's email and uses the references he provided:

**[1]** my talk Thursday - http://indico.cern.ch/conferenceDisplay.py?confId=91456

**[2]** the D0 paper - http://arxiv.org/abs/hep-ex/9807014

**[3]** the 2006-2007 CMS studies - http://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2007_039_v4.pdf

Another reference I will use is

**[4]** the 2009 CMS studies - http://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2009_161_v1.pdf

Our preferred limit setting technique for ICHEP, and the publication plan that motivates it, are described in [1]. This plan hinges on our expectations that for ICHEP the transition between the statistics-dominated and statistical-dominated regions and the transition between the regions explored by the Tevatron and those we will explore for the first time fall very close to each other, around 800GeV. It is clear that further development will be needed for future publication.

In particular, the systematic effects have large bin-to-bin correlations, which should dominate any quantitative treatment of the systematics (see [2], Fig. 2, off-diagonal elements). In particular, the relative JES effects at a particular jet pT should be propagated to a range of Mjj bins.

- Statistics - comparing AN2009/161's methods with LLR
- Analysis overview as of AN2009/161
- Jim's analysis update - everything but statistics

- The test statistics are log likelihood ratios that do not take systematic effects into account.
- the likelihood used only bins above the Tevatron limits, e.g., above Mjj=800GeV, and in the case of contact interactions only bins below the contact interaction scale.
- as per the standard technique for a Poisson ratio, the distributions are conditioned on the observed total counts.

- Use ensemble tests to account for the systematic uncertainties and derive "hybrid" 95%CL limits:
- leading systematics are the relative and absolute JES. The corresponding nuisance parameters are drawn from Gaussian distributions.
- we set limits as in Section 3.4 of [4].
- the delay was to verify that this procedure holds when including the leading systematics correctly

- in [4] we used the limit corresponding to the median of the SM distributions as the "predicted limit". It is not an expectation value. We're toying with adding +/- 1 sigma and 2 sigma bands.
- Any objections to basing them all on the corresponding SM quantiles?
- this assumes negligible uncertainties on the curve used to translate the observable into a limit. Only statistical uncertainties are relevant, and they can be reduced arbitrarily by increasing the size of the NP ensembles. We haven't yet chosen how to estimate the uncertainty on the 95th quantile, and welcome any SB input.

- For each m_q* we can plot (x-axis is m_q*, y-axis is x-section):
- the SM predicted limit,
- preferably with +/- 1 and 2 sigma bands

- the predicted NP x-section,
- can add the predicted NP limit, if worth the visual clutter

- the observed limit Since the q* peaks are local, the q*/SM log likelihood ratios have similar localities, and such a plot provide a possible answer to the question: is the data consistent with the SM?

- the SM predicted limit,
- For each lambda we can plot (x-axis is m_q*, y-axis is LLR):
- the SM predicted LLR (dashed line below),
- preferably with +/- 1 and 2 sigma bands

- the 95% quantiles for the corresponding contact interaction scenario (dark red line and points),
- the observed LLR (shown for a "golden dataset" below by the solid black line)
- Examples of visual summaries of the statistical analysis of contact interaction:

- the SM predicted LLR (dashed line below),

Without systematics | With systematics |
---|---|

- must be published (my views against this are in [1], slide 9)
- is interesting enough to include in a paper
- this probably depends on how difficult it will be to present such a test statistic. In [4] we developed a possible generic test statistic to answer that question ("G").

- this region will be statistics dominated in ICHEP

- "Is the data below the Tevatron limits consistent with our SM predictions?" - generic null hypothesis testing
- This region is dominated by systematics, so any method that ignores systematics will likely yield "no" here.

- "Can the data below the Tevatron limits be fit by the expected systematic variations of our SM predictions?" - goodness of fit statistic
- This is one method to answer the question above. If the fit is limited to the high statistics regime, even a "standard" chi^2 test may do well enough, as the binomial distributions for such large numbers can easily be approximated to Gaussians. Another variation is to perform the test on the inner and outer spectra, taking into account their correlation. Perhaps this is what was done in [2] (note the diagonal correlations in their Fig. 2).

To a large extend, this question will be answered in the "R" plot, which summarizes our data and predictions. We will draw Clopper-Pearson intervals on the data points, and the systematics on the predictions, which allow the reader to see any significant deviation.

It would be nice to have a statistical statement that covers both the high-statistics, Tevatron-excluded-NP region and the low-statistics, we're-exploring-new-energies region. Especially as the fact that the cross over between the two regimes is basically at the same point is a temporary coincidence expected for an ICHEP result.

I plan to explore the 2nd approach, using a profile likelihood fit with the "usual" binomial statistics. But we do not yet have a consensus on whether this is necessary for an ICHEP publication.

- Currently, we do not fit the (nuisance parameters of the) prediction to the data, and we do not have a detailed model of the bin-to-bin correlations. We have not yet discussed internally whether such a detailed description of the systematics is realistic for the ICHEP timescale.

- Contact interaction models:

- q* models (rough version from 2010-Apr-18):

Following the discussion just before midnight in the 16-Mar statistics board meeting, here is a status report for the statistics in the di-jet ratio, with context and links to notes and talks.

There are several reasons to adopt for an ICHEP analysis different tools, and a different approach, than was taken so far in the dijet ratio. This demotes some of the documentation to a documentation of historical attempts - if you wish to skip that, don't read sections 3.2 to 3.4 in the analysis note, and just look for "LLR" plots in the presentations, ignoring other test statistics.

Given an observable sensitive to a wide range of models, I started by looking for a test statistic that maintains that generality. This is documented in the AN. The presentation shows extensions, and also that systematic uncertainties have a huge effect (*). This shifts the focus to how to handle the systematics, and indicates we should switch to using the more familiar log-likelihood ratios (LLRs) so we can focus on the systematics.

(*) This is a very preliminary and unexpected result which I haven't fully debugged yet.

Given Jim's 7TeV numbers, and assuming ~3pb-1 for ICHEP, we are looking at systematic uncertainties of roughly 0.03 (absolute on R) and statistical uncertainties will dominate (i.e. >0.06) from around 800GeV. The Tevatron exclusions are Lambda<2.7TeV and Mq*<0.87TeV. So we can analyze our data starting from 870GeV, where systematics should not dominate, and ruling out a q* of 1TeV is in play (see Fig. 11 in AN, though the wrong Ecm is used), and so is a 3TeV contact interaction (see Jim's presentation).

To put it differently - for ICHEP we have the option of doing the analysis so that statistics dominates IN EVERY SINGLE BIN OF INTEREST.

- covering only this case will leave us badly placed if some systematic effect turns out bigger than expected and increases the systematics by more than a factor of 2.

- AN 2009/161
- In the appendix the definition of the "N" statistics is wrong - the correction was given in the "statistics presentation" below
- a presentation of the AN material

- Dijet with 7TeV - Jim's talk in Exotica Multijets Working Group Meeting)
- Statistics presentation (v12 is a corrected version of talk in Exotica Multijets Working Group Meeting)
- Note on systematic variations
- For those that wish to review the earlier stages of the statistical treatment, these talks might be relevant:

- use the current tools, with local limit setting for q* and an LLR, and ignoring all bins below ~800GeV.
- Should work.
- Latest test in a different "edge of statistical strength" scenario (in presentation) unexpectedly failed - some more tests and debugging is in order.

- I will test with contact interactions early next week, and with Jim's help I'll also prepare the q* results later next week.
- if the current tools are insufficient, I expect that incorporating the basic uncertainty in the test statistic, e.g. by doing a fit for the relative JES within each likelihood ("log profile-likelihood ratio"?) will handle the leading, and all other systematics.

- Should work.
- start over in RooStats, using only likelihoods.
- re-examining RooFit and RooStat's latest versions with excellent help from Genadi Kukarzev, it seems that it should be possible to do our analysis in RooStats. Some of the things we need seem to be a bit unusual / supported in a cumbersome way, which I find a bit worrisome. But hopefully all works as it should.
- the features we need are:
- ensemble testing by histograms (without individual events) - supported in latest version
- variable binning - supported in a round-about way (seems we need to name each bin)
- using a different statistics model and ensemble-generation models
- defining the test statistic as a function of functions of the data, rather than the data itself

- the last two (c & d) look supported and that they should work cleanly, but they also look like unusual and complicated use cases.

- thus migrating the code to RooStats is possible but not be trivial, and is detrimental to having an ICHEP result.

I | Attachment | History | Action | Size | Date | Who | Comment |
---|---|---|---|---|---|---|---|

png | climLLR0_4K.png | r1 | manage | 12.1 K | 2010-04-19 - 04:07 | AmnonHarel | Limits with no systematics |

png | climLLR0_s2_1K.png | r1 | manage | 12.2 K | 2010-04-19 - 04:21 | AmnonHarel | Without systematics, 1K PDSs per ensemble |

png | climLLR1_1K.png | r2 r1 | manage | 14.4 K | 2010-04-20 - 20:54 | AmnonHarel | Limits with systematics |

png | climLLR2_4K.png | r2 r1 | manage | 15.9 K | 2010-04-20 - 20:52 | AmnonHarel | Limits without systematics |

png | plot_ci_model.png | r3 r2 r1 | manage | 19.3 K | 2010-04-20 - 19:32 | AmnonHarel | Contact interaction models |

png | plot_qs_model.png | r2 r1 | manage | 18.2 K | 2010-04-20 - 18:34 | AmnonHarel | q* models (rough version from 2010-Apr-18) |

c | priors.c | r2 r1 | manage | 3.7 K | 2010-04-28 - 19:34 | AmnonHarel | Visualization of nuisance parameter priors |

png | priors_10_1.png | r2 r1 | manage | 18.4 K | 2010-04-28 - 19:34 | AmnonHarel | m=10,w=1 |

png | priors_1_001.png | r2 r1 | manage | 20.7 K | 2010-04-28 - 19:35 | AmnonHarel | m=1,w=0.01 |

png | priors_1_01.png | r2 r1 | manage | 21.5 K | 2010-04-28 - 19:35 | AmnonHarel | m=1,w=0.1 |

ratio_sys_var_note.pdf | r1 | manage | 74.6 K | 2010-03-17 - 19:22 | AmnonHarel | Note on systematic variations |

Topic revision: r10 - 2010-04-28 - AmnonHarel

**Webs**

- ABATBEA
- ACPP
- ADCgroup
- AEGIS
- AfricaMap
- AgileInfrastructure
- ALICE
- AliceEbyE
- AliceSPD
- AliceSSD
- AliceTOF
- AliFemto
- ALPHA
- ArdaGrid
- ASACUSA
- AthenaFCalTBAna
- Atlas
- AtlasLBNL
- AXIALPET
- CAE
- CALICE
- CDS
- CENF
- CERNSearch
- CLIC
- Cloud
- CloudServices
- CMS
- Controls
- CTA
- CvmFS
- DB
- DefaultWeb
- DESgroup
- DPHEP
- DM-LHC
- DSSGroup
- EGEE
- EgeePtf
- ELFms
- EMI
- ETICS
- FIOgroup
- FlukaTeam
- Frontier
- Gaudi
- GeneratorServices
- GuidesInfo
- HardwareLabs
- HCC
- HEPIX
- ILCBDSColl
- ILCTPC
- IMWG
- Inspire
- IPv6
- IT
- ItCommTeam
- ITCoord
- ITdeptTechForum
- ITDRP
- ITGT
- ITSDC
- LAr
- LCG
- LCGAAWorkbook
- Leade
- LHCAccess
- LHCAtHome
- LHCb
- LHCgas
- LHCONE
- LHCOPN
- LinuxSupport
- Main
- Medipix
- Messaging
- MPGD
- NA49
- NA61
- NA62
- NTOF
- Openlab
- PDBService
- Persistency
- PESgroup
- Plugins
- PSAccess
- PSBUpgrade
- R2Eproject
- RCTF
- RD42
- RFCond12
- RFLowLevel
- ROXIE
- Sandbox
- SocialActivities
- SPI
- SRMDev
- SSM
- Student
- SuperComputing
- Support
- SwfCatalogue
- TMVA
- TOTEM
- TWiki
- UNOSAT
- Virtualization
- VOBox
- WITCH
- XTCA

Welcome Guest

Copyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.

Ideas, requests, problems regarding TWiki? Send feedback

Ideas, requests, problems regarding TWiki? Send feedback