-- AmnonHarel - 17-Mar-2010

New prior shapes for nuisance parameters

The statistics board asked for gamma and lognormal priors. These distributions have 3 degrees of freedom, two of which are fixed by the requirements of given mean and RMS. From a literature scan I found that the common practice is to assume mu=0, where I use the variable names used in ROOT's TMath.

To see what these look like I used the attached priors.c (may require ROOT5.26 to run). Some output examples are:

mean: 1 1 10
width: 0.01 0.1 1
plot: priors_1_001.png priors_1_01.png priors_10_1.png

The numbers in parenthesis are the errors on the last digits. The first number is the mean of the histogram, and the 2nd is the RMS. The Gauss and LogNormal histogram is filled by transforming a normal variable, rather than from the plotted function.

19-Apr-2010 Information

Introduction and references

Some material in preparation of the April 20th meeting.

This supplements Robert's email and uses the references he provided:
[1] my talk Thursday - http://indico.cern.ch/conferenceDisplay.py?confId=91456
[2] the D0 paper - http://arxiv.org/abs/hep-ex/9807014
[3] the 2006-2007 CMS studies - http://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2007_039_v4.pdf
Another reference I will use is
[4] the 2009 CMS studies - http://cms.cern.ch/iCMS/jsp/openfile.jsp?tp=draft&files=AN2009_161_v1.pdf

Our preferred limit setting technique for ICHEP, and the publication plan that motivates it, are described in [1]. This plan hinges on our expectations that for ICHEP the transition between the statistics-dominated and statistical-dominated regions and the transition between the regions explored by the Tevatron and those we will explore for the first time fall very close to each other, around 800GeV. It is clear that further development will be needed for future publication.

In particular, the systematic effects have large bin-to-bin correlations, which should dominate any quantitative treatment of the systematics (see [2], Fig. 2, off-diagonal elements). In particular, the relative JES effects at a particular jet pT should be propagated to a range of Mjj bins.

Additional presentations

The preferred technique

(with a few questions to the SB interleaved)

  1. The test statistics are log likelihood ratios that do not take systematic effects into account.
    • the likelihood used only bins above the Tevatron limits, e.g., above Mjj=800GeV, and in the case of contact interactions only bins below the contact interaction scale.
    • as per the standard technique for a Poisson ratio, the distributions are conditioned on the observed total counts.
  2. Use ensemble tests to account for the systematic uncertainties and derive "hybrid" 95%CL limits:
    • leading systematics are the relative and absolute JES. The corresponding nuisance parameters are drawn from Gaussian distributions.
    • we set limits as in Section 3.4 of [4].
      • the delay was to verify that this procedure holds when including the leading systematics correctly
    • in [4] we used the limit corresponding to the median of the SM distributions as the "predicted limit". It is not an expectation value. We're toying with adding +/- 1 sigma and 2 sigma bands.
      • Any objections to basing them all on the corresponding SM quantiles?
      • this assumes negligible uncertainties on the curve used to translate the observable into a limit. Only statistical uncertainties are relevant, and they can be reduced arbitrarily by increasing the size of the NP ensembles. We haven't yet chosen how to estimate the uncertainty on the 95th quantile, and welcome any SB input.
  3. For each m_q* we can plot (x-axis is m_q*, y-axis is x-section):
    • the SM predicted limit,
      • preferably with +/- 1 and 2 sigma bands
    • the predicted NP x-section,
      • can add the predicted NP limit, if worth the visual clutter
    • the observed limit Since the q* peaks are local, the q*/SM log likelihood ratios have similar localities, and such a plot provide a possible answer to the question: is the data consistent with the SM?
  4. For each lambda we can plot (x-axis is m_q*, y-axis is LLR):
    • the SM predicted LLR (dashed line below),
      • preferably with +/- 1 and 2 sigma bands
    • the 95% quantiles for the corresponding contact interaction scenario (dark red line and points),
    • the observed LLR (shown for a "golden dataset" below by the solid black line)
    • Examples of visual summaries of the statistical analysis of contact interaction:
Without systematics With systematics
climLLR2_4K.png climLLR1_1K.png

Additional statistical analysis of high mass region?

We would appreciate your input on whether any additional test statistics that answer the question: "Is the data above the Tevatron limits consistent with our SM predictions?"
  1. must be published (my views against this are in [1], slide 9)
  2. is interesting enough to include in a paper
    • this probably depends on how difficult it will be to present such a test statistic. In [4] we developed a possible generic test statistic to answer that question ("G").
  • this region will be statistics dominated in ICHEP

Additional statistical analysis of low mass region?

There are two more, closely-related questions that can be asked, and we would appreciate your input on them.
  1. "Is the data below the Tevatron limits consistent with our SM predictions?" - generic null hypothesis testing
    • This region is dominated by systematics, so any method that ignores systematics will likely yield "no" here.

  1. "Can the data below the Tevatron limits be fit by the expected systematic variations of our SM predictions?" - goodness of fit statistic
    • This is one method to answer the question above. If the fit is limited to the high statistics regime, even a "standard" chi^2 test may do well enough, as the binomial distributions for such large numbers can easily be approximated to Gaussians. Another variation is to perform the test on the inner and outer spectra, taking into account their correlation. Perhaps this is what was done in [2] (note the diagonal correlations in their Fig. 2).

To a large extend, this question will be answered in the "R" plot, which summarizes our data and predictions. We will draw Clopper-Pearson intervals on the data points, and the systematics on the predictions, which allow the reader to see any significant deviation.

It would be nice to have a statistical statement that covers both the high-statistics, Tevatron-excluded-NP region and the low-statistics, we're-exploring-new-energies region. Especially as the fact that the cross over between the two regimes is basically at the same point is a temporary coincidence expected for an ICHEP result.

I plan to explore the 2nd approach, using a profile likelihood fit with the "usual" binomial statistics. But we do not yet have a consensus on whether this is necessary for an ICHEP publication.

  • Currently, we do not fit the (nuisance parameters of the) prediction to the data, and we do not have a detailed model of the bin-to-bin correlations. We have not yet discussed internally whether such a detailed description of the systematics is realistic for the ICHEP timescale.

Additional information

Input models

  • Contact interaction models:
  • q* models (rough version from 2010-Apr-18):

17-Mar-2010 Information

Following the discussion just before midnight in the 16-Mar statistics board meeting, here is a status report for the statistics in the di-jet ratio, with context and links to notes and talks.

There are several reasons to adopt for an ICHEP analysis different tools, and a different approach, than was taken so far in the dijet ratio. This demotes some of the documentation to a documentation of historical attempts - if you wish to skip that, don't read sections 3.2 to 3.4 in the analysis note, and just look for "LLR" plots in the presentations, ignoring other test statistics.

Given an observable sensitive to a wide range of models, I started by looking for a test statistic that maintains that generality. This is documented in the AN. The presentation shows extensions, and also that systematic uncertainties have a huge effect (*). This shifts the focus to how to handle the systematics, and indicates we should switch to using the more familiar log-likelihood ratios (LLRs) so we can focus on the systematics.

(*) This is a very preliminary and unexpected result which I haven't fully debugged yet.

Given Jim's 7TeV numbers, and assuming ~3pb-1 for ICHEP, we are looking at systematic uncertainties of roughly 0.03 (absolute on R) and statistical uncertainties will dominate (i.e. >0.06) from around 800GeV. The Tevatron exclusions are Lambda<2.7TeV and Mq*<0.87TeV. So we can analyze our data starting from 870GeV, where systematics should not dominate, and ruling out a q* of 1TeV is in play (see Fig. 11 in AN, though the wrong Ecm is used), and so is a 3TeV contact interaction (see Jim's presentation).

To put it differently - for ICHEP we have the option of doing the analysis so that statistics dominates IN EVERY SINGLE BIN OF INTEREST.

  • covering only this case will leave us badly placed if some systematic effect turns out bigger than expected and increases the systematics by more than a factor of 2.

Selected documentation:

Now what?

I see two main options:
  1. use the current tools, with local limit setting for q* and an LLR, and ignoring all bins below ~800GeV.
    • Should work.
      • Latest test in a different "edge of statistical strength" scenario (in presentation) unexpectedly failed - some more tests and debugging is in order.
    • I will test with contact interactions early next week, and with Jim's help I'll also prepare the q* results later next week.
    • if the current tools are insufficient, I expect that incorporating the basic uncertainty in the test statistic, e.g. by doing a fit for the relative JES within each likelihood ("log profile-likelihood ratio"?) will handle the leading, and all other systematics.
  2. start over in RooStats, using only likelihoods.
    • re-examining RooFit and RooStat's latest versions with excellent help from Genadi Kukarzev, it seems that it should be possible to do our analysis in RooStats. Some of the things we need seem to be a bit unusual / supported in a cumbersome way, which I find a bit worrisome. But hopefully all works as it should.
    • the features we need are:
      1. ensemble testing by histograms (without individual events) - supported in latest version smile
      2. variable binning - supported in a round-about way (seems we need to name each bin)
      3. using a different statistics model and ensemble-generation models
      4. defining the test statistic as a function of functions of the data, rather than the data itself
      • the last two (c & d) look supported and that they should work cleanly, but they also look like unusual and complicated use cases.
    • thus migrating the code to RooStats is possible but not be trivial, and is detrimental to having an ICHEP result.
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng climLLR0_4K.png r1 manage 12.1 K 2010-04-19 - 04:07 AmnonHarel Limits with no systematics
PNGpng climLLR0_s2_1K.png r1 manage 12.2 K 2010-04-19 - 04:21 AmnonHarel Without systematics, 1K PDSs per ensemble
PNGpng climLLR1_1K.png r2 r1 manage 14.4 K 2010-04-20 - 20:54 AmnonHarel Limits with systematics
PNGpng climLLR2_4K.png r2 r1 manage 15.9 K 2010-04-20 - 20:52 AmnonHarel Limits without systematics
PNGpng plot_ci_model.png r3 r2 r1 manage 19.3 K 2010-04-20 - 19:32 AmnonHarel Contact interaction models
PNGpng plot_qs_model.png r2 r1 manage 18.2 K 2010-04-20 - 18:34 AmnonHarel q* models (rough version from 2010-Apr-18)
C source code filec priors.c r2 r1 manage 3.7 K 2010-04-28 - 19:34 AmnonHarel Visualization of nuisance parameter priors
PNGpng priors_10_1.png r2 r1 manage 18.4 K 2010-04-28 - 19:34 AmnonHarel m=10,w=1
PNGpng priors_1_001.png r2 r1 manage 20.7 K 2010-04-28 - 19:35 AmnonHarel m=1,w=0.01
PNGpng priors_1_01.png r2 r1 manage 21.5 K 2010-04-28 - 19:35 AmnonHarel m=1,w=0.1
PDFpdf ratio_sys_var_note.pdf r1 manage 74.6 K 2010-03-17 - 19:22 AmnonHarel Note on systematic variations
Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r10 - 2010-04-28 - AmnonHarel
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback