Confidence levels for ratios of measurements

Often we have two measurements with a given uncertainty (x±Dx and y±Dy) and are interested in the ratio r=x/y and its uncertainty. This is a particular case of the problem of estimating the confidence interval of a derived quantity, given the pdf of the original measurements. A common practice is to derive the uncertainty in the ratio following error propagation as (for independent x and y)

It is important to note that this expression is derived assuming that r is approximately linear. If additionally we what to interpret these “D" as (gaussian-like) standard deviations providing a 68.3% coverage, we need to impose the requirement that x and y follow a Normal law. There are many common cases when some of these approximations are not justified and this simple expression provides confidence intervals with a significant undercoverage. Even for normally distributed x and y, the ratio x/y is ill-behaved, since it is highly nonlinear when y gets close to zero, which to some extent always happens since the pdf extends from -¥ to +¥. It is usually said this approximation is reasonable provided the errors are small compared with the central value, but it also depends how far from the central value you are interested in (i.e. if you just stay in ±s or rather are working in ±5s)

For example, if we follow this approach for the ratio of two gaussian distributed variables with a standard deviation that is 50% of the mean, we obtain a reasonable coverage for the ±1s interval, 67.6% instead of 68.3%, but totally wrong for the upper tail above 3s, 5% instead of 0.1% . Similarly for the ratio of two Poisson distributions with mean 5, the ±1s coverage is 68.1% and that for the upper tail above 3s is 4% (more examples can be found here)

If these approximations are not enough for your particular application one of the approaches described below might give you better coverage.

Ratio of Gaussians

[copy Jochen’s twiki]

Ratio of Poisson

It is rather common to face the situation where our magnitude of interest is derived from the ratio of two integer counts, following two independent Poisson laws. This is the case when we compare data to a MC-based prediction (only one MC sample) or two data counting experiments (ie the relative rate of two different decays). A detailed description of this problem can be found in this talk. Error propagation is, in general, not a good approximation in this case, even for relatively large number of counts ( as shown here), but alternative analytical solutions can be used.

The CI for the ratio of two independent Poissonian observations n and m can be demonstrated to be equivalent to that of a binomial with n successes and (n+m) trials (observed by J. Przyborowski and H. Wilenski, [1], Introduced in HEP by F. James and M. Roos, [2]).

Confidence intervals can be set for the binomial case according to the so-called “exact CI” or Clopper-Pearson (C-P) Confidence Interval [4]. For a given confidence level, an asymmetric interval around the mean is set that guarantees a coverage larger or equal than the defined CL. One can, for example, derive the “asymmetric error” from the interval corresponding to 68.3% CL. It is important to remark this interval is “exact” in the sense that guarantees a minimum coverage, but in most cases overcovers due to the discrete nature of the binomial distribution, as shown in the following figure (blue curve). The saw-tooth shape is also a consequence of the discreteness of binomial distribution.

Using the equivalence described above between the Poisson and Binomial counts, we can derive the CI for the ratio of Poisson intervals with:

lower_poisson = lowerbinom/(1 - lowerbinom)

upper_poisson = upperbinom/(1 - upperbinom)

While overcovering might be a reasonable practice when setting a limit, being on the conservative side, it can have the opposite effect when using a confidence interval to set error. Overcoverage will mean larger “error bars” and hence we could claim for compatibility of two measurements more often than we should.

To avoid this problem, an extensive review of alternative methods was done by Cousins, Hymes, and Tucker [3]. We recommend the usage of the so-called Lancaster mid-P [5] instead of exact Clopper-Pearson, using “an intermediate value of the tail probability” to overcome discreteness. This method provides good coverage properties, much closer to 68.3% for all cases, although it can show some slight undercoverage.

The following graphs compare the performance of these two methods, together with the naive error propagation with gaussian approximation. Left figure shows the intervals as a function of the observed counts in the denominator for a fixed denominator of 5 counts, the error bars you would get for each of the methods. Center and right figure show the coverage provided as a function of the denominator counts for two values of the ratio (0.1 and 1). More examples can be found in [3]. It can be seen that midP provides a coverage very close to the required 68.3%. CP gives wider CI, larger error bars, with overcoverage. The simple standard deviation approach is often not so bad approximation but can yield to dramatic undercoverage and unphysical bounds.

Root and R tools.

Upper and lower bounds of C-P intervals at a given CL confidence level for s successes out of t trials following a binomial distribution, can be set in Root as TEfficiency::ClopperPearson(s,t,CL, Upper) Upper=true/false for upper/lower bound. Similarly midP bounds are obtained with TEfficiency:: MidPInterval (s,t,CL, Upper)

In R, they can be obtained after loading PropCIs package with exactci(s,t,CL) and midPci(s,t,CL)

Hence to obtain the confidence interval for a ratio of Poisson with n and m observations, one has to set s=n, t=n+m call the functions above and transform the binomial limits obtained to the Poisson equivalent with the equation limit/(1-limit)

Extension

This method can be extended with good approximation to more general cases, like the ratio of Poisson and Gaussian or Poisson and linear combination of Poisson ( discussed here)

-- FranciscoMatorras - 2017-06-09

Edit | Attach | Watch | Print version | History: r6 | r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r2 - 2017-06-09 - FranciscoMatorras
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback