Confidence levels for ratios of measurements

It often happens that we have two measurements with a given uncertainty (x±Δx and y±Δy) and are interested in their ratio r=x/y and its uncertainty. This is a particular case of the problem of estimating the confidence interval of a derived quantity, given the pdf of the original measurements. A common practice is to derive the uncertainty in the ratio following error propagation as (for independent x and y)

It is important to note that this expression is derived assuming that r is approximately linear. If additionally we what to interpret these “Δ" as (gaussian-like) standard deviations providing a 68.3% coverage, we need to impose the requirement that x and y follow a Normal law. There are many common cases when some of these approximations are not justified and this simple expression provides confidence intervals with a significant undercoverage. Even for normally distributed x and y, the ratio x/y is ill-behaved, since it is highly nonlinear when y gets close to zero, which to some extent always happens since the pdf extends from -∞ to +∞. It is usually said this approximation is reasonable provided the errors are small compared with the central value, but it also depends how far from the central value you are interested in (i.e. if you just stay in ±σ or rather are working in ±5σ)

For example, if we follow this approach for the ratio of two gaussian distributed variables with a standard deviation that is 50% of the mean, we obtain a reasonable coverage for the ±1σ interval, 67.6% instead of 68.3%, but totally wrong for the upper tail above 3σ, 5% instead of 0.1% . Similarly for the ratio of two Poisson distributions with mean 5, the ±1σ coverage is 68.1% and that for the upper tail above 3σ is 4% (more examples can be found here)

If these approximations are not enough for your particular application one of the approaches described below will provide you with better coverage.

Ratio of Gaussians

A precise treatment, without approximations, of the case when both the numerator and denominator follow a normal law is decribed in RatioOfGaussians

Ratio of Poisson

Another common situation appears when our magnitude of interest is derived from the ratio of two integer counts, following two independent Poisson laws. This is the case when we compare data to a MC-based prediction (only one MC sample) or two data counting experiments (ie the relative rate of two different decays). A detailed description of this problem can be found in this talk. Error propagation is, in general, not a good approximation in this case, even for relatively large number of counts ( as shown here), but alternative analytical solutions can be used.

The CI for the ratio of two independent Poissonian observations n and m can be demonstrated to be equivalent to that of a binomial with n successes and (n+m) trials (observed by J. Przyborowski and H. Wilenski, [1], Introduced in HEP by F. James and M. Roos, [2]).

Confidence intervals can be set for the binomial case according to the so-called “exact CI” or Clopper-Pearson (C-P) Confidence Interval [4]. For a given confidence level, an asymmetric interval around the mean is set that guarantees a coverage larger or equal than the defined CL. One can, for example, derive the “asymmetric error” from the interval corresponding to 68.3% CL. It is important to remark this interval is “exact” in the sense that guarantees a minimum coverage, but in most cases overcovers due to the discrete nature of the binomial distribution, as shown in the following figure (blue curve). The saw-tooth shape is also a consequence of the discreteness of binomial distribution.

Using the equivalence described above between the Poisson and Binomial counts, we can derive the CI for the ratio of Poisson intervals with:

lower_poisson = lowerbinom/(1 - lowerbinom)

upper_poisson = upperbinom/(1 - upperbinom)

While overcovering might be a reasonable practice when setting a limit, being on the conservative side, it can have the opposite effect when using a confidence interval to set error. Overcoverage will mean larger “error bars” and hence we could claim for compatibility of two measurements more often than we should.

To avoid this problem, an extensive review of alternative methods was done by Cousins, Hymes, and Tucker [3]. We recommend the usage of the so-called Lancaster mid-P [5] instead of exact Clopper-Pearson, using “an intermediate value of the tail probability” to overcome discreteness. This method provides good coverage properties, much closer to 68.3% for all cases, although it can show some slight undercoverage.

The following graphs compare the performance of these two methods, together with the naive error propagation with gaussian approximation. Left figure shows the intervals as a function of the observed counts in the denominator for a fixed denominator of 5 counts, the error bars you would get for each of the methods. Center and right figure show the coverage provided as a function of the denominator counts for two values of the ratio (0.1 and 1). More examples can be found in [3]. It can be seen that midP provides a coverage very close to the required 68.3%. CP gives wider CI, larger error bars, with overcoverage. The simple standard deviation approach is often not so bad approximation but can yield to dramatic undercoverage and unphysical bounds.

Confidence intervals and their coverage for different examples of ratio of Poisson distribution. CI calculated with different methods for a fixed mean of 5 in the denominator as a function of the numerator (left). Coverage for each of the CI as a function of the denominator mean for a fixed ratio of 1 (center) or 0.1 (right).

Root and R tools.

Upper and lower bounds of C-P intervals at a given CL confidence level for s successes out of t trials following a binomial distribution, can be set in Root as TEfficiency::ClopperPearson(s,t,CL, Upper) Upper=true/false for upper/lower bound. Similarly midP bounds are obtained with TEfficiency:: MidPInterval (s,t,CL, Upper)

In R, they can be obtained after loading PropCIs package with exactci(s,t,CL) and midPci(s,t,CL)

Hence to obtain the confidence interval for a ratio of Poisson with n and m observations, one has to set s=n, t=n+m call the functions above and transform the binomial limits obtained to the Poisson equivalent with the equation limit/(1-limit)

Ratio of a Poisson and a Gaussian or of a Poisson and a linear combination of Poisson

This method can be extended with good approximation to more general cases, like the ratio of Poisson and Gaussian or Poisson and linear combination of Poisson. Being approximate, it is recommended to use it only for presentational porposes. This method is described in section FMDataMC

References

[1] . Przyborowski and H. Wilenski, “Homogeneity of Results in Testing Samples from Poisson Series," Biometrika 31 (1940)

[2] F. James and M. Roos, “Errors on Ratios of Small Numbers of Events,” Nuclear Physics B172 (1980)

[3] Robert D. Cousins, Kathryn E. Hymes, Jordan Tucker, Frequentist Evaluation of Intervals Estimated for a Binomial Parameter and for the Ratio of Poisson Means, Nucl. Instr A. 612-2 (2010)

[4] Clopper and E.S. Pearson, “The Use of Confidence or Fiducial Limits illustrated in the Case of the Binomial," Biometrika 26 (1934) 404

[5] H.O. Lancaster, Significance Tests in Discrete Distributions," J. Amer. Stat. Assoc. 56 (1961) 223.

-- FranciscoMatorras - 2017-06-09