Not keeping the FCR means ${\displaystyle {\text{FCR}}>q}$  when ${\displaystyle q={\frac {V}{R}}={\frac {\alpha m_{0}}{R}}}$ , where ${\displaystyle m_{0}}$  is the number of true null hypotheses, ${\displaystyle R}$  is the number of rejected hypothesis, ${\displaystyle V}$  is the number of false positives, and ${\displaystyle \alpha }$  is the significance level. Intervals with simultaneous coverage probability ${\displaystyle 1-q}$  can control the FCR to be bounded by ${\displaystyle q}$ .

Classification of multiple hypothesis tests edit

The following table defines the possible outcomes when testing multiple null hypotheses. Suppose we have a number m of null hypotheses, denoted by: H₁, H₂, ..., H_m. Using a statistical test, we reject the null hypothesis if the test is declared significant. We do not reject the null hypothesis if the test is non-significant. Summing each type of outcome over all H_i  yields the following random variables:

• m is the total number hypotheses tested
• ${\displaystyle m_{0}}$  is the number of true null hypotheses, an unknown parameter
• ${\displaystyle m-m_{0}}$  is the number of true alternative hypotheses
• V is the number of false positives (Type I error) (also called "false discoveries")
• S is the number of true positives (also called "true discoveries")
• T is the number of false negatives (Type II error)
• U is the number of true negatives
• ${\displaystyle R=V+S}$  is the number of rejected null hypotheses (also called "discoveries", either true or false)

In m hypothesis tests of which ${\displaystyle m_{0}}$  are true null hypotheses, R is an observable random variable, and S, T, U, and V are unobservable random variables.

Selection edit

Selection causes reduced average coverage. Selection can be presented as conditioning on an event defined by the data and may affect the coverage probability of a CI for a single parameter. Equivalently, the problem of selection changes the basic sense of P-values. FCR procedures consider that the goal of conditional coverage following any selection rule for any set of (unknown) values for the parameters is impossible to achieve. A weaker property when it comes to selective CIs is possible and will avoid false coverage statements. FCR is a measure of interval coverage following selection. Therefore, even though a 1 − α CI does not offer selective (conditional) coverage, the probability of constructing a no covering CI is at most α, where

${\displaystyle \Pr[\theta \not \in \mathrm {CI} ,\ {\text{CI constructed}}]\leq \Pr[\theta \not \in \mathrm {CI} ]\leq \alpha }$ 

Selection and multiplicity edit

When facing both multiplicity (inference about multiple parameters) and selection, not only is the expected proportion of coverage over selected parameters at 1−α not equivalent to the expected proportion of no coverage at α, but also the latter can no longer be ensured by constructing marginal CIs for each selected parameter. FCR procedures solve this by taking the expected proportion of parameters not covered by their CIs among the selected parameters, where the proportion is 0 if no parameter is selected. This false coverage-statement rate (FCR) is a property of any procedure that is defined by the way in which parameters are selected and the way in which the multiple intervals are constructed.

Bonferroni procedure (Bonferroni-selected–Bonferroni-adjusted) for simultaneous CI edit

Simultaneous CIs with Bonferroni procedure when we have m parameters, each marginal CI constructed at the 1 − α/m level. Without selection, these CIs offer simultaneous coverage, in the sense that the probability that all CIs cover their respective parameters is at least 1 − α. unfortunately, even such a strong property does not ensure the conditional confidence property following selection.

FCR for Bonferroni-selected–Bonferroni-adjusted simultaneous CI edit

The Bonferroni–Bonferroni procedure cannot offer conditional coverage, however it does control the FCR at <α In fact it does so too well, in the sense that the FCR is much too close to 0 for large values of θ. Intervals selection is based on Bonferroni testing, and Bonferroni CIs are then constructed. The FCR is estimated as, the proportion of intervals failing to cover their respective parameters among the constructed CIs is calculated (setting the proportion to 0 when none are selected). Where selection is based on unadjusted individual testing and unadjusted CIs are constructed.

FCR-adjusted BH-selected CIs edit

In BH procedure for FDR after sorting the p values P(1) ≤ • • • ≤ P(m) and calculating R = max{ j : P( j) ≤ j • q/m}, the R null hypotheses for which P(i) ≤ R • q/m are rejected. If testing is done using the Bonferroni procedure, then the lower bound of the FCR may drop well below the desired level q, implying that the intervals are too long. In contrast, applying the following procedure, which combines the general procedure with the FDR controlling testing in the BH procedure, also yields a lower bound for the FCR, q/2 ≤ FCR. This procedure is sharp in the sense that for some configurations, the FCR approaches q.

1. Sort the p values used for testing the m hypotheses regarding the parameters, P(1) ≤ • • • ≤P(m).

2. Calculate R = max{i : P(i) ≤ i • q/m}.

3. Select the R parameters for which P(i) ≤ R • q/m, corresponding to the rejected hypotheses.

4. Construct a 1 − R • q/m CI for each parameter selected.

• False positive rate
• Post-hoc analysis

Footnotes

Other Sources

• Zhao, Zhigen; Hwang, J. T. Gene (2012). "Empirical Bayes false coverage rate controlling confidence intervals" (pdf). Journal of the Royal Statistical Society, Series B. doi:10.1111/j.1467-9868.2012.01033.x.^{[permanent dead link]}