11
NHST is greatly complicated when there is more than one result (in this case, 400,000 results) to test. If the probability of being incorrect about any single hypothesis test is equal to Îą, then the probability of being incorrect on at least one of k hypothesis tests equals 1-(1-Îąk), which approaches 1.0 very quickly. Social science has developed modest technologies for dealing with the problem, like the familiar Bonferroni correction1, but such methods do not begin to apply to the enormous number of tests conducted in GWAS, for which somewhat more sophisticated methods have been developed. Significance testing in GWAS incorporates several steps. First, the full distribution of test probabilities is plotted against the expected distribution under the null hypothesis, to establish that something is disturbing the null distribution. In Weedon et al. (2008; see their Figure 1) there was an unmistakable overrepresentation of low probabilities. In the largest sample (combined meta-analytically across several studies), for example, there were 27 tests with significance levels less than 10-5, compared to the four that would be expected on the basis of sampling error under the null hypothesis of no association. Weedon et al. conclude, "Approximately 23 of these loci are therefore likely to represent true positives." (p. 576) The associations are then subjected to an even more stringent test. Thirty-nine of the original 400,000 SNPs (the 27 that exceeded the 10-5 criterion plus 11 that exceeded a 10-4 criterion, plus one more identified as a candidate in another study) were retested in 1 In which the required significance level, usually p<.05, is divided by the total number of tests to be conducted in the experiment.