What is Multiple Hypothesis Testing and Why It is a Problem?

What is Multiple Hypothesis Testing?

Multiple hypothesis testing problem refers to an increase in type I error when you perform multiple statistical tests simultaneously.

The type I error (also known as false positive) occurs when the null hypothesis (H0) is actually true but is rejected.

A multiple hypothesis testing problem occurs when we have to conduct many hypothesis tests at the same time. In genomics experiments, we often perform thousands of hypothesis tests simultaneously to study differences in gene expression between samples.

Why Multiple Hypothesis Testing is a problem?

When we perform a multiple hypothesis test, what is the probability (p) of at least 1 is false positive

The significance level or α is also known as a type I error (false positive).

Traditionally, we try to set the significance level as 0.05. It means that you are willing to accept a 5% chance of making a type I error, that is, if the p value is < 0.05, you reject the null hypothesis in favor of an alternative hypothesis.

For example, when you perform one hypothesis test, what is the probability of making at least 1 error by chance?

For one hypothesis test, the probability of making at least 1 error is the same as type I error (0.05). This is the same as the traditional significance level.

But, when you perform 50 hypothesis tests at the same time, what is the probability of making at least 1 error by chance?

For 50 simultaneous hypothesis tests, the probability of making at least 1 error is 0.92 (92%).

This indicates that when you perform 50 hypothesis tests at the same time, the probability of obtaining at least one error due to random chance is 92%. This probability further increases with the increase in number of hypothesis tests.

This can lead to an inflated false discovery rate and potentially incorrect conclusions. Hence, it is very important to control the false discovery rate when you are performing many hypothesis tests at the same time.

You can use methods such as Bonferroni Correction, Holm’s Method, Benjamini-Hochberg Procedure, Benjamini-Yekutieli Procedure, and Storey’s q-value to mitigate the risk of making a type I error (false positives) when conducting multiple hypothesis testing at the same time.