A Type 1 error, also known as a "false positive," is a kind of statistical error that occurs when a hypothesis test incorrectly rejects a true null hypothesis. In other words, it's the error of accepting an alternative hypothesis (the real hypothesis of interest) when the results can be attributed to chance.

Plainly speaking, a Type 1 error is detecting an effect that isn't present.

Example

Let's say you're running an A/B test on your website to determine if a new feature increases user engagement. Your null hypothesis (H0) is that the new feature has no effect on user engagement, and your alternative hypothesis (H1) is that the new feature does affect user engagement.

• If you conclude that the new feature increases user engagement when it actually does not, you've made a Type 1 error. You've incorrectly rejected the null hypothesis.

Context in Multi-arm Experiments

In the context of multi-arm experiments, Type 1 error rates are most suitable when:

1. You have prior knowledge or data that the control group is suboptimal.

2. The real objective of the experiment is to determine the best test group.

3. Your team/company is already committed to making a change.

However, you don't want to be at the mercy of statistical noise, which can thrash your user experience, trigger unknown secondary effects, and/or create extra product work. As a rule of thumb, if you use ⍺=0.05 (a common threshold for Type 1 error), you should feel comfortable running up to 4 variations. This slightly biases you towards making a change, but keeps the overall Type 1 error rate below 0.05.