Frequently Asked Questions

When conducting multiple experiments, the decision to run them in the same layer versus different layers has significant implications.

Placing experiments in the same layer ensures that there is no overlap between participants in different experiments. This is beneficial for eliminating interaction effects between experiments, as no user will be part of more than one experiment at a time. However, a critical consideration is that using layers divides the user base, which can substantially reduce the experimental power and sample size.

This division of the user base means that, at a minimum, the number of participants in each experiment is halved. Consequently, this reduction can limit the number of experiments that can be conducted simultaneously and may prolong the duration required to achieve statistically significant results.

When experiments are run in a layer and thus have a smaller sample size, any effects observed while the experiment is running will also be smaller.

For a more in-depth discussion on the topic, including the trade-offs between isolating experiments and embracing overlapping A/B tests, refer to the article Embracing Overlapping A/B Tests and the Danger of Isolating Experiments.