Frequently Asked Questions

In A/B testing, the inclusion of non-interacting users in the sample size does not inherently affect the p-values, provided that the metric being evaluated is relevant to the feature being tested. When non-interacting users are included in the sample, it can dilute the experiment results by adding a number of zeros to both the control and test groups.

This dilution can decrease the experimental power, meaning that a larger sample size or a longer experiment duration may be required to reach statistical significance, given the underlying effect size.

However, the directionality of the results should remain correct, and the zeros should be evenly distributed between the groups, maintaining the integrity of the experiment results.

It is recommended to expose users to the experiment as close as possible to the point where their experiences diverge, to avoid overexposure and to increase the efficiency of the experiment.

f the overexposed users do not contribute to the metric, they do not introduce any noise and therefore do not affect the experimental power.

It is important to note that if the treatment affects the rate at which people reach the relevant step in the process, manual filtering to users who reached that step could introduce bias and should be approached with caution.