A/B Test Sample Size Calculator

Calculate the sample sizes for your experiments

Setting up an AB Test: What sample size do I need?

Setting up a proper A/B experiment involves selecting the appropriate sample size. This ensures the test is adequately powered to detect the change while minimizing statistical noise. In statistical terms, we want to detect the minimum detectable effect (MDE) with statistical confidence. Having too few samples produces statistically ambiguous results, and having too many samples can be overkill and a waste of time and resources.

Baseline Conversion Rate

The preexisting or expected conversion rate of the control group. This could be a click-through rate, retention rate, or positive rating rate.

1 99

Minimum Detectable Effect

The relative minimum difference in conversion rate between the test and control group we want to observe. This can be the expected effect, or some minimum threshold where the experiment is not worth running (ie. impact is too small to care about).

1 50

We want to detect a new conversion rate < 7.0% or > 13.0%

Advanced Settings

The following values are set to their optimal defaults, but feel free to play with them to see how they affect the exposures.

One-sided test
Used to determine if the test variation is better than the control (Recommended)
Two-sided test
Used to determine if the test variation is different than the control

A/B Split Ratio (Test vs. Control)
0.5 is an evenly split 50/50 test

Significance Level (α)
The probability that a statistically significant difference is detected even though no actual difference exists

Power (1-ß)
The probability that the minimum detectable effect will be detected (assuming it exists)

Test Size

1,289

Control Size