How to plan test duration when using CUPED

Test planning is crucial in controlled experiments to ensure accurate and reliable results. Proper planning of sample size helps avoid underpowered tests, which can fail to detect real effects. Proper planning also reduces the risk of inflated false positive rates when not using sequential testing methods, because when you do not determine the experiment sample size you tend to peek at the results more than once. Here's the formula for calculating sample size for a t-test for two means:

$$n={\left(\frac{\sigma (Z_{\alpha /2}+Z_{\beta })}{\mathrm{\Delta }}\right)}^2$$

n: Sample size for each group, assuming equal allocation
$Z_{\alpha /2}$: Z-value for the desired significance level, (assuming two-tailed hypothesis)
$Z_{\beta }$: Z-value for the desired power
$\mathrm{\Delta }$: Minimum Detectable Effect (MDE) in absolute terms, which is the difference between two means
$\sigma$: Standard deviation

Notice that the variance, ${\sigma }^2$, is directly proportional to sample size. Therefore, reducing variance by 40% means you need 40% less sample size. For instance, if you have Original Sample Size = n and Reduced Variance = 0.6${\sigma }^2$, then your new required sample size will be: 0.6n, then your new required sample size will be: 0.6n.

CUPED (Controlled Utilization of Pre-Existing Data) is a technique designed to leverage pre-existing data to reduce variance in experiment KPIs, thereby enhancing test sensitivity. By adjusting for variability unrelated to the experiment, CUPED reduces noise in the data, making it easier to detect the true effect of the experiment.

To illustrate, let's consider a business metric $\bar{Y}$, such as revenue. In a traditional t-test, we would compare the average revenue of the control group with that of the treatment group.

With CUPED, we introduce a new variable, ${\bar{Y}}_{\text{CUPED}}$, by adjusting the original metric $\bar{Y}$ using pre-existing data $\bar{X}$. This data $\bar{X}$ is scaled by a constant $\theta$, which must be determined. Instead of comparing the average revenues in a t-test, we compare the ${\bar{Y}}_{\text{CUPED}}$ values using a modified variance.

The intuition behind CUPED is that the total variance of the business metric $\bar{Y}$ can be decomposed into two components: the portion attributed to the variance of the control variate $\bar{X}$ and the portion explained by other unknown factors.

Here's the CUPED formula:

$${\bar{Y}}_{\text{CUPED}}=\bar{Y}-\theta \times \bar{X}$$

Where:
$\bar{Y}$: The business metric average during the experiment
$\bar{X}$: Pre-existing data (e.g., business metric average from a pre-experiment period)
$\theta$: The constant. Variance reduction optimality is achieved when $\theta =\frac{\text{Cov}(X,Y)}{\text{Var}(X)}$

The variance of ${\bar{Y}}_{\text{CUPED}}$ is reduced by the square of the Pearson correlation between $\bar{X}$ and $\bar{Y}$:

$$\text{Var}\left({\bar{Y}}_{\text{CUPED}}\right)=\text{Var}(\bar{Y})(1-{\rho }^2)$$

Where $\rho$ is the Pearson correlation coefficient. For instance, if $\rho =0.9$, variance is reduced by ${0.9}^2=0.81$. Reducing the sample size by 80% is a huge achievement, and it indeed happens in practice. The higher the correlation, the bigger the variance reduction. This technique is powerful, especially when testing on existing users with rich historical data.

Example:

In reality, we don't know the true values of the variables, so we must estimate them.

Suppose you have KPI data for a website's page views. In the pre-experiment period, the average page views per user $\bar{X}$ is 1000. During the experiment period, the page views per user $\bar{Y}$ is 1100. The sample covariance between $\bar{X}$ and $\bar{Y}$ is 200, and the sample variance of $\bar{X}$ is 250.

Estimate $\theta$:

$$\hat{\theta }=\frac{200}{250}=0.8$$

Using CUPED, the adjusted KPI for the experiment period would be:

$${\bar{Y}}_{\text{CUPED}}=1100-0.8\times 1000=300$$

CUPED is particularly useful in situations where there is a lot of variability in the business metric that is not related to the treatment effect, but can be explained by pre-existing data. By using pre-existing data to adjust for this variability, CUPED can help to isolate the true effect of the treatment, leading to more reliable and accurate results. This is especially important in A/B testing, where the goal is to detect small differences between treatment and control groups.

Integrating CUPED into test planning involves the following steps:

1. Calculate the Non-CUPED Sample Size: Use the regular t-test sample size formula.
2. Determine Pearson Correlation: Check the correlation between pre-experiment data $X$ and expected experiment data $Y$ using historical data. For example, compare data from the last two weeks with the two weeks prior.
3. Adjust Sample Size: Reduce the calculated sample size by the factor of ${\rho }^2$.

Example Procedure:

• Suppose the non-CUPED sample size is 1000.
• Historical sampled data shows an estimated Pearson correlation of 0.9 between $X$ and $Y$.
• Calculate the variance reduction factor: ${0.9}^2=0.81$.
• Adjust the sample size: $1000\times (1-0.81)=190$.

Thus, the required sample size using CUPED is 190.