CUPED is a statistical technique that leverages pre-experimental data to reduce variance and pre-exposure bias in experiment results. It was first popularized for online testing by Microsoft in 2013.

In the context of Statsig, CUPED is used to enhance the accuracy and speed of running experiments. It is particularly effective for metrics and behaviors that are predictable from past behavior. If a metric is consistent over time for the same user, CUPED can be very effective.

CUPED works by using information about an experiment's users from before an experiment started to reduce the variance in their experiment metrics. This pre-experiment information is referred to as a "control variate". The user's metric value is adjusted based on this control variate multiplied by a coefficient θ.

The more correlated the pre-experiment information is with the post-experiment information, the more of the error or noise in the experiment results is explained by the covariate, and the more the variance in the experimental term is reduced.

However, CUPED does not work on new users, because there is not pre-exposure data to leverage. It is also less effective if a user's metric value is uncorrelated with historical behavior.

In Statsig, CUPED is automatically applied to experiments and is run for the topline results on key metrics in Pulse. This leads to significant variance reduction in the large majority of metrics where CUPED can be applied.

CUPED is also used to address pre-experiment bias, which can occur when users in two experiment groups have meaningfully different average behaviors before any intervention is applied. If this difference is maintained after the experiment starts, it could cause misinterpretations of the results. CUPED helps to debias these experiments.

In the Statsig platform, all Scorecard metrics by default have CUPED applied to them. The "CUPED" flag above key metrics indicates that CUPED-adjusted results are available.

Related:

• CUPED Explained

• CUPED Methodology in Statsig