Experiment Scorecard and CUPED

Product Updates: Experiment Scorecard and CUPED

Rounding out the week with two exciting product launches! As always, don't hesitate to reach out here or 1:1 with product feedback, ideas, questions, etc. We love to hear from folks! 

Experiment Scorecard

Today, we’re introducing the ability to include an experiment hypothesis and primary/ secondary metrics at experiment creation, which will manifest in the form of an experiment “Scorecard” in your results tab.

While these fields are optional, the hope is that this feature makes it easier to standardize your experiment design process within the Statsig console, as well as improves the experience for non-experiment-creators reading experiments, enabling them to more fully understand key experiment context. 

CUPED (Controlled experiment Using Pre-Experimental Data)

As part of our bigger investment in a true Experiment “Scorecard”, we have implemented CUPED to automatically reduce variance and bias on all Scorecard metrics. CUPED is a statistical technique first popularized for online testing by Microsoft in 2013 that leverages pre-experimental data to reduce variance and pre-exposure bias in experiment results. Tactically, CUPED can significantly shrink confidence intervals and p-values, ultimately reducing the sample size and duration required to run an experiment. Which means you can run more experiments, faster!

CUPED will be applied by default to all Scorecard metrics (both Primary and Secondary), however you can toggle it on/ off directly above your Pulse results in the Scorecard. CUPED will not be applied to non-Scorecard metrics.

To read more about CUPED, check out our data scientist Craig’s awesome blog post here.