Hacks with customers: Experiment quality score

Building is better with friends

We recently had a visit to Statsig HQ from several members of a customer’s team that is responsible for ensuring that the A/B testing performed across the entire company is done rigorously. They have their own platform for evaluating which experiments adhere to best practices, but the biggest challenge was getting each team to look in two places for information about how they were doing.

While they were here on the ground, we dug into how they score experiments and what it might look like if we put this information front and center in the Statsig product. We also needed to explore how this could be customized and used for other organizations.

After a few hours of iterating, we landed on Experiment quality score, a metric designed to help experimenters identify potential issues early and measure experimentation maturity over time.

What is the experiment quality score?

The experiment quality score provides a structured way to assess the setup and execution of an experiment in Statsig. By evaluating key criteria—like targeting, sample size, and metric selection—it helps ensure that experiments are configured correctly, data collection is reliable, and results can be trusted.

Measuring experiment quality over time also helps teams track improvements in their experimentation process and level up their decision-making confidence.

How to enable and configure experiment quality score

To start using the experiment quality score, head to:

Settings → Experimentation → Experiment Quality Score

Here, you’ll find a list of pre-defined assessment criteria that Statsig uses to evaluate experiment quality. You can also customize the weight of each factor based on what’s most relevant to your team.

Where to view the experiment quality score

Once enabled, experiment quality score will appear on any configured experiment. Each applicable check will be evaluated automatically, giving you a quick sense of how well the experiment is set up.

This feature is rolling out now, and we’re excited to see how it helps teams run better, more reliable experiments.

For help with setup, check out our documentation or join our Slack community and reach out.

Request a demo

Statsig's experts are on standby to answer any questions about experimentation at your organization.

Grab a Demo