What is minimal detectable effect? How it impacts experiment results

Ever wondered how sensitive your experiments need to be to detect meaningful changes? When running A/B tests, it's crucial to know how small of a change you can reliably observe. That's where the concept of Minimum Detectable Effect (MDE) comes into play. MDE helps you understand the smallest effect size that your experiment is statistically powered to detect.

In this blog, we'll dive into what MDE is, why it's important, and how it impacts your experiment design. We'll also share some best practices for applying MDE in your experiments to drive impactful results. Let's get started!

Understanding minimum detectable effect (MDE)

When running experiments, especially A/B tests, it's important to know just how small of a change you can reliably detect. That's where the Minimum Detectable Effect (MDE) comes in. Simply put, MDE is the smallest change in a metric that your experiment can detect with statistical significance. It's like setting the sensitivity dial on your experiment.

Choosing the right MDE is crucial because it affects the sample size you need. If you set a lower MDE to detect tiny changes, you'll need a larger sample size. That means more traffic and often more time and cost. On the flip side, setting a higher MDE requires a smaller sample size but might cause you to miss out on smaller yet meaningful improvements.

Calculating MDE involves understanding your baseline metrics, like your current conversion rate, and deciding on your desired statistical significance and power levels. For example, if your conversion rate is 5%, and you set an MDE of 10%, you're aiming to detect a change of 0.5 percentage points. Detecting a 10% change might need about 2,922 conversions, but aiming for a 5% change could bump that up to 11,141 conversions!

MDE is all about aligning your experiment's sensitivity with your business goals. After all, you want to detect changes that are not just statistically significant but also make a practical difference. Tools like Statsig's power analysis calculator can help you set the right MDE and plan your experiments efficiently.

The role of MDE in experiment design

So, how does MDE fit into your overall experiment design? Well, it's a key player in determining how sensitive your experiment is and how many users you'll need. Setting the MDE involves a trade-off: lower MDE increases sensitivity but ups the cost and duration of your test. Higher MDE means you can run the experiment faster and cheaper, but you might miss out on detecting smaller impacts.

It's important to align your MDE with your business goals and the practical significance of potential changes. You don't want to spend resources chasing tiny effects that don't matter in the big picture. At the same time, you don't want to overlook small but meaningful improvements that could add up over time.

Tools like Statsig's power analysis calculator can help you find the sweet spot. By inputting your baseline metrics and desired significance levels, you can determine the appropriate MDE and sample size for your experiment. This way, you can make informed decisions and design experiments that are both efficient and effective.

Calculating MDE and its impact on sample size

So, how do you actually calculate your MDE? It starts with knowing your baseline metrics—like your current conversion rate. MDE is usually expressed as a percentage of this baseline. For instance, if your conversion rate is 5%, and you set an MDE of 10%, you're looking to detect a change of 0.5 percentage points.

Remember, setting a lower MDE (to detect smaller changes) requires a larger sample size. That's because detecting small differences with confidence needs more data. For example, detecting a 10% change might need about 2,922 total conversions, but aiming for a 5% change could require up to 11,141 conversions.

Using tools like Statsig's power analysis calculator makes this process easier. You can input your baseline rates, desired significance level, and power to see how MDE affects your required sample size. Adjusting parameters like split ratios and test types helps you fine-tune your experiment to fit your goals and resources.

Ultimately, it's all about balancing sensitivity with practicality. By understanding how MDE impacts your sample size and leveraging the right tools, you can design experiments that are both effective and efficient.

Best practices for applying MDE in experiments

When it comes to setting your MDE, start by aligning it with your business objectives and KPIs. Think about what changes would make a real difference to your organization. You want to detect changes that are not only statistically significant but also matter in the real world.

Make sure to balance sensitivity with practicality. It's usually not worth investing resources to detect minuscule changes that have little impact. Tools like Statsig's power analysis calculator can help you strike the right balance by estimating the sample sizes needed for different MDEs.

Also, consider the context and your users. The appropriate MDE might vary depending on factors like user segments or platforms. Tailoring your MDE settings to these nuances can make your experiments more effective.

By following these best practices and using the right tools, you can design experiments that provide meaningful insights and help drive your product forward.

Closing thoughts

Understanding and carefully setting your Minimum Detectable Effect (MDE) is key to running effective experiments. By aligning MDE with your business goals and balancing sensitivity with practicality, you can design tests that uncover meaningful insights without wasting resources. Tools like Statsig's power analysis calculator make it easier to plan and execute your experiments.

If you're interested in learning more about experiment design and MDE, check out our resources at Statsig. Hope you found this helpful!