Products
Solutions
Resources
“The sample size calculator says we need to run our experiment for 3 months, but we can’t wait that long. What should we do?”
We’ve gotten this question from many, many customers, so I wanted to write up an answer with the nuance it deserves. As it turns out, there’s a lot of nuance, so this is going to be a 4-part series.
Why nuance? Technically, one could write a quick “Top Tips for Shorter Experiments” listicle, because there are ways to adjust your test design for a shorter experiment.
However, any of these changes introduces risks and tradeoffs into your test. There’s no such thing as a free lunch. You have to be aware of what those are to assess your results accurately.
And sometimes, the answer to this question is, “Don’t run the test.” It’s completely valid to conclude that an experiment will take too long and that tradeoffs for shorter runtime aren’t worth it.
To evaluate that question accurately, you need to understand your options. That’s why we’ll cover them in this series. The series breakdown:
Post 1: How a sample size calculator works and what adjusting its inputs does to experiment runtime
Post 2: Adjusting your primary metric for faster results
Post 3: Different ways to target users for shorter runtime
Post 4: Four other types of experiments and how switching may speed things up
These solutions are by no means exhaustive, and not every method is applicable to every experiment, but they’re a good starting point.
Sample size calculators 101
Before beginning an experiment, one of the essential questions to answer is “How many days do I need to run the experiment for?” To do this, you’ll often use a sample size calculator (like this one).
A typical sample size calculator doesn’t directly tell you experiment duration, though. Instead, it calculates how many samples are needed per variant to detect a specific effect size with a specified false positive and false negative rate. To convert that output to a duration, you divide it by the average traffic per day to the experiment.
Statsig has a more advanced Power Analysis Calculator that can calculate duration for you based on actual user behavior patterns for the feature you want to test. Try it out!
Why does it matter how sample size calculators work when you just want to run a faster test? Because these are tools, not absolute authorities. By understanding the inputs for calculating experiment duration, you can start to see which levers to pull to output a shorter runtime directly: traffic rate, effect size, and error rate.
These levers can have a big impact on experiment duration, but also on the risk of errors. It’s helpful to learn how these work as a way to deepen your understanding of why experiments take time, but in terms of practicality, it may be preferable to use one of the less-risky fine-tuning adjustments I’ll cover in this series.
But in the interest of education, let’s discuss how they work.
1. Double-check your traffic rate
So this isn’t actually a method for pulling a lever differently; it’s just a way to make sure you’re pulling it accurately. If your experiment runtime seems way too long (or too short), confirm that the traffic input you fed to the calculator or used in your manual division is the right one.
Often, an accidental error can creep into this step from sample size to duration. Three things to be careful with when doing this division:
Divide by average traffic, not single-day traffic. You should take the average over at least one or two business cycles so that you get a representative number. You should also make sure to exclude holidays or other outlier days when you take the average.
Divide by the number of users exposed to the experiment per day, not total DAU. For example, if you are using some targeting criteria like running the experiment only in the US, then you should only divide by US traffic. Also, take into account where in the product you are running the experiment. Say you are running an experiment on your checkout page; you should divide by the users who view the checkout page, and not by daily active users on your whole site.
Be aware that this is a first-order approximation. You are assuming that the rate of users getting exposed to the experiment stays constant over the duration of the experiment, but that rate can decrease. Say you’re targeting users who have been on your platform for 30+ days, giving them a discount code. You design your experiment to only give one discount per user. On day 1, your target pool of 30+-day users is pretty large, and they’ll most likely have pretty similar characteristics. But by day 50, you’ll have already given most of those original 30+-day users a discount code, and the pool you’re drawing from will now contain more new users who have just entered the 30+-day cohort. See Amplitude’s “How to read the cumulative exposures graph” article for more information.
Changing the traffic to your experiment is a major topic in its own right that involves higher-level adjustments to your experiment design, and I’ll cover that more in the third post.
2. Increase MDE
Your minimum detectable effect (MDE) is the smallest change that you can reliably detect with your experiment. Bigger changes take less time to detect, so if you increase the MDE, your experiment duration decreases.
For example, say you’re experimenting to increase a feature’s conversion rate. The current rate is 5%. If your MDE is 10%, that means you’ll be able to detect a change of 0.5 percentage points. You’d also be able to detect changes larger than 0.5pp, but if the change is smaller, you wouldn’t see it.
Decreasing MDE to something like 5% would let you detect a change of 0.25pp, but you’d need more time at a given traffic rate to make sure you aren’t just seeing noise. Increasing MDE to something like 20% would mean your experiment takes less time, but you could only detect a change of 1pp.
Increasing your MDE in this way may align with practicality. Think about what your success criteria actually mean; if your experiment boosts your conversion rate from 5% to 5.25%, is that enough of a change to merit productionizing? Maybe for a feature that every single user interacts with, but maybe not for something that just serves a niche subset. If the engineering maintenance costs are only worth it with a larger lift, then a larger MDE (and therefore faster runtime too) makes total sense.
If you aren’t sure your one experiment justifies a coarser MDE, you could combine multiple experiments that are all trying to lift the same/similar metric. Essentially, you test a bigger change. Although this does mean you lose the ability to tell which change caused the effect.
As you reflect on your MDE, be careful to check if your sample size calculator uses a relative or absolute change. A 2% absolute difference on a 50% conversion metric will identify changes bigger than 52% and smaller than 48%, while a 2% relative difference would catch 51% and 49%. Set according to what you need.
3. Change your error rates
A caveat for this section: even though error rates are a fundamental lever for sample size calculators, data scientists disagree about if it’s advisable to change them.
Changing the error rate for a single given experiment may not be a huge risk, but it can quickly become a bigger problem if you do it across your entire experimentation program:
The more experiments you run with looser error thresholds, the more opportunities for errors. The risk of a false positive or false negative compounds.
To have experiments be comparable with each other, they need to be consistent within a certain scope/domain. Changing error rates for some experiments and not others makes them more difficult to compare.
If one team routinely uses a looser error rate and ships more “wins,” human nature can kick in and make it harder to pull your experimentation program back to rigorous levels.
More false positives
A false positive means that you’d make the mistake of claiming there was a difference between your treatment and control when there wasn’t one. Detecting and avoiding false positives takes data, so if you loosen your precision and allow for more possible false positives, you’ll need less data and can speed up your experiment.
Generally, people set their false positive rate, also called alpha, at 5%. Some calculators capture this as a “confidence level,” which is the complement of your alpha: a 5% alpha is a 95% confidence level. This rate means that if there’s truly no difference between treatment and control, you have a 1 in 20 chance of concluding there is one.
(Note: a 5% alpha doesn’t mean that 1 in 20 experiments will have a false positive result. People generally self-select features to experiment on that they think have a chance to succeed, so in practice, the occurrence of false positives across your experiment program would be rarer than 1 in 20.)
In some cases, you’d actually want to use a smaller alpha than 5%. For example, in medical trials, a mistake may mean giving a patient medicine that causes harm. You’d need to be more exact to avoid a serious negative outcome, and that would lengthen your experiment.
If you’re increasing your alpha to shorten the experiment, you could go as high as 20%. However, this would mean a 1 in 5 possibility of a false positive, and should be reserved for cases where the consequences of acting on a false positive are relatively benign.
More false negatives
A false negative means that there was a difference between treatment and control, but you did not detect it. Just like with false positives, these need data to detect, so if you allow for more possible false negatives, you can also speed up your experiment.
The industry-standard false negative rate, also called beta, is 20%. (Your experiment’s “power” is 1 - β.) That gives you a 1 in 5 chance of missing the signal on a real effect, which would mean you make the mistake of not rolling out something that could improve your product.
The standard beta is higher than the standard alpha because a false negative is a softer type of failure: not seeing an effect probably means you don’t ship whatever you were testing, so no actual mistake rolls out like with a false positive. And you’d still leave the door open to retest and see the missing signal at some later point. Absence of evidence is not necessarily evidence of absence.
If you decide it’s acceptable to increase your beta for a shorter experiment, you really shouldn’t go higher than 30%.
What else to try?
These three levers (or two levers and one double-check) are the most direct way to change your sample size output. If you pull on them as described, the calculator will tell you that you can run a shorter test.
However, as I said throughout this post, pulling these levers could also lead to more pronounced errors and long-term consequences. They’re heavy-handed changes instead of finessed adjustments.
If pulling these levers doesn’t get the time reduction you need, or the tradeoffs don’t work for you, those finessed strategies may be what you need. I’ll cover them in the next installments.
Next up is how to get a faster runtime by fine-tuning your metrics. I’ll follow that up with ways to change your targeting, and then the type of experiment you’re running.