Running faster tests: Shift your targets (Part 3)

“The sample size calculator says we need to run an experiment for 3 months, but we can’t wait that long. What do we do?”

We’ve gotten this question from many, many customers, so I wanted to write up an answer with the nuance it deserves. That has turned into a whole blog series, of which this is the third part.

The high-level answer: Yes, there are ways to change your experiment to reduce runtime. However, they all introduce tradeoffs and assumptions that can affect your results. There’s no such thing as a free lunch. And if the tradeoffs and timing both aren’t worth it, maybe you don’t run the test.

In order to make the tradeoff decision effectively, you need to understand what those tradeoffs are. In this part of the series, I cover how to adjust runtime by changing how you target users. These solutions are by no means exhaustive, but they’re a good starting point.

The Running Faster Tests series

Part 1: How sample size calculators work and how to adjust their inputs

Part 2: Adjusting your primary metric for faster results

Part 3: You’re here!

Part 4 (to come): Four other types of experiments and how switching may speed things up

How targeting works

When you run an experiment, you don’t show the treatment to all users. If you did, you’d have no information to compare the treatment’s results to. You couldn’t say if the treatment caused the result or if something else did, like an unrelated product glitch, or even a sudden change in the weather.

Instead, you split your user population, assigning some of them to the treatment and some to the control. If there’s a change in the treatment group’s behavior that you don’t see in the control group, you can infer that the change was caused by the treatment.

How you determine that split is called targeting. You might use a simple targeting rule, like randomly selecting 2% of your users to see the treatment. Or you might use a more complicated targeting system, like only showing a treatment to a portion of users in EMEA who interacted with another feature in the last 5 days.

Your targeting rules shape what your results mean, but they also influence your experiment’s runtime. The tighter your rule, the fewer the users who match it and the slower your rate of exposure. But that slower exposure rate may be worth a more precise learning. It may not even lead to a slower test.

Here are some targeting techniques you can try to adjust your runtime.

Target more users with your experiment

If you’re only doing a 2% rollout, what happens if you increase it to 20%? You may be keeping the percentage rollout small because you are risk-averse or unsure of the effect on your customers. If that’s the case, you could run the experiment for one business cycle at 2% just to make sure there isn’t a negative effect, and if things look good, increase the rollout percentage to 20% the next cycle. (See this article for why it’s important to run for an integer number of business cycles before changing the percentage rollout.)

You may have a smaller pool of users because a teammate is running a different experiment, and they want you to exclude their users. If you are running mutually exclusive experiments like this, it may be worth it to just wait until the other experiment is done to run yours. Or you may be able to figure out with your teammate if mutual exclusion is really needed; it’s possible the features are unrelated and have a very low chance of affecting each other.

Similarly, maybe you are only targeting a subset of users because the feature doesn’t work for all users, like a feature on iOS that you haven’t had engineering resources to build on Android yet. Can you wait for the feature to be built on Android? It may take less time to wait and run the test with a larger sample size than to start it now for only iOS users.

Who you target may also depend on your experiment design. If you are testing out multiple variants, can you remove one or more variants that you are less confident in having a positive lift? Fewer variants means more of your users are valid targets for the remaining variants. (This also helps with the multiple hypothesis testing problem.)

Also, consider if your exposure event is filtering out more users than it needs to. For example, say you’re experimenting on your search algorithm. If your exposure event only sends when someone clicks on a search result, you’re losing all the users who saw the treatment but didn’t click. Make sure to send the exposure event when the control and treatment experiences are first different.

Exposures vs assignment (or allocation)

In an experimentation platform, there are two aspects of targeting to configure and track: assignment and exposure. An assignment event fires when your targeting rule selects a user for either treatment or control. In Statsig, this typically happens automatically with the SDK, but you can also set it up to trigger manually. An exposure event fires when the user actually encounters the treatment.

Target fewer users with your experiment

On the other hand, instead of adding more users to your experiment, you can target fewer users and still speed things up. This may seem counterintuitive, as you generally want large sample sizes for faster tests. However, by restricting who is in the experiment, you can have a larger MDE since you are including only the important users.

To make sure your target is precise enough, the first thing is to identify the people who are likely to be affected by the treatment. For example, say you are experimenting on a feature that the user needs to buy, and it costs a lot of money. Targeting all of your users will include people who have never spent any money on your platform. Those users are highly unlikely to purchase the feature, which means they’re also highly unlikely to be affected by any experiments on that feature (i.e., the lift on those users is going to be very small). On the other hand, users who have spent a lot of money on your platform are more likely to purchase, and therefore likely to be affected one way or the other. The key point here is that detecting a small lift on lots of users is sometimes harder than detecting a larger lift on fewer users.

Similarly, say you’re trying to increase the number of paying customers for a “freemium” product. To increase a very low conversion rate, you need a very large sample size. But with freemium products, some users may not spend money, no matter what you do. Essentially, they are lost causes and shouldn’t be included in experiments. By identifying and filtering them out, you’re able to target a set of users who have a higher potential conversion rate, which makes identifying changes in that conversion rate easier, even though the sample size is smaller.

Likelihood may not even have anything to do with it: if you are experimenting on a feature that only subscribers have access to, and you include free users and subscribers in the experiment, you’re including users who can’t be affected by the treatment by definition.

Exposure events can also play a factor in focusing your target. If you’re experimenting on checkout flows, you may have your assignment event fire when a user lands on the homepage. If the exposure event fires then, too, you’ll be counting a user who may not visit the checkout page or even see the treatment. Just like you need to properly define your exposure event so it doesn’t fire too late, you also need it not to fire too early.

Targeting to reduce standard deviation

In Post 2, we talked about why reducing standard deviation can make your experiment run faster, and how you can transform or pick different metrics in order to do that. You can also reduce standard deviation through targeting.

Different segments of users may have larger or smaller standard deviations for your experimental effect. Often, a generic targeting rule may capture multiple segments with higher and lower deviations. If you’re able to target just the segments with a smaller standard error, you can get a faster test.

Say you have two segments of 30 users each in control. One segment has a mean = 0 and standard deviation = 1, and the other segment has a mean = 0 and standard deviation = 3. The treatment lift is 2, and the standard deviations are the same. Most of the time, you get a smaller p-value if you run a t-test on the first segment of users than if you run a t-test on both segments of users.

The reason for this is that, if you look at the standard error for control with the first 30 users, it is 1/sqrt(30) = .183. The standard error for control with 60 users is sqrt((.5 * 1 + .5 * 3^2)/60) = .289. We get the numerator from the Law of Total Variance. The computations are similar for the treatment standard error in both cases. You can play with the examples to see what happens when the lifts in the two groups are the same and when the lifts are different.

This method does not always work. For example, if the standard deviation were 1.5 instead of 3, then about half the time you’d get a smaller p-value if you combine the 30 segments. And sometimes it is better to have more users.

Here’s some R code if you want to try it out yourself:

sim <- function()
{
  control = c(rnorm(n=30), rnorm(n=30, sd=3))
  treatment = c(rnorm(n=30, mean=2), rnorm(n=30, mean=2, sd=3))
  together = t.test(control, treatment)
  separate = t.test(control[1:30], treatment[1:30])
  return(together$p.value - separate$p.value)
}
simulations = replicate(1000, sim())
hist(simulations)
quantile(simulations, probs = seq(0, 1, .01))

One thing to keep in mind is that you should not be removing users from analysis after the experiment has run unless you have a really good reason to do so. Overall, it is really important to understand if you are targeting the right users and sending the exposure event at the right time. 

Use a different bucketing unit

Instead of randomizing on users, you may be able to increase your sample size by randomizing on sessions or some other smaller unit. Oftentimes, this is not possible because sessions for the same user may not be independent: if the same user sees both treatment and control, it is hard to attribute the metric change to one or the other. Also, the user will have an inconsistent user experience. 

Along a similar line of thinking, if you are a B2B company, you may bucket by company ID instead of user ID; every user at a given company is a chance to lift the company’s impact as a unit, instead of just per user.

Bucketing by company ID often makes sense from a user-experience perspective, too. Everyone in the company sees the treatment or the control, so you don’t have one person see a new button while their colleague doesn’t. This can make for odd inter-office conversations, especially if the feature you’re testing is an in-company social feature like a video chat. This also reduces the load on your customer support team, since they’ll know which accounts have which features enabled.

If you do bucket by company ID, it’s important to make sure whether your metric is at the company ID  level or the user level. Do you care if at least one person in a company converted, or do you care how many times each user converted? If you’re looking at user-level metrics, you can use the delta method to correct for the randomization unit being different than the analysis unit. Using user-level metrics may help decrease time to statistical significance. The two competing effects are: 

1. You’ll increase the sample size because you have more users than companies

2. Generally, you’ll increase variance because people in the same company are positively correlated

What else to try?

Adjusting your targeting can speed up your experiment in a few different ways. Targeting more users can increase your sample size. Targeting fewer, more-affected users can increase the impact on your MDE. And targeting different segments or buckets can yield similar results.

If you try any of these methods, just make sure you don’t over-optimize to get statistical significance. If you get statistical significance on a really small sample size, the business impact could be really low. Think about whether it is better to get statistical significance on a small sample size or not get statistical significance but a directionally positive result on a much larger sample size. I would take caution when doing this.

If changing your targeting criteria still doesn’t get the time reduction you need, or the tradeoffs don’t work for you, check out the other posts in this series for more to try. In the next and final installment, I’ll go over different kinds of experiments you could try instead of a typical A/B test.