This is why health checks are a critical part of an experimentation platform—the more you’re proactively alerted about potential issues, the less likely you are to make a bad ship decision—and worse (in this case), have a bad learning experience.
I was pretty sure that the copy on the Statsig homepage was a little clunky and decided to sneak in a test. This was an evenly split a/b/c test that experimented with different CTA texts.
I peeked after a few days. I applied sequential testing, and the results had a high observed lift, but without enough power to be statistically significant. My initial takeaway was that we needed to run this for longer.
Had I waited and let this run, we would have likely ended up seeing a “Statistically Significant, Positive Result” for these simple conversion metrics. I now know that these results were actually driven by bad input data.
I caught this because Statsig automatically checked for outliers influencing my results. In this case, when we had set up our homepage project, we set up our click metric without winsorization. This was causing issues since a few outlier users dominated the metric value.
I was able to quickly use Metrics Explorer to confirm my hypothesis about what had happened here. Using the “group by experiment” functionality, it was easy to see that, on different days, two of the branches had a huge number of clicks.
Drilling into this by stable ID, I identified the two users who caused the erroneous data and ended up running an explore query to filter these users out.
There were a few ways to approach this problem:
Change/Add winsorization to manage the influence of these outlier users, or add metric caps to a reasonable number like 5 signup clicks/day
Use an explore query or qualifying event filter to eliminate these two users from the analysis
Use an event-user metric instead
Use Statsig’s recently released Bot Detection
In this case, I chose to make a filter query since these users had odd behavior across a number of metrics, and I was confident that their behavior wasn’t caused by the experiment arms. This allowed us to run this experiment and get results that weren’t tainted by the bad users—you can check out our website to see the current variant.
At Statsig, we use our own product for internal infra experimentation as well as client-side releases and homepage web optimization. This has really helped us in looking for opportunities—like this one—to prevent issues that users might run into.
It’s also been a big factor in our investment into analytics—in particular, being able to analyze root causes of your experiment results is an incredibly powerful tool for understanding “what happened under the hood.”
Learn key insights from Ronny Kohavi and Allon Korem on building a strong experimentation culture, infrastructure, and learning from failures in A/B testing.
From Marketplace failures to a game-changing A/B test, Deltoid and causal evidence reshaped Facebook's product strategies as well as my own beliefs.
My first few months at Statsig were full of hackathons, team building, and some seriously cool projects. Find out what makes Statsig's culture special.
Kayak reacted quickly to news coverage of airline-related catastrophes and gave its Aircraft Filter feature more visibility, resulting in a 15x increase in user engagement.
A/B testing is the most reliable way to get evidence. Whether you're an advanced experimenter, or delving into testing for the first time, here's what you should know:
The new Pluto design system is an effort to simplify and enhance your experience with Statsig. Take a look for yourself!