A/B testing for dummies

Emma Dahl
Wed Oct 06 2021

This is what I googled on my first day with Statsig.

Since then, my level of understanding has graduated from preschool to elementary- nice!

In marketing and sales, we often advocate for platforms and tools that we don’t use ourselves- after all, we’re not engineers. But, that means folks like me have a lot of learning to do. I’m fortunate enough at Statsig to be surrounded by really, really smart people, and after picking their brains for the last month, I’d like to pass along some learning to my fellow non-technical people.

A/B testing is, in short, the reason why the apps and websites you use work so well. Building great products used to be a lot of guesswork, and the issue with this is that we always think our ideas are good ideas. Turns out, only about 1/3 of our ideas are good. This is where A/B testing matters.

Let me give you an example- for awhile, Facebook was testing out pimple popping videos to engage more folks on video. It’s click-batey, but people can’t help but watch (read: I can’t help but watch). The issue though, is that after a few minutes, people are disgusted with themselves and stop using the platform altogether.

On the backend, Facebook was running an A/B test- some people got pimple videos (A group), and some didn’t (B group). After enough folks are exposed to the videos, we can make data-informed decisions about which ideas are good, and which aren’t.

It’s easier to call someone’s baby ugly when you have data to back it up.

A/B testing means we can try out more features and quickly see which ones work. Instead of just holding onto one idea and running with it for 6 months, by progressively rolling out new ideas all the time, companies build better products faster.

This is why Statsig is cool- product teams normally spend a ton of time building out their testing infrastructure, so every feature (think- new button, different color, change in search feature, etc.) you test matters a lot. Statsig takes care of the back-end infrastructure, which translates to your team spending a nominal amount of time on testing new features, and more time coming up with new ideas.

If you’re interested in learning more, join our Slack community or sign up for a demo account.

Try Statsig Today

Explore Statsig’s smart feature gates with built-in A/B tests, or create an account instantly and start optimizing your web and mobile applications. You can also schedule a live demo or chat with us to design a custom package for your business.


Recently published

My Summer as a Statsig Intern


This summer I had the pleasure of joining Statsig as their first ever product design intern. This was my first college internship, and I was so excited to get some design experience. I had just finished my freshman year in college and was still working on...

Read more

Long-live the 95% Confidence Interval


The 95% confidence interval currently dominates online and scientific experimentation; it always has. Yet it’s validity and usefulness is often questioned. It’s called too conservative by some [1], and too permissive by others. It’s deemed arbitrary...

Read more

Realtime Product Observability with Apache Druid


Statsig’s Journey with Druid This is the text version of the story that we shared at Druid Summit Seattle 2022. Every feature we build at Statsig serves a common goal — to help you better know about your product, and empower you to make good decisions for...

Read more

Quant vs. Qual


💡 How to decide between leaning on data vs. research when diagnosing and solving product problems Four heuristics I’ve found helpful when deciding between data vs. research to diagnose + solve a problem. Earth image credit of Moncast Drawing. As a PM, data...

Read more

The Importance of Default Values


Have you ever sent an email to the wrong person? Well I have. At work. From a generic support email address. To a group of our top customers. Facepalm. In March of 2018, I was working on the games team at Facebook. You may remember that month as a tumultuous...

Read more

CUPED on Statsig


Run experiments with more speed and accuracy We’re pleased to announce the rollout of CUPED for all our customers. Statsig will now automatically use CUPED to reduce variance and bias on experiments’ key metrics. This gives you access to a powerful experiment...

Read more

We use cookies to ensure you get the best experience on our website.

Privacy Policy