Platform
Resources
As AI becomes more integrated into the product development stack, it’s tempting to let it take the wheel—especially when it comes to experimentation.
But great experiments don’t run on autopilot. They require strong hypotheses, thoughtful analysis, and, yes, human judgment.
It's important to have a clear understanding of how AI can support experimentation across the lifecycle—before, during, and after a test—without replacing the critical thinking that makes it meaningful.
TLDR
Running A/B testing shouldn’t be automatic, and AI can make that problem even worse if it’s misused.
When used properly, AI can greatly improve the quality of A/B tests.
Generally, AI can boost experiments at three stages: before, during, and after.
Plus, a glance at features we’ve published, the ones on our roadmap, and ways you can share feedback and suggestions.
Why A/B testing shouldn’t be automatic
I’ve seen two main approaches to A/B testing in my career. The first one focuses on making clear decisions from test results. The second focuses on hypothesis and learning.
In practice, most experiments land somewhere in the middle. And many teams aim to do both, especially early on: make decisions efficiently and learn effectively.
However, the second approach, which emphasizes learning, has a natural disadvantage at many companies. Truly valuable learning calls for a solid mental model of the product.
Mental models are often hard to articulate, and they’re also tough for everyone to appreciate in the same way. In addition, the payoff from testing a hypothesis or exploring an idea is hard to pin down. It’s not always obvious or immediate.
Because of this challenge, many experimentation programs gradually tilt toward the first approach—making decisions and calling it a day. Without the right champion, though, this turns into a bureaucratic process: fill in the template, check off the boxes, trust the mechanical decision criteria. The more certainty, the better.
Under this mindset, we might want to remove humans from the loop entirely and automate more steps in the experiment process. Letting AI run every step only feeds that temptation—and that’s a slippery slope.
It’s especially risky now because several companies are already pitching “self-improving websites” that supposedly hand over all decision-making to AI. That might sound great on paper, but it’s a trap.
(Side note: this could easily be a whole separate article—on why humans need to stay in the driver’s seat of experiment decisions, why mental models matter more than we admit, and why we can’t ignore the qualitative side of results just because the quantitative side is easier to measure.)
In short, here’s my stance:
Experiments must be driven by good hypotheses.
Learning is just as important as making decisions.
Except in rare cases like ML experiments, final calls shouldn’t be purely quantitative.
Once we ditch the idea that “AI should do everything” and keep people in charge, we’ll find many ways that AI can help us run better experiments. Let’s check out how AI can enhance each stage of an A/B test—before, during, and after.
How AI can help at each stage
Before the experiment
In the planning phase, AI can offer insights on potential pitfalls or help you fine-tune your hypothesis. For instance, a large-language-model (LLM) tool can scan your test plan and point out assumptions you might want to question. It can also suggest relevant metrics or propose ways to collect data that’ll make your hypothesis more solid.
AI can also assist with ideation by mining historical experiment logs, support tickets, or product reviews to surface recurring pain points or areas of friction. By identifying common themes or unmet needs across large datasets, LLMs can help teams brainstorm more impactful hypotheses—ones grounded in real user signals, not just intuition. This keeps your tests focused on problems that actually matter.
During the experiment
While the experiment is in progress, AI can help spot weird anomalies or alert you if certain user segments behave in unexpected ways. Instead of waiting until the end of your test to find out something went wrong, you can use AI-driven monitoring to catch issues early. This can save a lot of time and keep you from making bad decisions based on skewed data.
After the experiment
Once you’ve wrapped up your test, AI can help interpret the results in a more nuanced way. It can sift through mountains of data, highlight hidden patterns, and link insights to your original hypothesis. You can then compare these patterns to your mental model, decide whether they confirm or challenge it, and figure out the best path forward.
Post-analysis, AI can help connect the dots between experiments, finding patterns or themes across multiple tests that might otherwise go unnoticed. For instance, it might recognize that multiple tests related to onboarding all point to friction around a certain feature. This meta-analysis can help you zoom out and prioritize broader strategic opportunities, not just individual wins.
How Statsig supports your AI experimentation journey
We’ve already rolled out a few AI-driven features aimed at making each stage of the experiment lifecycle smoother and more reliable. Some of these include:
Hypothesis assistance: quick suggestions on how to refine your test idea before you start.
Automated anomaly detection: real-time alerts for odd behavior or shifts in key metrics.
Intelligent post-test reports: clear summaries of complex findings, with calls to action you can tweak.
On our roadmap, we’re looking at more ways to use large language models to make the entire process more intuitive. This could mean deeper integration of LLMs into your analytics tools or plugins that help you track changes in user sentiment over time.
What’s next
We believe human insight should guide experiment decisions, and that AI should offer support rather than replace people.
We’d love to know what you think about these ideas and our approach. Let us know if there’s anything in particular you’d like to see, or if you have suggestions on how to refine our roadmap.
Experimentation is a journey that should bring both solid decision-making and genuine learning. With the right mindset and well-placed AI tools, you’ll get the best of both worlds. We look forward to building that future with you.
Talk to the pros, become a pro
