The term "Bayesian" refers to a statistical approach that is based on Bayes' theorem, named after the 18th-century mathematician Thomas Bayes. This approach differs from classical (frequentist) statistics in its interpretation of probability. Bayesian statistics allows for the incorporation of prior knowledge or belief into the statistical analysis, which is updated with new data to provide a posterior probability.

In the context of Bayesian experiments, the Bayesian approach allows you to start with some preconceived notion of what the probability is, and since you don’t have perfect information about what it is, it is allowed to be a probability distribution. This distribution is parameterized by a certain parameter (for example, a beta distribution with the parameters α and β). When you collect new data, you have new information and thus can update your beliefs of what the probability is. This updating process leverages the Bayes Rule.

Bayesian A/B testing leverages this statistical framework for hypothesis testing. It is often selected for its simplicity and intuitiveness. Practitioners do not have to deal with the complexities of p-values, null hypotheses, nor unintuitive confidence intervals. Instead, you can make statements like: "We expect a lift in our metric by 10% +/- 5%. Overall, there is an 80% chance that the test variant beats the control variant and if we roll it out, there is a risk that we degrade our metric by an expected amount of 0.5."

However, building a useful and robust prior model depends on a lot of factors including the amount of information you have, experimental context, and assumptions you’ve made. This is quite challenging to construct and is often criticized for introducing bias to your experimental results.

Bayesian A/B test calculator

Statsig's Bayesian calculator is a quick way to determine the chances that a test variant beats a control variant.

Join the #1 Community for Product Experimentation

Connect with like-minded product leaders, data scientists, and engineers to share the latest in product experimentation.

Try Statsig Today

Get started for free. Add your whole team!

What builders love about us

OpenAI OpenAI
Brex Brex
Notion Notion
SoundCloud SoundCloud
Ancestry Ancestry
At OpenAI, we want to iterate as fast as possible. Statsig enables us to grow, scale, and learn efficiently. Integrating experimentation with product analytics and feature flagging has been crucial for quickly understanding and addressing our users' top priorities.
Dave Cummings
Engineering Manager, ChatGPT
Brex's mission is to help businesses move fast. Statsig is now helping our engineers move fast. It has been a game changer to automate the manual lift typical to running experiments and has helped product teams ship the right features to their users quickly.
Karandeep Anand
At Notion, we're continuously learning what our users value and want every team to run experiments to learn more. It’s also critical to maintain speed as a habit. Statsig's experimentation platform enables both this speed and learning for us.
Mengying Li
Data Science Manager
We evaluated Optimizely, LaunchDarkly, Split, and Eppo, but ultimately selected Statsig due to its comprehensive end-to-end integration. We wanted a complete solution rather than a partial one, including everything from the stats engine to data ingestion.
Don Browning
SVP, Data & Platform Engineering
We only had so many analysts. Statsig provided the necessary tools to remove the bottleneck. I know that we are able to impact our key business metrics in a positive way with Statsig. We are definitely heading in the right direction with Statsig.
Partha Sarathi
Director of Engineering
We use cookies to ensure you get the best experience on our website.
Privacy Policy