Bayesian A/B Test calculator

Test the Bayesian probability to be best for each group in your experiment
Sample size calculator
OpenAI Univision Brex Microsoft ea Notion Anthropic affirm Tempo flutter grammarly OfferUp
Group A
Group B
A: SUCCESS RATE

1.5%

B: SUCCESS RATE

1.4%

A: CHANCE TO BEAT

72.3%

B: CHANCE TO BEAT

27.7%


Some prefer Bayesian A/B tests for their simplicity and intuitiveness (and Statsig makes them effortless!)

In contrast with frequentist methodologies, Bayesian A/B tests don’t require p-values, null hypotheses, or confidence intervals. Instead, we can simply state the chances that our test variant beats our control variant, as provided by this calculator. Statsig has full support for Bayesian A/B tests.

How to use the Bayesian A/B Test calculator:

Enter the number of samples and number of positive outcomes you had for each of your test and control groups. Click calculate and you’re done! The calculator will provide the probability to be best of each group. Here you should apply your expected significance rate: a Probability to be Best of 95% or more is often considered best practice.

Limitations of This Calculator

Our calculator assumes infinite variation of priors, just like Bayesian A/B testing in Statsig, and is limited to integer inputs. If you have a percent or ratio to measure success, you’ll need to represent it as an integer. This calculator also only considers two variants (groups), but if you need more, consider launching an experiment on Statsig.

A/B Testing on Statsig

Statsig offers a robust toolkit for A/B testing including Bayesian and Frequentist methods, Switchback testing, Sequential testing, and a Multi-armed bandit model. If you’re interested in running experiments with Statsig, feel free to sign up for Statsig, join our Slack community, or book a demo.

Join the #1 experimentation community

Connect with like-minded product leaders, data scientists, and engineers to share the latest in product experimentation.

Try Statsig Today

Get started for free. Add your whole team!
We use cookies to ensure you get the best experience on our website.
Privacy Policy