Bayesian A/B Test calculator

Some prefer Bayesian A/B tests for their simplicity and intuitiveness (and Statsig makes them effortless!)

In contrast with frequentist methodologies, Bayesian A/B tests don’t require p-values, null hypotheses, or confidence intervals. Instead, we can simply state the chances that our test variant beats our control variant, as provided by this calculator. Statsig has full support for Bayesian A/B tests.

How to use the Bayesian A/B Test calculator:

Enter the number of samples and number of positive outcomes you had for each of your test and control groups. Click calculate and you’re done! The calculator will provide the probability to be best of each group. Here you should apply your expected significance rate: a Probability to be Best of 95% or more is often considered best practice.

Limitations of This Calculator

Our calculator assumes infinite variation of priors, just like Bayesian A/B testing in Statsig, and is limited to integer inputs. If you have a percent or ratio to measure success, you’ll need to represent it as an integer. This calculator also only considers two variants (groups), but if you need more, consider launching an experiment on Statsig.

A/B Testing on Statsig

Statsig offers a robust toolkit for A/B testing including Bayesian and Frequentist methods, Switchback testing, Sequential testing, and a Multi-armed bandit model. If you’re interested in running experiments with Statsig, feel free to sign up for Statsig, join our Slack community, or book a demo.