Frequently Asked Questions

To roll out and monitor multiple new features simultaneously using Statsig, you can utilize the platform's Feature Gates and Experiments.

For each new feature you plan to roll out, create a corresponding Feature Gate. This approach automatically converts a feature roll-out into an A/B test, allowing you to measure the impact of the roll-out on all your product and business metrics as the roll out proceeds.

If you wish to test hypotheses between product variants, create an Experiment. An Experiment can offer multiple variants and returns a JSON config to help you configure your app experience based on the group that the user is assigned to.

To show features randomly to different sets of customers, use the 'Pass%' in a Feature Gate and 'Allocation%' in an Experiment. This allows you to control the percentage of users who see each feature.

Statsig's Experiments offer more capabilities for advanced experiment designs. For example, you can analyze variants using stable IDs for situations when users have not yet signed-up (or signed-in), or using custom IDs to analyze user groups, pages, sessions, workspaces, cities, and so on. You can also run multiple isolated experiments concurrently.

Remember to define your company goals and key performance indicators (KPIs) to measure the success of your features. You can break down these strategic goals into actionable metrics that you can move with incremental, iterative improvements.

If you use three different feature gates, you will find out how each feature, individually, performed against the baseline. If you want combinatorial impact analysis, like, A vs B vs C vs AB vs BC vs AC vs ABC, then you will need to setup an experiment with 7 variants and specify the combinations via parameters and measure.

However, in practice, this level of combinatorial testing isn’t always fruitful and will consume a lot of time. A pragmatic recommendation would be to use feature gates to individually launch and measure the impact of a single feature, launch the ones that improve metrics and wind down the ones that don’t.