Frequently Asked Questions

A curated summary of the top questions asked on our Slack community, often relating to implementation, functionality, and building better products generally.
Statsig FAQs
Filter Applied:

Can I get sticky results for an A/B test based on IP address?

Statsig does not support sticky results for A/B tests based on IP address. The primary identifiers used for consistency in experiments are the User ID and the Stable ID. The User ID is used for signed-in users, ensuring consistency across different platforms like mobile and desktop. For anonymous users, a Stable ID is generated and stored in local storage.

While the IP address can be included in the user object, it's not used as a primary identifier for experiments. The main reason is that multiple users might share the same IP address (for example, users on the same network), and a single user might have different IP addresses (for example, when they connect from different networks). Therefore, using the IP address for sticky results in A/B tests could lead to inconsistent experiences for users.

If you want to maintain consistency across a user's devices, you might consider using a method to identify the user across these devices, such as a sign-in system, and then use the User ID for your experiments.

For scenarios where users revisit the site multiple times without logging in, there are two potential options:

1. Run the test sequentially, only control at first, then only test group. This is known as Switchback testing. You can learn more about it in this blog post and the technical documentation.

2. Offer a way to switch between control/test group visually for the user so they can bounce back to the behavior they'd expect from being on another device.

However, if there's a lengthy effect duration, Switchback may not be ideal. If you are able to infer the IP address, you can use this as the user identifier (maybe even as a custom identifier) and randomize on this. But be aware that skew in the number of users per IP address may introduce a significant amount of noise. You may want to exclude certain IP addresses from the experiment to get around this.

The skew comes from IP addresses that represent dozens if not hundreds of users. This can skew some of the stats when we try to infer confidence intervals. For example, instead of a conversion rate of 0/1, or 1/1, this metric looks like 36/85. This overweights both the numerator and denominator for this "user" which can skew the results.

Can Statsig analyze the impact of an experiment on CLV after a longer period, like 6 months?

In a subscription-based business model, understanding the long-term impact of experiments on Customer Lifetime Value (CLV) is crucial.

Statsig provides the capability to analyze the impact of an experiment on CLV over extended periods, such as 6 months. To facilitate this, Statsig allows for the setup of an experiment to run for a specific duration, such as 1 month, and then decrease the allocation to 0%, effectively stopping new user allocation while continuing to track the analytics for the users who were part of the experiment.

This tracking can continue for an additional 5 months or more, depending on the requirements. It is important to note that the experience delivered to users during the experiment will not continue after the allocation is set to 0%. However, there are strategies to address this, which can be discussed based on specific concerns or requirements.

Additionally, Statsig experiments by default expire after 90 days, but there is an option to extend the experiment duration multiple times for additional 30-day periods. Users will receive notifications or emails as the expiration date approaches, prompting them to extend the experiment if needed.

This functionality is available on the Pro plan, ensuring that businesses can effectively measure the long-term impact of their experiments on CLV without direct integration with a data warehouse and by updating CLV through integrations such as Hightouch.

Can two different web projects share the same Statsig API keys and what is the impact on billing?

How can we conduct QA for an experiment if another experiment is active on the same page with an identical layer ID?

To conduct Quality Assurance (QA) for your experiment while another experiment is active on the same page with an identical layer ID, you can use two methods:

1. Creating a New Layer: You can create a new layer for the new experiment. Layers allow you to run multiple landing page experiments without needing to update the code on the website for each experiment. When you run experiments as part of a layer, you should update the script to specify the layerid instead of expid. Here's an example of how to do this:

html <script src="[API_KEY]&layerid=[LAYER_NAME]"></script>

By creating a new layer for your new experiment, you can ensure that the two experiments do not interfere with each other. This way, you can conduct QA for your new experiment without affecting the currently active experiment.

2. Using Overrides: For pure QA, you can use overrides to get users into the experiences of your new experiment in that layer. Overrides take total precedence over what experiment a user would have been allocated to, what group the user would have received, or if the user would get no experiment experience because it is not started yet. You can override either individual user IDs or a larger group of users. The only caveat is a given userID will only be overridden into one experiment group per layer. For more information, refer to the Statsig Overrides Documentation.

When you actually want to run the experiment on real users, you will need to find some way to get allocation for it. This could involve concluding the other experiment or lowering its allocation.

How does Statsig differentiate data from different environments and can non-production data be used in experiments?

Statsig differentiates data from different environments by the environment tier you specify during the SDK initialization. You can set the environment tier to "staging", "development", or "production". By default, all checks and event logs are considered "production" data if the environment tier is unset.

Experiments and metrics primarily factor in production data. Non-production events are visible in diagnostics, but they are not included in Pulse results. This is because most companies do not want non-production test data being included. If you want to include these, you can log them as regular events. However, non-production data is filtered out of the warehouse and there is no other way to include it.

When initializing, if you add { environment: { tier: "production" } }, this would set your environment to "production", not "staging" or "development". If you want to set your environment to "staging" or "development", you should replace "production" with the desired environment tier.

Pulse results are only computed for “Production” tier events. To see Pulse results, you need: • An experiment that is “Started” (aka, enabled in production) • Exposures & events in production tier • Exposures for users in all test groups • Events/metrics associated to users in test groups

As long as you have initialized using { environment: { tier: "production" } }, your Pulse will compute. This means that even if your code is deployed to staging, as long as you initialize with the production tier, you will be able to see Pulse results.

How to change the owner of our account and upgrade our tier when the previous owner has left the company?

If you need to change the owner of your account and upgrade your tier, but the previous owner has left the company, you can reach out to our support team for assistance. Please email them at from an account that has administrative privileges.

Our support team can help you change the owner of your account and upgrade your tier. To do this, you will need to provide the email of the person you would like to change the owner to.

Please note that this process requires backend changes, which our support team can handle for you. Ensure that you have the necessary permissions and information before reaching out to the support team.

How to configure holdouts to measure the impact of different product teams?

How to ensure real-time logging of events works via the HTTP API?

When utilizing the HTTP API to log events in real-time, it is crucial to ensure that the events are properly formatted to be recognized by the system. If events are not appearing in the Metrics Logstream, it may be necessary to wrap the events in an array using the events:[] syntax. This is particularly important because events are typically sent in bulk, and the correct formatting is essential for them to be processed and displayed in the stream.

Additionally, it is recommended to use the SDKs provided for various languages, as they handle a lot of the heavy lifting, including performance and reliability improvements. For cases where the SDKs are not an option, such as with certain desktop applications or specific frameworks like Qt, it may be possible to use language bindings to integrate with the SDKs of supported languages. This can potentially save significant development effort and ensure more reliable event logging.

Furthermore, for the best practice regarding experiments and configurations, it is advised to fetch the experiment config at the time when it is needed to ensure accurate exposure logging. The SDKs typically fetch all evaluations for a single user in one shot and cache it for the session duration, allowing for local and instantaneous experiment config or gate checks. If using the HTTP API directly, it is possible to use an 'initialize' endpoint, which is typically reserved for SDK use.

This endpoint fetches all evaluated results for a given user without generating exposure logs at the same time, thus avoiding overexposure issues.

How to optimize landing page loading without waiting for experiment configurations

When optimizing landing page loading without waiting for experiment configurations, it is recommended to use a custom script approach if you need to pass in specific user identifiers, such as Segment's anonymous ID.

This is because the standard landing page script provided by Statsig does not allow for the initialization with a different user ID. Instead, it automatically generates a stableID for the user, which is used for traffic splitting and consistent user experience upon revisits. This stableID is stored in a cookie and is used by Statsig to identify users in experiments.

However, if you need to synchronize with metrics from Segment using the Segment anonymous ID, you may need to deconstruct the landing page tool and implement a custom version that allows you to set the userObject with your Segment anonymous ID.

Additionally, you can enrich the Segment User with an additional identifier, such as statsigStableID, which can be obtained in JavaScript using statsig.getStableID(). This ID can then be mapped from the Segment event payload to Statsig's stableID. If performance is a concern, you can bootstrap your SDK with values so you don't have to wait for the network request to complete before rendering.

This can help mitigate performance issues related to waiting for experiment configurations. For more information on bootstrapping the SDK, you can refer to the official documentation on bootstrapping.

How to roll out and monitor multiple new features simultaneously using Statsig?

To roll out and monitor multiple new features simultaneously using Statsig, you can utilize the platform's Feature Gates and Experiments.

For each new feature you plan to roll out, create a corresponding Feature Gate. This approach automatically converts a feature roll-out into an A/B test, allowing you to measure the impact of the roll-out on all your product and business metrics as the roll out proceeds.

If you wish to test hypotheses between product variants, create an Experiment. An Experiment can offer multiple variants and returns a JSON config to help you configure your app experience based on the group that the user is assigned to.

To show features randomly to different sets of customers, use the 'Pass%' in a Feature Gate and 'Allocation%' in an Experiment. This allows you to control the percentage of users who see each feature.

Statsig's Experiments offer more capabilities for advanced experiment designs. For example, you can analyze variants using stable IDs for situations when users have not yet signed-up (or signed-in), or using custom IDs to analyze user groups, pages, sessions, workspaces, cities, and so on. You can also run multiple isolated experiments concurrently.

Remember to define your company goals and key performance indicators (KPIs) to measure the success of your features. You can break down these strategic goals into actionable metrics that you can move with incremental, iterative improvements.

If you use three different feature gates, you will find out how each feature, individually, performed against the baseline. If you want combinatorial impact analysis, like, A vs B vs C vs AB vs BC vs AC vs ABC, then you will need to setup an experiment with 7 variants and specify the combinations via parameters and measure.

However, in practice, this level of combinatorial testing isn’t always fruitful and will consume a lot of time. A pragmatic recommendation would be to use feature gates to individually launch and measure the impact of a single feature, launch the ones that improve metrics and wind down the ones that don’t.

How to send heads-up emails to users before exposing them to a new feature?

In order to notify users before they are exposed to a new feature, you can create a separate feature gate to control the rollout percentage and a segment to contain the account IDs that can be exposed to the feature. This approach allows you to effectively manage the rollout process and ensure that users are notified in advance.

The main feature gate will only pass for the accounts in the segment that also pass the separate feature gate. This provides a clear distinction between the users who are eligible and those who have been exposed to the feature.

Here's a brief overview of the process:

1. Create a main feature gate (rollout_feature_gate). The users that pass this gate will be exposed to the feature.

2. Create a separate feature gate (exposure_eligibility_gate) to control the rollout percentage. The users that pass this gate are the ones eligible to be exposed to the feature.

3. Create a segment (allegeable_accounts) that contains all the account IDs that can be exposed to the feature.

The rollout_feature_gate will return pass only for the accounts in the allegeable_accounts segment, while questioning the exposure_eligibility_gate. After a certain amount of time, export all account IDs in the exposure_eligibility_gate to the allegeable_accounts segments, and increase the exposure_eligibility_gate percentage.

This approach allows you to have a distinction between the eligible users that were exposed to the feature (allegeable_accounts segment) to the ones that are allegeable but potentially not yet exposed to it (exposure_eligibility_gate). You can manage additional rules and environment conditions in the main feature gate (rollout_feature_gate).

Remember to test your setup thoroughly in a pre-production environment before rolling it out to ensure everything works as expected.

Is there a limit to the number of dynamic configs in Statsig and what are the effects of having a large number?

In Statsig, there is no hard limit to the number of dynamic configs you can create. However, the number of configs can have practical implications, particularly on the response size and latency.

Having a large number of dynamic configs can impact the initialization for both server and client SDKs. For Server SDKs, they will have to download every single config and all of their payloads during initialization, and on each polling interval if there’s an update available. This won't necessarily impact user experience, but it does mean large payloads being downloaded and stored in memory on your servers. You can find more information on Server SDKs here.

On the other hand, Client SDKs, where 'assignment' takes place on Statsig’s servers by default, will have to download user-applicable configs and their payloads to the user’s device during initialization. This increases the initialization latency and could potentially impact user experience. More details on Client SDKs can be found here.

In conclusion, while there is no explicit limit to the number of dynamic configs, having a large number can increase complexity and affect performance due to the increased payload size and latency. Therefore, it's important to consider these factors when creating and managing your dynamic configs in Statsig.

Understanding the difference between daily participation rate and one time event metrics in Statsig

In Statsig, the Daily Participation Rate and One Time Event metrics are used to track user behavior in experiments. The choice between these two metrics depends on the nature of the event you're tracking.

1. Daily Participation Rate: This metric is calculated as the total number of days that a user has the selected event, divided by the number of days the user is in the experiment. This is done for each user in the experiment. The mean event_dau, or the average active days per user, is then calculated by aggregating this average event_dau for each user in the experiment, with each user weighted equally. This metric is more suitable for events that are expected to occur repeatedly for a given user.

2. One Time Event: This metric is ideal for events that are only expected once per user, such as booking events. If the event is expected to occur only once per user during the experiment or holdout period, then the One Time Event metric would be suitable.

For longer experiments and holdouts, the choice of metric would still depend on the frequency of the event. If the event is expected to occur approximately once a month or less frequently, the One Time Event metric should be appropriate. However, if the event is expected to occur approximately weekly or more frequently, the Daily Participation Rate metric might be more appropriate as it captures recurring behavior.

When reviewing experiments, consider all related metrics:

- One-time events best capture the number of unique users who participated in the event. - Daily participation rate is an effective proxy for "how much" people are participating in the event. - Total events (event_count) is a better proxy for revenue or downstream metrics.

For holdouts, it can be helpful to use different rollups. For example, looking at one-time metrics for the 7-day or 28-day rollup would tell you what % of users participated (at all) within the last 7-day or 28-day window. This can be an effective way to get past the history issue.

What is the best practice for using one Statsig account for both dev/test and production environments?

Statsig currently supports metrics calculation for a single production environment. If you have multiple environments set up, you will only be able to use one as your production environment for metrics and experiments.

When running an experiment, the default behavior is that the experiment will run in all environments. This means that if you try to access the config values of the experiment, you will get a valid value, even if you are in a non-production environment.

If you want your experiment to only run on production, you can set a targeting gate. This will ensure that only users in the production environment will pass the gate and be included in the experiment.

Here is an example of how you might access the config values of an experiment:

javascript useExperiment(XXX).config.get(YYY, false);

In this example, XXX is the experiment you are running, and YYY is the config value you are trying to access. If the experiment is not running in the current environment, you will get the default fallback value, which is false in this case.

Remember, once you start an experiment, it will run in all environments unless you set a targeting gate to restrict it to specific environments.

The best practice is to use the experiment checklist and diagnostics tab to instrument the test, enable it in lower environments, and validate that exposures are flowing through. Then, when you’ve validated these, you click “Start” to go to production. This workflow is typically adequate for most users.

Please note that this is subject to change as Statsig continues to evolve and add new features. Always refer to the official Statsig documentation for the most up-to-date information.

What is the recommended way to roll out a feature customer by customer in a B2B context using Statsig?

In a B2B context, the recommended way to roll out a feature customer by customer is by using feature gates. You can create a feature gate and add a list of customer IDs to the conditions of the gate. This way, only the customers in the list will pass the gate and have access to the feature, while the rest will go to the default behavior.

Here's an example of how you can do this:  

const user = {      userID: '12345',      email: '',      ...   };       const showNewDesign = await Statsig.checkGate(user, 'new_homepage_design');   if (showNewDesign) {      // New feature code here } else {      // Default behavior code here }  

 In this example, 'new_homepage_design' is the feature gate, and '12345' is the customer ID. You can replace these with your own feature gate and customer IDs.On the other hand, Dynamic Configs are more suitable when you want to send a different set of values (strings, numbers, etc.) to your clients based on specific user attributes, such as country.

Remember to follow best practices for feature gates, such as managing for ease and maintainability, selecting the gating decision point, and focusing on one feature per gate.

Alternatively, you can target your feature by CustomerID. You could either use a Custom Field and then pass a custom field to the SDK {custom: {customer: xyz123} or create a new Unit Type of customerID and then target by Unit ID. For more information on creating a new Unit Type, refer to the Statsig documentation.

What is the status of the experiment if start is not clicked and what actions can I take?

When an experiment is in the "Unstarted" state, the code will revert to the 'default values' in the code. This refers to the parameter you pass to our get calls as documented here.

You have the option to enable an Experiment in lower environments such as staging or development, by toggling it on in those environments prior to starting it in Production. This allows you to test and adjust the experiment as needed before it goes live.

Remember, the status of the experiment is determined by whether the "Start" button has been clicked. If it hasn't, the experiment remains in the "Unstarted" state, allowing you to review and modify the experiment's configuration as needed.

Join the #1 Community for Product Experimentation

Connect with like-minded product leaders, data scientists, and engineers to share the latest in product experimentation.

Try Statsig Today

Get started for free. Add your whole team!

What builders love about us

OpenAI OpenAI
Brex Brex
Notion Notion
SoundCloud SoundCloud
Ancestry Ancestry
At OpenAI, we want to iterate as fast as possible. Statsig enables us to grow, scale, and learn efficiently. Integrating experimentation with product analytics and feature flagging has been crucial for quickly understanding and addressing our users' top priorities.
Dave Cummings
Engineering Manager, ChatGPT
Brex's mission is to help businesses move fast. Statsig is now helping our engineers move fast. It has been a game changer to automate the manual lift typical to running experiments and has helped product teams ship the right features to their users quickly.
Karandeep Anand
At Notion, we're continuously learning what our users value and want every team to run experiments to learn more. It’s also critical to maintain speed as a habit. Statsig's experimentation platform enables both this speed and learning for us.
Mengying Li
Data Science Manager
We evaluated Optimizely, LaunchDarkly, Split, and Eppo, but ultimately selected Statsig due to its comprehensive end-to-end integration. We wanted a complete solution rather than a partial one, including everything from the stats engine to data ingestion.
Don Browning
SVP, Data & Platform Engineering
We only had so many analysts. Statsig provided the necessary tools to remove the bottleneck. I know that we are able to impact our key business metrics in a positive way with Statsig. We are definitely heading in the right direction with Statsig.
Partha Sarathi
Director of Engineering
We use cookies to ensure you get the best experience on our website.
Privacy Policy