Frequently Asked Questions

When configuring alerts for conversion rate changes in experiments, the type of data the alerts are based on is determined by the alert window setting.

If the alert window is set to 'daily', the system will generate alerts based on the data specific to the day the alert is triggered. This approach is particularly useful for monitoring daily fluctuations in metrics.

It is important to note that if a metric tends to exhibit noise throughout the day but remains stable over a 24-hour period, setting a daily alert window may be more beneficial than an hourly one, as it focuses on the daily consistency rather than hourly variations.

In the context of Autotune experiments, it is important to note that adding a new variation to an existing experiment is not possible without resetting the experiment. If you need to introduce a new variation, the process involves pausing the current experiment, making the necessary changes, and then restarting the experiment.

However, it's crucial to understand that this process will reset the experiment, which means that any results obtained prior to the reset will be lost. This is a fundamental aspect of the Autotune experiment's design and operation.

Please ensure to consider this factor when planning to add new variations to your Autotune experiments.

When updating log events for a feature gate in Statsig, there is no need to restart the feature gate to see the revised metrics data, as changes should take effect immediately. However, if the updated metrics are not reflecting as expected, it is advisable to verify that the events are being logged correctly.

To obtain cleaner data for metrics lifts after modifying the code for a log event, you can adjust the distribution of users between the control and experiment groups or 'resalt' the feature to reshuffle users without changing the percentage distribution.

For experiments with delayed events, setting the experiment allocation to 0% after the desired period ensures that delayed events still count towards the analytics. It is important to note that the sizes of variants cannot be adjusted during an ongoing experiment to maintain the integrity of the results.

To increase the exposure of a variant, the current experiment must be stopped, and a new one with the desired percentage split should be started.

In the context of managing experiments with Terraform, the status field can be updated to reflect one of four possible values: setup, active, decision_made, and abandoned, aiding in the management of the experiment's lifecycle.

For those utilizing Statsig Warehouse Native, creating an Assignment Source and a new experiment allows for the definition of the experiment timeline and subsequent calculation of results.

Statsig pipelines typically run in PST, with data landing by 9am PST, although enterprise companies have the option to switch this to UTC.

Statsig Cloud calculates new or changed metrics going forward, and Statsig Warehouse Native offers the flexibility to create metrics after experiments have started or finished and to reanalyze data retrospectively.

In Statsig, you have the ability to both restart and extend an experiment.

To restart an experiment, you can utilize the 'Reset' option. This action will revert the experiment to an unstarted state, and all users will receive the default experience. The "salt" used to randomize a user's group will also be changed. This means that when you start the experiment again, your users will be randomly assigned to a group that is not necessarily the same group they were in prior to the experiment being reset.

When you reset an experiment, it also clears the metrics. However, you will still be able to see the old snapshot in the history, but the experiment itself will start afresh.

To extend an experiment, you can adjust the target duration in the advanced settings. Changing the target experiment length will not result in data loss or change the assignment to Control/Test. The target experiment length is primarily used for tracking progress against your experiment's duration. It does not impact the data collected or the group assignments within the experiment.

Please note that the sizes of variants cannot be adjusted while the experiment is in progress. If you wish to increase the exposure of a variant, you would need to stop the current experiment and start a new one with the desired percentage split.

Yes, it is possible to run feature gates within existing experiments. You can add new features to a variant by creating a feature gate that only passes for users who are in that variant. However, it's important to note that adding new features to a variant while an experiment is running could potentially impact the interpretation of the experiment results. Therefore, it's generally recommended to plan your experiment design carefully before starting the experiment to avoid making changes midway.

If you need to run multiple experiments or add new features, you might want to consider using layers. Layers are a powerful way to run many experiments in sequence, one building on top of the learning from the other. This could be a solution for running multiple experiments on the same users without having to change code.

Remember, once you start an experiment, it will run in all environments unless you set a targeting gate to restrict it to specific environments.

If you want to change the behavior of a variant, the recommended approach is to start a new experiment that is a vs. the new variant with the new feature being part of the new variant.

In the context of setting experiment group splits with decimal precision, the system is designed to limit the total number of buckets to 10,000, which allows for one digit of decimal precision.

When dealing with multiple variations, such as 32 in an experiment, it is recommended to assign 3.1% for 31 variants and 3.9% for the control group. This approach is considered acceptable for the experiment, and the slight difference in precision (e.g., 3.1% vs. 3.125%) is deemed negligible, especially in larger sample sizes where the difference might equate to a small number of users.

For experiments requiring more precise splits, alternating between 3.1% and 3.2% for every eighth variant is a suggested strategy. This allocation method is also applicable to more complex experimental designs, such as a 2-level fractional factorial experiment.

The key consideration is the statistical precision, which may be more critical depending on the user base size. It is important to note that the fixed number of buckets behind the scenes necessitates these allocation strategies.

In the event that an incorrect decision group is selected during an experiment, there are a couple of approaches to address the issue.

Firstly, if the parameters on the control and test groups can be swapped without any hardcoded behavior specific to the control or test, this action could potentially resolve the problem. However, it is important to note that this may lead to corrupted data.

Alternatively, the experiment can be reset, and a decision can be made immediately thereafter. It is important to understand that resetting the experiment and making a new decision will replace the top-level results with the new iteration of the experiment, which will not contain data.

However, data from previous iterations of the experiment can still be accessed through the 'View History' feature. This allows for the preservation of historical data while addressing the issue of the incorrect decision group selection.

To view raw events for an experiment group in Statsig, you can utilize the Events Explorer feature. This tool is designed to provide you with sample rows for events, which is particularly useful when you need to familiarize yourself with the data schema or wish to observe the exact data being logged.

The Events Explorer operates on a 14-day rolling window of sampled data, which allows for near real-time data availability. It is an excellent resource for identifying trends, making day-to-day or week-to-week comparisons, and gaining a high-level understanding of event occurrences.

However, it is important to recognize that the Events Explorer is not intended for retrieving all data associated with a specific user or for producing a precise count or export of a particular event. For tracking individual user events and exposures, the Users tab should be utilized instead.

Please note that the Events Explorer's sampling approach may not provide a complete dataset for exhaustive analysis but is suitable for general insights and trend monitoring.

In a subscription-based business model, understanding the long-term impact of experiments on Customer Lifetime Value (CLV) is crucial.

Statsig provides the capability to analyze the impact of an experiment on CLV over extended periods, such as 6 months. To facilitate this, Statsig allows for the setup of an experiment to run for a specific duration, such as 1 month, and then decrease the allocation to 0%, effectively stopping new user allocation while continuing to track the analytics for the users who were part of the experiment.

This tracking can continue for an additional 5 months or more, depending on the requirements. It is important to note that the experience delivered to users during the experiment will not continue after the allocation is set to 0%. However, there are strategies to address this, which can be discussed based on specific concerns or requirements.

Additionally, Statsig experiments by default expire after 90 days, but there is an option to extend the experiment duration multiple times for additional 30-day periods. Users will receive notifications or emails as the expiration date approaches, prompting them to extend the experiment if needed.

This functionality is available on the Pro plan, ensuring that businesses can effectively measure the long-term impact of their experiments on CLV without direct integration with a data warehouse and by updating CLV through integrations such as Hightouch.

Yes, it is possible to run two landing page type of experiments simultaneously with different layerIDs. Each experiment will have its own layerID, which allows you to run multiple experiments without needing to update the code on the website for each experiment.

The first experiment can run on the landing page, and the second experiment can run on the ThankYou page. Even though the flow from the landing page leads to the ThankYou page, the two experiments are independent because they have different layerIDs.

The code for setting up these experiments is as follows:

<script  src="https://cdn.jsdelivr.net/npm/statsig-landing-page-exp?apikey=[API_KEY]&layerid=[LAYER_NAME]"> </script>

Remember to replace [API_KEY] and [LAYER_NAME] with your actual API key and the layer name for each experiment. Also, update the script to specify the layerid instead of expid when you run experiments as part of a layer.

However, if you are trying to measure an impact after the thank you page, you probably want both of these experiments to be in the same layer.

Lastly, ensure that the two experiments are not in the same layer, because in that case, you will mutually exclude the same users from each other experiment.

Currently, the capability to backfill results for a metric that was added midway through an experiment is not available. This limitation is recognized, and plans are in place to address this issue in the future. However, there is no estimated time of arrival for this feature.

As of now, if a new metric is introduced, it will only be calculated from the date of its creation and cannot retrospectively apply to data collected before that point.

When you add a list in the override in Statsig, the system ensures that the users in the override list are assigned to the specified group. This action, however, does not affect the 50/50 split for the rest of the users who are not in the override list. The assignment of users is deterministic based on their ID, so your overrides won't affect other users.

Furthermore, it's important to note that overrides don't affect assignment for users outside of the overridden users. Overrides are typically meant for testing and by default, they are excluded from analysis. This means that even with the override assigned to a specific group, the test/control group split will still maintain a 50/50 ratio.

When using checkGateWithExposureLoggingDisabled at launch, it will not log an exposure event. This is particularly useful when you don't want to log exposures for users who may not reach the feature you're testing.

Later, when the user reaches the feature, calling checkGate will indeed log an exposure event. This is because checkGate by default logs an exposure event when it is called. Therefore, you do not need to call manuallyLogGateExposure in this case.

However, it's important to be aware that manually logging exposures can lead to an imbalance in exposure events if not done correctly. For example, only triggering exposures for users in the Test group of an experiment could imbalance the experiment.

Both checkGate and checkGateWithExposureLoggingDisabled will return the same result for a given user & gate. The thing to watch out for is if there are any changes between checkGateWithExposureLoggingDisabled and checkGate (e.g. gate is updated or user is updated).

When an experiment is reset in Statsig, it should not alter the overrides configured for it. The expected behavior is that resetting an experiment does not modify layer settings.

However, an issue was identified where resetting an experiment resulted in the overrides for that experiment in the layer being removed. This was not the intended functionality, and a fix has been implemented to ensure that resetting an experiment will no longer affect the layer overrides.

If any unexpected behavior is observed related to experiment resets or layer overrides, users are encouraged to report it for further investigation.

In the context of hypothesis testing, the p-value is the probability of observing an effect equal to or larger than the measured metric delta, assuming that the null hypothesis is true. A p-value lower than a pre-defined threshold is considered evidence of a true effect.

The calculation of the p-value depends on the number of degrees of freedom (ν). For most experiments, a two-sample z-test is appropriate. However, for smaller experiments with ν < 100, Welch's t-test is used. In both cases, the p-value is dependent on the metric mean and variance computed for the test and control groups.

The z-statistic of a two-sample z-test is calculated using the formula: Z = (Xt - Xc) / sqrt(var(Xt) + var(Xc)). The two-sided p-value is then obtained from the standard normal cumulative distribution function.

For smaller sample sizes, Welch's t-test is the preferred statistical test due to its lower false positive rates in cases of unequal sizes and variances. The t-statistic is computed in the same way as the two-sample z-test, and the degrees of freedom ν are computed using a specific formula.

While the normal distribution is often used in these calculations due to the central limit theorem, the specific distribution used can depend on the nature of the experiment and the data. For instance, in Bayesian experiments, the posterior probability distribution is calculated, which can involve different distributions depending on the prior beliefs and the likelihood.

It's important to note that it's typically assumed that the sample means are normally distributed. This is generally true for most metrics thanks to the central limit theorem, even if the distribution of the metric values themselves is not normal.

To manually test different experiment variants without the necessity of opening a new tab, you can utilize overrides to control which variant is exposed to a QA user or developer. Overrides can be implemented in two primary ways to facilitate manual testing within the same context.

Firstly, you can add the QA user ID to the overrides in the experiment setup. This method allows you to manage the experiment assignment behavior directly from the Console. Secondly, you can use a local override API to set specific overrides via code.

This approach enables you to change the behavior on the fly, similar to modifying a value in local storage. For instance, calling Statsig.overrideConfig in a browser's developer tools window can alter the experiment's behavior immediately.

These methods ensure that manual testing of different variants can be conducted efficiently and effectively within the same browser tab, providing a seamless experience for QA testers and developers alike.

To create your own StableID for server-side experiments, follow these steps:

1. Generate a unique identifier on your server. This could be a UUID or any other form of unique ID that you prefer.

2. Store this ID in a cookie or another method of your choice to be used on the client side as well.

3. Set this ID as the customID on the StatsigUser object when generating the initialize response from the SDK. Here's an example of how you can do this:

javascript const userObj = {      customIDs: {          stableID: '<generated in your code or passed via cookie or via db lookup>'      }   }

4. Get that ID from the cookie, and set it as the customID on the StatsigUser object when using the client SDK, so all event data and exposure checks tie back to the same user.

If you wish to use the stableID field rather than a custom ID, you still need to do step (2) above. Then:- Override the stableID in the client SDK by getting the value from the cookie and setting the overrideStableID parameter in StatsigOptions. - Set the stableID field on the StatsigUser object in the customIDs map when generating the initialize response from the SDK.

Please note that this information is based on the documentation's opinion.

Statsig differentiates data from different environments by the environment tier you specify during the SDK initialization. You can set the environment tier to "staging", "development", or "production". By default, all checks and event logs are considered "production" data if the environment tier is unset.

Experiments and metrics primarily factor in production data. Non-production events are visible in diagnostics, but they are not included in Pulse results. This is because most companies do not want non-production test data being included. If you want to include these, you can log them as regular events. However, non-production data is filtered out of the warehouse and there is no other way to include it.

When initializing, if you add { environment: { tier: "production" } }, this would set your environment to "production", not "staging" or "development". If you want to set your environment to "staging" or "development", you should replace "production" with the desired environment tier.

Pulse results are only computed for “Production” tier events. To see Pulse results, you need: • An experiment that is “Started” (aka, enabled in production) • Exposures & events in production tier • Exposures for users in all test groups • Events/metrics associated to users in test groups

As long as you have initialized using { environment: { tier: "production" } }, your Pulse will compute. This means that even if your code is deployed to staging, as long as you initialize with the production tier, you will be able to see Pulse results.

The current functionality of the "contains" operator allows for only one target in the condition. If you want to filter on multiple tags, you will need to add more events to your configuration. This can be done by using the "+ Add Another Event" link.

Here's how you can do it:

1. Use the "Filter" option in the Custom Query fields for a metric. 2. Add an individual filter for each tag you want to include in your analysis. 3. To add multiple values in one filter, use the "+ Add Another Event" link to add more events to your configuration.

Please note that the current documentation does not provide a method for adding multiple values in one filter in a more streamlined way. If you need further assistance, please reach out to the support team.

When an experiment decision has been made and there is a need to continue collecting metrics without resetting user assignments, the platform does not offer a native solution to directly address this scenario.

Once a decision is made on an experiment, the platform does not support undoing that decision. The 'Reset' option is available; however, it will revert the experiment to an unstarted state, and all users will receive the default experience. If the experiment is reset or a new experiment is started, users will be randomly assigned to a group, which may differ from their original assignment.

It is important to note that resetting an experiment will not preserve the initial user group assignments, which may be critical for long-term metric collection and analysis. To monitor the performance of long-term metrics after an experiment has ended, the Metrics Explorer can be utilized. While it does not provide confidence intervals and p-values, it does offer absolute numbers for each metric across all groups in the experiment.

For future experiments, it is recommended to enable Reviews in the project settings to provide better oversight and prevent similar situations. Reviews offer guardrails that can help in managing changes that impact production environments.

Ensuring consistent experiment results across Client-Side Rendered (CSR) and Server-Side Rendered (SSR) pages when using Statsig involves establishing a single source of truth for user assignments. The challenge arises when independent network requests on different pages result in varying initialization times, potentially leading to different experiment variants being presented to the user.

To address this, it is recommended to have one consistent source of truth for the user's assignment. This can be achieved by using the assignments from the first page (CSR) and carrying them over to the second page (SSR), rather than initializing a new request on the second page.

This approach ensures that the user experiences a consistent variant throughout their session, regardless of the page they navigate to. It is important to note that the user's decision for any getExperiment call is determined at the time of initialization, not when getExperiment is called. The getExperiment call simply returns the value fetched during initialization and logs the user's participation in the experiment and their assigned group.

To implement this, developers should ensure that the assignments are fetched from the network request on the first page and then passed to the second page, avoiding the need for another initialization request and the management of an ever-growing list of parameters.

In the event that an experiment in Statsig has been restarted and there is a need to export user data from the previous iteration of the experiment, the process is not as straightforward as using an 'Export Report' button.

However, it is still possible to retrieve the necessary data. To do so, the user should contact Statsig support directly with a request for the data export. The Statsig team can generate the report and send it to the email address associated with the user's Statsig account and project.

It is important to note that this process is manual and requires intervention from the Statsig team.

To retrieve currently active experiments using JavaScript in the browser console, you can use the Statsig.getExperiment or Statsig.getLayer methods. However, these methods require you to know the name of the experiment or layer. There is no documented method to retrieve all active experiments without knowing their names.

For group overrides, there is no documented method to retrieve group IDs directly. However, you can use the Statsig.overrideConfig method to override the value of a specific experiment for a specific user. The value you provide should match the structure of the experiment's parameters. Here is an example:

javascript Statsig.overrideConfig("a_config_or_experiment_name", { key: "value" }, "a_user_id"); In this case, { key: "value" } should match the structure of the parameters for the experiment you're trying to override. There is no documented way to override groups directly using group IDs.

If you're unsure about the structure of the experiment's parameters or need further assistance, it is recommended to reach out to the Statsig team for more specific guidance.

Please note that it may not be possible to override the group. You can only override the config object that's returned by a getExperiment call.

Incorporating an app version check into Statsig experiment variants can be achieved using Feature Gates or Segments. Here are the steps you need to follow:

1. Capture the Initial App Version: The first step is to capture and store the initial app version that a user starts using. This information is crucial as it will be used to determine whether a user is new and started using the app from a specific version onwards.

2. Use Custom Fields for Targeting: The next step is to use Custom Fields for targeting. This will require some code on the client-side that passes in user upgrade/create timestamps as a custom field.

3. Pass the Initial App Version as a Custom Field Key: You need to pass the value for users who started using the app as new users from a specific version as the Custom Field Key.

4. Configure the Custom Field Key: Once you create the key, configure it using the "Greater than or equal to version" operator. This operator checks the current version of the user's app.

For your specific case, you can create two separate experiments or feature gates. One for users on app version 1, where the variants are v1 and v2, and another for users on app version 2, where the variants are v1 to v4. You can then use the app version as a custom field for targeting.

Please note that the Key set in the Custom Field will be included in the events called by Feature Gates. However, it's not a chargeable event. It's just attributes that will be in the payload for events you're already tracking.

When optimizing landing page loading without waiting for experiment configurations, it is recommended to use a custom script approach if you need to pass in specific user identifiers, such as Segment's anonymous ID.

This is because the standard landing page script provided by Statsig does not allow for the initialization with a different user ID. Instead, it automatically generates a stableID for the user, which is used for traffic splitting and consistent user experience upon revisits. This stableID is stored in a cookie and is used by Statsig to identify users in experiments.

However, if you need to synchronize with metrics from Segment using the Segment anonymous ID, you may need to deconstruct the landing page tool and implement a custom version that allows you to set the userObject with your Segment anonymous ID.

Additionally, you can enrich the Segment User with an additional identifier, such as statsigStableID, which can be obtained in JavaScript using statsig.getStableID(). This ID can then be mapped from the Segment event payload to Statsig's stableID. If performance is a concern, you can bootstrap your SDK with values so you don't have to wait for the network request to complete before rendering.

This can help mitigate performance issues related to waiting for experiment configurations. For more information on bootstrapping the SDK, you can refer to the official documentation on bootstrapping.

When running experiments with both anonymous and logged-in users using Segment and Statsig, it is crucial to have a common identifier to track user interactions accurately.

For anonymous users, you can either pass the stable ID generated by Statsig to Segment or generate a stable ID and pass it to Statsig. This ensures that there is a consistent identifier across both platforms.

For logged-in users, you can use the Segment userID, which should be included in the userObject during initialization. This setup allows Statsig to recognize both anonymous and logged-in users in the experiment. To facilitate this, you may need to provision a new custom ID, such as segmentAnonymousId, and map Segment's anonymousId to this new custom ID in the integration settings.

This approach ensures that exposure events sent via the Statsig SDK and custom events sent via Segment are associated with the same user, regardless of their logged-in status.

To start a new version of an existing experiment in Statsig, follow the steps below:

1. Log into the Statsig console.

2. Once logged in, navigate to the Experiments section. You can find this in the left-hand navigation panel.

3. Click on the Create button to start a new experiment.

4. Enter the name and description for your new experiment. Remember, the name of the experiment should be unique. If you're creating a new version of an existing experiment, consider including a version number or date in the name to distinguish it from the original.

5. Configure your experiment's allocation, targeting criteria, and groups and parameters. This step is crucial to ensure your experiment runs as expected.

6. Specify your hypothesis and add the Primary and Secondary metrics for your experiment. These metrics will help you measure the success of your experiment.

7. After all the details are filled in, press Save and Analyze to start the experiment. You'll be prompted to finalize the dates, experiment status, and optionally set up a schedule to reload experiment results.

By following these steps, you can easily start a new version of an existing experiment in Statsig.

To test whether a metric based on a custom event is tracked in non-production environments, you can follow a specific process. While non-production events are visible in diagnostics, they are not included in Pulse results.

To include these events, you can log them as regular events. It is important to note that non-production data is intentionally not included in metrics to prevent non-production test data from being included in the metrics. When testing your event logging, you can suffix your events with '_test' and use the production environment. Once you are ready to finalize your event logging, you can remove these suffixes.

If you encounter issues where the Metrics Logstream is not visible, it may indicate a connection or schema issue, and Statsig is not receiving your precomputed metrics. In such cases, you should reach out for assistance, and a team member will provide support.

To view statistics for multiple landing pages separately in Statsig when analyzing a test, you need to set up separate experiments for each landing page. This involves adding a unique script to each landing page you're testing. The script should look like this:

<script src="https://cdn.jsdelivr.net/npm/statsig-landing-page-exp?apikey=[API_KEY]&expid=[EXP_NAME]"></script>

Replace [API_KEY] with your Client API Key and [EXP_NAME] with the unique Experiment Name for each landing page.

Once you've set up and published your landing pages, you can monitor the setup by navigating to the Diagnostics Tab in your experiment and looking at the 'Exposure Stream' at the bottom of the page. This shows a real-time stream of the page loads along with the variant they were allocated.To interpret the results, go to the Pulse Results tab in the Experiment page. Here, you can add metrics you want to monitor and verify which variant is doing better.

Remember, you can extend this setup to run ABC or ABn tests by adding more variants in the Experiment Setup tab. If you want to run multiple landing page experiments without needing to update the code on the website for each experiment, consider using layers.

Additionally, you can find group level stats in the hover card. Just hover over the metric and click "view details".

Yes, it is possible to view the scorecard metrics of an experiment for a custom date range that starts after the experiment start date. This can be achieved by creating an Explore report with a specific date range. The report will filter the events and metrics to be within that range. However, the users included in the report will also encompass those exposed before the specified timeframe. This is designed to capture the downstream effects of the experiment treatment.

Here are the steps to do this:

1. Navigate to the Explore tab.

2. Fire off a custom query with your desired date range.

If you're having trouble selecting a date range that starts after the experiment start date, it might be a technical issue. Check your inputs and try again. If the issue persists, please flag it for assistance.

In Statsig, the "Days since exposure" graph in the View Details section can be utilized to understand if an experiment is pulling forward an event. This graph displays the cumulative lift of a metric over time since a user's first exposure to an experiment. If you observe a significant lift immediately after exposure that diminishes over time, this could suggest that the experiment is pulling forward events.

However, it's important to note that there isn't a built-in tool specifically designed to identify pulled forward events. The closest feature available is a toggle in the top right of the scorecard that allows you to select “Days Since Exposure” instead of “Cumulative”. This feature enables you to look across all metrics inline.

Remember, the interpretation of the "Days since exposure" graph and the use of the toggle feature require a good understanding of your metrics and the potential impact of your experiments.

When conducting an experiment with Statsig, you may encounter a warning stating 'Event metrics do not have data.' This warning is designed to ensure that there are participants in your gate or experiment who have also had some event logged for them. It serves as a validation that the users in the experiment have corresponding metrics that exist for them.

This warning can be triggered during the initial stages of an experiment, especially when all components of Statsig, such as the SDK and experiment configurations, are deployed simultaneously. It is important to note that there is a waiting period after the gate gets modified, and it may take some time for the system to recognize that users have been exposed to the experiment and have logged events.

The warning is expected to resolve once the system detects overlap between users in the experiment and the logged events, which can take up to an hour after the events have been logged.

For most metrics, computation occurs on a daily basis, and results will be available the following day. However, for 'event_count' type metrics, leading indicators can be computed and may be visible in about an hour.

If you encounter this warning, it is advisable to wait for the system to update and check if the warning persists before taking further action.

When utilizing a client SDK that involves experiments with targeting gates based on dynamic user attributes, there is a mechanism to handle changes in user eligibility. If a user's attributes alter in such a way that they no longer meet the experiment's targeting criteria, the client SDK can be configured to store the initial evaluation results in local storage.

This ensures that the user's experience remains consistent, and they continue to have access to the feature or release associated with the experiment, despite any subsequent changes in their dynamic attributes.

This approach is beneficial for maintaining a stable user experience and ensuring that the integrity of the experiment is not compromised by fluctuating user states.

In the context of Pulse results, the cutoff time for data pertaining to the previous day is established at 11:59 PM Pacific Standard Time (PST).

This timeframe ensures that all data collected falls within the standard daily cycle, which begins at 12:00 AM PST and concludes at 11:59 PM PST of the same day. It is important for users to align their data analysis and reporting activities with this cutoff to accurately reflect the results for each individual day.

In the context of Bayesian testing, the credible interval is the Bayesian equivalent of the confidence interval. Specifically, it is reported for the conversation delta for your conversion rate.

As for the interpretation of expected loss, it is computed as the expected value of delta when delta is negative, multiplied by the probability of delta being negative. This can be loosely interpreted as the risk to your metric if you launch the test group.

The current Statsig SDK does not provide a method for bulk querying experiments for a large number of users in a single call. The getExperiment function is designed to be called for individual users. If you need to get data about all experiments for a specific user ID, you can use the getExperiment function with different user IDs as many times as you want.

Please note that calling getExperiment will trigger an exposure for that user on that day. If they weren’t in a test or control group, they would be exposed and count towards experiment metrics from that day forward.

If you have a use case where you need to know the evaluation but do not want to enroll the user in an experiment group, you can use the getExperimentWithExposureLoggingDisabled API.

To determine if a user has participated in an experiment, you need to download the exposure report for each experiment to see which users have checked it. The daily reports API is an enterprise feature that can be enabled for your project.

If you're using an SDK to evaluate experiments, it should be pretty simple to just iterate through and check for each user. This is the recommended route for handling large numbers of users.

There are two main methods for performing A/B tests for static pages in Next.js using Statsig.

The first method involves using getClientInitializeResponse and storing the initializeValues in a cookie. This approach is suitable if you want to avoid generating separate static pages for each variant. However, the cookie size is limited to 4KB, so this method might not be suitable if the initializeValues are large.

The second method involves generating a separate static page for each experiment's variant. This approach is suitable if you have a small number of variants and want to avoid the cookie size limitation. However, this method might require more setup and maintenance if you have a large number of variants.

If you're unsure which method to use, you can start with the method that seems easier to implement and switch to the other method if you encounter issues.

If you're concerned about the size of initializeValues, there are a couple of ways to bring down the response size. One way is to use target apps to limit the gates/experiments/etc included in the response. Another way is to add an option to getClientInitializeResponse to specify which gates/experiments/etc to include in the response.

If you plan on stitching together multiple cookies, a simple string splice might be easier. An alternative that doesn't involve stitching together multiple initializeValues is to use multiple client SDK instances. This wouldn't be supported in react, but using the js SDK, you could have multiple statsig instances each with its own set of configs. You would have to keep track of which instance to use for which experiment but this may be a "cleaner" approach.

The JS SDK can be synchronously loaded using initializeValues similarly to how the StatsigSynchronousProvider works. So you should be able to just call statsig.initialize(..., {initializeValues}) without needing to worry about awaiting.

Finally, you can also use the local evaluation SDK to fetch the whole config before the page becomes interactive and then pass it to the synchronous SDK. This is a client SDK, but it solves the "flickering" issues because you don't need to wait for the experiment(s) data to be fetched on the fly.

The daily process that updates experiment metrics for the previous day typically completes by 9am PST. This process ensures that all data for newly added metadata dimensions in your experiment metrics are calculated and updated. Please note that the exact timing may vary, and for more specific information, you may need to consult with a Statsig team member.

If you're having trouble locating the entry point for holdouts in your dashboard, please note that it has been moved to a tab within experiments. This change might require some adjustment, but rest assured, all the functionalities remain the same. If you're unable to locate it within the experiments tab, please reach out for further assistance.

If you're not seeing data in the "Exposure Stream" tab, there could be a few reasons for this. Here are some steps you can take to troubleshoot:

1. Check Data Flow: Ensure that the id_type is set correctly and that your ids match the format of ids logged from SDKs. You can check this on the Metrics page of your project.

2. Query History: If your data is still not showing up in the console, check your query history for the user to understand which data is being pulled, and if queries are not executing or are failing.

3. Data Processing Time: If data is loading, it's likely we're just processing. For new metrics, give it a day to catch. If data isn't loaded after a day or two, please check in with us. The most common reason for metrics catalog failures is due to id_type mismatches.

4. Initialization Status: Each hook has an isLoading check, which you can use, or the StatsigProvider provides the StatsigContext which has initialization status as a field as well. This can be used to prevent a gate check unless you know the account has already been passed in and reinitialized.

If you've checked all these and are still experiencing issues, please let us know. We're here to help!

In some cases, the issue might be due to the way you are accessing the layer config and getting the value of the config. When you manually reach into the layer config and get the value of the config, we won’t know when to generate a layer exposure. For example, if you have a line of code like this:

const config = statsig?.getLayer('landing_page_gg')?.value;

While this will give you the value of the config, it won’t generate exposures and your experiment won’t work. In order to generate the correct exposures, you will need to get the Layer object and ask for the values using the “get” method. Like this:

const layer = statsig?.getLayer('landing_page_gg'); const block_1 = layer.get('block_1', '');

And similarly for other parameters you have defined in your layer.

If you are pulling multiple values at once and they control the same experience, it is recommended to pack them into a single parameter. However, if the parameters affect experiences in multiple locations, keep them separate. If you are pulling everything in one shot, treat them as a single object value. Use the json type. You could also just add a new parameter to the current experiment that has all the other values. There's no need to create a new experiment if you don’t want to.

In the event that some metrics in your ongoing experiment are showing as "unavailable", it could be due to an issue with how your events are being logged or matched with the experiment groups. To resolve this, ensure that the events you're sending include the same identifier (either userID or stableID) as the one used in your client SDKs for the experiment. This will help Statsig join the events with the correct experiment groups.

It's important to note that exposures on the last day (the day you made a decision) are not counted as exposures. Therefore, when analyzing your data, remember to filter out that day.

If you're still experiencing issues, it might be helpful to check your query history for the user to understand which data is being pulled, and if queries are not executing or are failing.

In some cases, metrics may temporarily show as unavailable due to Statsig recomputing the metric for the latest numbers. However, these metrics should become available once the final aggregated data is uploaded and any issues with this process are resolved.

If the issue persists, please let us know and we'll have a Statsig team member assist you further.

If you're unable to adjust the dates in the scorecard for your old experiment, it could be due to the experiment's end date being set in the past. When an experiment is created, it is marked with a specific completion date. If you want to extend the experiment or adjust the dates, you have two options:

1. Clone the existing experiment and restart it, making sure to set the experiment end date to the desired date or mark it as ongoing.

2. Reset the experiment from the same menu and restart it from there.

After resetting the experiment, you can change the end date. However, you might notice that the pulse loading takes a longer time than usual. This is not normal even for this experiment. The Wall time, which is the total "real time" it took to run the job, from when it started to when it finished, might be longer than the job time, which is the actual compute time run on your warehouse. This could be due to transient network errors communicating with your warehouse, which is unusual. If you encounter this issue, it's recommended to flag it for assistance.

In the Explore tab of Pulse Results, you can use user dimensions to drill down and explore results. However, the user properties you can segment on are only derived from exposure events, not custom events. Any overrides that are used to force a user into the test will be ignored from pulse results. If you only have override userIds in the results, nothing will show up in the pulse filters. If you're having trouble finding a specific user property, it might be due to these reasons.

It's also important to note that the availability of User Dimensions can vary depending on the gate/experiment you’re querying on. Some entities have User Dimensions available, some don’t based on whether there are exposures or not.

Regarding the issue of error ranges for Daily Topline Impact and Daily Projected Launch Impact not populating, this was due to a bug that occurred during our internal migration of infrastructure and data. We have fixed this bug and it affected certain metric types. All new custom queries should not have a problem. For some metrics, the topline impact could have a 0 as the CI, because all users can have the value of 1 if the metric is an indicator. However, we'll do an audit to make sure the calculation is consistent with our scorecard.

To check if a specific user ID or stable ID has been manually added to any experiments using "Manage Overrides", follow these steps:

1. Log into the Statsig console

2. On the left-hand navigation panel, select "Users".

3. In the User's tab, enter the specific user ID or stable ID you're looking for in the "ID" field and select the appropriate "ID Type" from the dropdown menu.

4. Click "Submit" to load the data for the entered ID.

The resulting information will include a section titled "Overrides", which lists all the entities (Feature Gates, Experiments, etc.) that the given ID has been manually overridden into. Please note that you need to enter both the ID and ID type to submit a query.

If a stable ID is not showing in search results despite being present in many experiments, there are a couple of things to check:

1. Ensure that the box “include in Pulse results” was checked in the override modal.

2. Consider the timing of the assignments/exposures. It may take some time for the system to update and reflect recent changes.

If you're only seeing overridden values in the Exposure Stream for your experiment, there could be several reasons for this. Here are some steps you can take to troubleshoot:

1. Check the Initialization Status: Each hook has an isLoading check, which you can use, or the StatsigProvider provides the StatsigContext which has initialization status as a field as well. This can be used to prevent a gate check unless you know the account has already been passed in and reinitialized.

2. Check the Data Flow: Ensure that the id_type is set correctly and that your ids match the format of ids logged from SDKs. You can check this on the Metrics page of your project.

3. Check the Query History: If your data is still not showing up in the console, check your query history for the user to understand which data is being pulled, and if queries are not executing or are failing.

4. Check the Exposure Counts: If you're seeing lower than expected exposure counts, it could be due to initializing with a StatsigUser object that does not have the userID set, and then providing that on a subsequent render. This causes the SDK to refetch values, but logs an exposure for the “empty” userID first. To prevent this, ensure the userID is set before initializing the StatsigUser object.

If you've checked all these and the issue persists, it might be best to reach out for further assistance.In some cases, users may still qualify for the overrides based on the attributes you’re sending on the user object. This may be due to caching, or it may be due to the user qualifying for other segments that control overrides.

For example, if users have "first_utm_campaign": "34-kaiser-db" being sent on the user object, they would qualify for a segment that’s being used in the overrides.

It's also important to note that overridden users will see the assigned variant but will be excluded from experiment results. We have a way to include users in results for experiments not in layers, but it seems we don’t have that option for experiments in layers.

Lastly, consider why you are using overrides in this scenario instead of a targeting gate. Overrides can be used to test the Test variant in a staging environment before starting the experiment on prod. However, if some of your customers have opted out of being experimented on, a targeting gate might be a more suitable option.

If you're running an Autotune experiment and notice that the Success Event is not being counted on the Pulse Results, it could be due to a few reasons.

Firstly, ensure that the events you're sending include the same identifier (either userID or stableID) as the one used in your client SDKs for the experiment. This is crucial for Statsig to join the events with the correct experiment groups.

If you're still not seeing the expected results, you might want to check your query history for the user to understand which data is being pulled, and if queries are not executing or are failing. If data is loading, it's likely we're just processing. For new metrics, give it a day to catch. If data isn't loaded after a day or two, please check in with us. The most common reason for metrics catalog failures is due to id_type mismatches.

In the case of using a Segment event for the success metric, it's important to note that there may be a delay in the Autotune results being updated. Data should be updated hourly but could be a bit later depending on your attribution window.

If you continue to not see the data as expected after more time has passed, please reach out for further assistance.

The discrepancy between the experiment allocation counts and the ssr_search_results_page_view (dau) counts could be due to several reasons:

1. **User Activity**: Not all users who are allocated to an experiment will trigger the ssr_search_results_page_view event. Some users might not reach the page that triggers this event, leading to a lower count for the event compared to the allocation.

2. **Event Logging**: There might be issues with the event logging. Ensure that the statsig.logEvent() function is being called correctly and that there are no errors preventing the event from being logged.

3. **Timing of Allocation and Event Logging**: If the event is logged before the user is allocated to the experiment, the event might not be associated with the experiment. Ensure that the allocation happens before the event logging.

4. **Multiple Page Views**: If a user visits the page multiple times in a day, they will be counted once in the allocation but multiple times in the ssr_search_results_page_view event.

If you've checked these potential issues and the discrepancy still exists, it might be a good idea to reach out to the Statsig team for further assistance.

Another possible reason for the discrepancy could be latency. If there is a significant delay between the experiment allocation and the event logging, users might abandon the page before the event is logged. This could lead to a lower count for the ssr_search_results_page_view event compared to the allocation.

When conducting experiments, it is crucial to ensure that there is no cross-contamination between control and treatment groups and that data is accurately reflected in both funnel and summary views.

Cross-contamination can occur due to implementation issues, such as a race condition with tracking. This happens when users, particularly those with slower network connections, land on a control page and a page-view event is tracked before the redirect occurs.

To mitigate this, it is recommended to adjust the placement of tracking scripts. The Statsig redirect script should be positioned high in the head of the page, ensuring that it executes as early as possible. Meanwhile, page tracking calls should be made later in the page load lifecycle to reduce the likelihood of premature event tracking. This adjustment is expected to decrease discrepancies in tracking and improve the accuracy of experiment results.

Additionally, it is important to confirm that there are no other entry points to the control URL that could inadvertently affect the experiment's integrity. Ensuring that the experiment originates from the correct page and that redirects are functioning as intended is essential for maintaining the validity of the test.

Lastly, it is necessary to have specific calls in the code to track page views accurately. These measures will help ensure that the experiment data is reliable and that the funnel and summary views are consistent.

In order to get the useExperiment hook to work correctly in your code, you need to ensure that the StatsigProvider wraps your entire app or at least the root component that uses Statsig. This is because the StatsigProvider initializes the SDK and passes down its state to the rest of the React application via a StatsigContext.Here's how you can modify your code:

jsx // Initialize Statsig at a higher level, ideally in your app's root component <StatsigProvider  sdkKey=CLIENT_SDK_KEY  waitForInitialization={true}  user={{userID: userData?.id, email: userData?.email}}  options={{    environment: { tier: "development" },  }} >  <App /> {/* Your app's root component */} </StatsigProvider> // Then in your ButtonModal component export function ButtonModal({ ...props }: ButtonModalProps) {  const classes = useStyles();  const [ isButtonModalOpen, setIsButtonModalOpen ] = useState(false);  const { config } = useExperiment("button_test");  // Rest of your code... } By doing this, the StatsigProvider will initialize the SDK once and provide the necessary context for all child components. This should allow the useExperiment hook to work correctly. Remember, you cannot call useExperiment in the same component that is rendering the provider. The provider needs to wrap a component above this in the hierarchy.

In the context of experiments, when a user is attributed to the 'uninitialized' state, it indicates that the user has not been properly initialized within the experiment framework. As a result, the user will not be assigned to a real experimental group but will instead fall into the 'Default' group. It is important to note that the 'Default' group is not included in the analysis of experiment metrics.

Therefore, data from users in the 'uninitialized' state will not be included in the experiment metrics. This ensures the integrity of the experiment results by only including data from users who have been correctly initialized and assigned to a valid experimental group.

Yes, the experiment buckets will be preserved for the Stable ID experiments when the user transitions from the Server Side Rendered (SSR) page to the Client Side Rendered (CSR) page or vice versa. This is because the Stable ID is a unique identifier that persists on the device and is used to ensure that your users have the same experience or group across different states, including logged out to logged in states.

Even though the user object (userId) changes between these pages, as long as the Stable ID remains the same, the user's experiment group will remain consistent. This is because the Stable ID is used for bucketing in the experiment, not the userId.

However, it's important to ensure that the Stable ID is correctly implemented and persisted across different pages and sessions. If you're facing issues with Stable ID changing across subdomains, one possible solution is to store the Stable ID in a cookie and reuse it. You can store and retrieve your own id and use the overrideStableID method on the SDK to send it to Statsig. Your overridden id will always take preference.

Remember to always follow the best practices as outlined in the documentation when implementing experiments with Statsig.

When a user qualifies for an experiment based on certain user properties and is assigned a variant, their allocation to that variant is deterministic and will remain consistent throughout the experiment, even if their qualifying properties change and they temporarily no longer qualify for the experiment.

If a user's properties change such that they no longer meet the experiment's qualifying criteria, they will receive the default parameter values until their properties revert to a state that re-qualifies them for the experiment. Upon re-qualification, the user will be reassigned to the same variant as initially allocated.

This deterministic allocation ensures that a user's experience remains consistent across multiple sessions and that the integrity of the experiment is maintained. The allocation is based on a stable identifier, such as a userID or stableID, which allows for consistent group assignment upon re-qualification.