Frequently Asked Questions

In the context of dynamic configurations, the concept of extending configs based on user tiers, while maintaining a base configuration, is a common query. The approach to achieve this functionality is not through dynamic addition of configuration segments, but rather through the use of targeted rules for each tier.

To implement tier-based dynamic configurations, it is recommended to create a complete version of the dynamic config for each tier. This means that instead of having a base config with incremental additions, each tier should have its own full set of configurations. User targeting is then employed to determine which configuration should be applied to a user based on their tier level.

This method ensures that the correct and complete set of configurations is delivered to the user, adhering to the tier they belong to. It is important to note that this process does not support the dynamic addition of configuration segments; instead, it relies on the pre-definition of full configurations for each tier.

Feature gates are a powerful tool for controlling the availability and behavior of features within an application. However, their scope of influence is limited to the application's internal features and does not extend to system-level settings or functionalities.

Specifically, feature gates cannot be used to hide custom keyboard extensions in system settings, as this would require direct interaction with the system's configuration, which is outside the purview of feature gates.

To achieve such functionality, developers would need to write code that interacts with the system settings directly.

Once such code is in place, feature gates can then be employed to control the execution of this code, effectively enabling or disabling the hiding of custom keyboard extensions based on the feature gate's configuration.

In the context of Autotune experiments, it is important to note that adding a new variation to an existing experiment is not possible without resetting the experiment. If you need to introduce a new variation, the process involves pausing the current experiment, making the necessary changes, and then restarting the experiment.

However, it's crucial to understand that this process will reset the experiment, which means that any results obtained prior to the reset will be lost. This is a fundamental aspect of the Autotune experiment's design and operation.

Please ensure to consider this factor when planning to add new variations to your Autotune experiments.

Yes, it is possible to extend the target duration of your A/B experiment after it has been set up. Changing the target experiment length in the advanced settings will not result in data loss or change the assignment to Control/Test. The target experiment length is primarily used for tracking progress against your experiment's duration. It does not impact the data collected or the group assignments within the experiment.

By default, Statsig will compute Pulse Results (i.e., metric lifts) for your experiment for the first 90 days. If you need to extend beyond this, you will be able to do so as you approach the 90 days cap. Please note that extending the duration of your experiment does not affect the data collected or the group assignments within the experiment.

Yes, it is possible to run two landing page type of experiments simultaneously with different layerIDs. Each experiment will have its own layerID, which allows you to run multiple experiments without needing to update the code on the website for each experiment.

The first experiment can run on the landing page, and the second experiment can run on the ThankYou page. Even though the flow from the landing page leads to the ThankYou page, the two experiments are independent because they have different layerIDs.

The code for setting up these experiments is as follows:

<script  src="https://cdn.jsdelivr.net/npm/statsig-landing-page-exp?apikey=[API_KEY]&layerid=[LAYER_NAME]"> </script>

Remember to replace [API_KEY] and [LAYER_NAME] with your actual API key and the layer name for each experiment. Also, update the script to specify the layerid instead of expid when you run experiments as part of a layer.

However, if you are trying to measure an impact after the thank you page, you probably want both of these experiments to be in the same layer.

Lastly, ensure that the two experiments are not in the same layer, because in that case, you will mutually exclude the same users from each other experiment.

When integrating metrics ingestion with Statsig, it is not necessary for developers to apply their own date filters within their SQL queries. Statsig is designed to manage date filtering autonomously during the data backfill process.

Upon setting up data ingestion, developers provide a SQL query that generates a view for Statsig to ingest. This query is then utilized by Statsig to incrementally load data on a daily basis. For the purpose of backfilling, Statsig will inspect the data for the past three days from the current date to determine if there has been a significant change, defined as a variation exceeding 5% in the number of rows.

If such a change is detected, Statsig will automatically initiate a backfill. For data outside of this three-day window, it is expected that the developer will manually trigger a backfill for the desired date range. Therefore, the SQL query provided for data ingestion should not be constrained by row limits, as Statsig will handle the necessary date filtering to ensure efficient and accurate data ingestion.

When a project member, who is part of a single review group, creates a new Feature Gate within a project that enforces reviews, the review group will not be automatically added to the ALLOWED REVIEWERS for the new Feature Gate.

The creation of a new Feature Gate does not require a review by the review group. However, any updates to the configuration of the Feature Gate will automatically include the review group as part of the review process.

In order to change the directionality of a custom metric in Statsig, you would typically navigate to the Metrics Catalog tab, select the metric you want to edit, and then click on the "Edit" button. However, there was a known issue where the edit icon was hidden until you mouse over it due to a design change. This issue has been resolved and you should now be able to see the edit icon without having to mouse over it.

If you still encounter any issues with changing the directionality of a custom metric, please refresh your page as the fix should be applied. If the problem persists, please contact the Statsig support team for further assistance.

When utilizing the HTTP API to log events in real-time, it is crucial to ensure that the events are properly formatted to be recognized by the system. If events are not appearing in the Metrics Logstream, it may be necessary to wrap the events in an array using the events:[] syntax. This is particularly important because events are typically sent in bulk, and the correct formatting is essential for them to be processed and displayed in the stream.

Additionally, it is recommended to use the SDKs provided for various languages, as they handle a lot of the heavy lifting, including performance and reliability improvements. For cases where the SDKs are not an option, such as with certain desktop applications or specific frameworks like Qt, it may be possible to use language bindings to integrate with the SDKs of supported languages. This can potentially save significant development effort and ensure more reliable event logging.

Furthermore, for the best practice regarding experiments and configurations, it is advised to fetch the experiment config at the time when it is needed to ensure accurate exposure logging. The SDKs typically fetch all evaluations for a single user in one shot and cache it for the session duration, allowing for local and instantaneous experiment config or gate checks. If using the HTTP API directly, it is possible to use an 'initialize' endpoint, which is typically reserved for SDK use.

This endpoint fetches all evaluated results for a given user without generating exposure logs at the same time, thus avoiding overexposure issues.

When you want to exclude users from a holdout group using a targeting gate, the users should pass the targeting gate. The targeting gate is used to define the users who are included in the holdout group. Therefore, if a user passes the targeting gate, they are included in the holdout group.

If you want to exclude certain users, you would set up your targeting gate such that these users fail the gate. This will effectively exclude them from the holdout group.

In other words, you should create a gate that qualifies users to be in the holdout. If the gate evaluates to true for a user, they will be held out or excluded.

Remember, the key is to design your targeting gate in a way that the users you want to exclude will fail the gate, thus ensuring their exclusion from the holdout group.

In order to implement StatSig tracking on Shopify Custom Pixels, you need to ensure that the StatSig SDK is properly initialized and that events are logged correctly. Here's a step-by-step guide on how to do this:

1. First, create a script element and set its source to the StatSig SDK. Append this script to the document head.

javascript const statsigScript = document.createElement('script'); statsigScript.setAttribute('src', 'https://cdn.jsdelivr.net/npm/statsig-js/build/statsig-prod-web-sdk.min.js'); document.head.appendChild(statsigScript);

2. Next, initialize the StatSig SDK. This should be done within an asynchronous function that is called when the StatSig script has loaded.

javascript const statsigInit = new Promise(async (resolve) => {    statsigScript.onload = async () => {      await statsig.initialize("client-sdk-key", { userID: "user-id" });      resolve();    } })

3. Subscribe to the event you want to track and log it with StatSig. Make sure to wait for the StatSig SDK to initialize before logging the event.

javascript analytics.subscribe("checkout_started", async () => {    await statsigInit;    statsig.logEvent("checkout_started"); });

Remember that statsig will be the global, you want to wait for initialization before logging, no need to require.Please note that the SDK accumulates events within, and flushes periodically. Ensure that Statsig.logEvent() was ever called, and that the application didn’t exit right away after that so the SDK has enough time to flush the events.

Inspect the network requests and see what requests were made and what their responses were to verify if the events are being sent correctly.

To roll out and monitor multiple new features simultaneously using Statsig, you can utilize the platform's Feature Gates and Experiments.

For each new feature you plan to roll out, create a corresponding Feature Gate. This approach automatically converts a feature roll-out into an A/B test, allowing you to measure the impact of the roll-out on all your product and business metrics as the roll out proceeds.

If you wish to test hypotheses between product variants, create an Experiment. An Experiment can offer multiple variants and returns a JSON config to help you configure your app experience based on the group that the user is assigned to.

To show features randomly to different sets of customers, use the 'Pass%' in a Feature Gate and 'Allocation%' in an Experiment. This allows you to control the percentage of users who see each feature.

Statsig's Experiments offer more capabilities for advanced experiment designs. For example, you can analyze variants using stable IDs for situations when users have not yet signed-up (or signed-in), or using custom IDs to analyze user groups, pages, sessions, workspaces, cities, and so on. You can also run multiple isolated experiments concurrently.

Remember to define your company goals and key performance indicators (KPIs) to measure the success of your features. You can break down these strategic goals into actionable metrics that you can move with incremental, iterative improvements.

If you use three different feature gates, you will find out how each feature, individually, performed against the baseline. If you want combinatorial impact analysis, like, A vs B vs C vs AB vs BC vs AC vs ABC, then you will need to setup an experiment with 7 variants and specify the combinations via parameters and measure.

However, in practice, this level of combinatorial testing isn’t always fruitful and will consume a lot of time. A pragmatic recommendation would be to use feature gates to individually launch and measure the impact of a single feature, launch the ones that improve metrics and wind down the ones that don’t.

Currently, it is possible to add custom criteria for feature gating. You can create user targeting rules based on user attributes such as email and user ID, and additional ID types can be added in project settings. These can be used for all new experiments, allowing control over the experience a test user sees.

However, there isn't a feature to set these custom criteria as default options for a project. The ability to add these as default criteria options with a dropdown menu is not mentioned in the current documentation.

That being said, we are working on Templating features, and any default configuration criteria, including custom field defaults, are being considered. This includes the possibility of having default custom config for metrics so when the feature gate is created, the custom default metrics are selected/not selected.

In terms of the level at which these metrics are custom to, we are considering project-level defaults. This would be particularly useful for organizations with multiple teams utilizing different repositories, allowing for defaults to be set based on the specific project.

Please note that these features are currently in the scoping phase, and we are actively seeking input on feature requirements.

When utilizing the checkGate function within Statsig, it is imperative to provide a userID. This identifier is crucial as it ensures a consistent experience for each user. The userID is a mandatory field, as it is utilized to maintain uniformity in user experiences and is essential for the randomization process during partial rollouts.

It is important to note that even if an email is provided as a top-level field on StatsigUser, it does not substitute the requirement for a userID. In instances where there is no logged-in user, a stable identifier should be established, which could be stored as a cookie or in local storage, and used consistently with each call to Statsig.

This practice is vital for the system to function correctly and provide reliable results.

When transitioning from event import to metric import, it is essential to understand the ingestion status and the steps required to archive metrics that are no longer needed. After switching to metric import, if you have precomputed metrics, ingesting these metrics directly can be more efficient than importing raw events.

Once the change is made, there may be a need for adjustments on the backend to accommodate the new import method. This initial setup is a one-time speedbump, and once resolved, metrics should start appearing as expected. If you notice that some metrics have complete data for a specific date range while others do not, it may be due to the ingestion process still being underway or a need for a backfill to import historical data.

For metrics generated from the previous event ingestion job, such as event_count and event_dau types, consider archiving them if they are no longer relevant to your current data structure and needs. Archiving these metrics can help maintain a clean and efficient data environment.

Statsig currently supports metrics calculation for a single production environment. If you have multiple environments set up, you will only be able to use one as your production environment for metrics and experiments.

When running an experiment, the default behavior is that the experiment will run in all environments. This means that if you try to access the config values of the experiment, you will get a valid value, even if you are in a non-production environment.

If you want your experiment to only run on production, you can set a targeting gate. This will ensure that only users in the production environment will pass the gate and be included in the experiment.

Here is an example of how you might access the config values of an experiment:

javascript useExperiment(XXX).config.get(YYY, false);

In this example, XXX is the experiment you are running, and YYY is the config value you are trying to access. If the experiment is not running in the current environment, you will get the default fallback value, which is false in this case.

Remember, once you start an experiment, it will run in all environments unless you set a targeting gate to restrict it to specific environments.

The best practice is to use the experiment checklist and diagnostics tab to instrument the test, enable it in lower environments, and validate that exposures are flowing through. Then, when you’ve validated these, you click “Start” to go to production. This workflow is typically adequate for most users.

Please note that this is subject to change as Statsig continues to evolve and add new features. Always refer to the official Statsig documentation for the most up-to-date information.

The waitForInitialization option in Statsig's React SDK is used to delay the rendering of your components until the SDK has finished initializing. This ensures that the correct feature gate values are available when your components are rendered. If you're seeing users in the "uninitialized" group, it could mean that the SDK initialization is not yet complete when the feature gate check is being made. This could be due to a slow network connection or other issues that delay the initialization process.

To resolve this, you could consider increasing the initTimeoutMS value, which gives the SDK more time to complete the network roundtrip before falling back to cached values. You could also use the init callback for debugging to check when it's returning. If the issue persists, it might be worth considering using a server SDK on one of your servers to bootstrap with. This way, if you already have a network request serving stuff to your clients, you can have it evaluate the user and pass those values back without needing a roundtrip to the SDK.

Remember, the initCalled: true value doesn't necessarily mean the initialization succeeded. It's important to check for any errors thrown from the initialization method. If you're trying to avoid unnecessary updateUser calls, consider building a statsigUser and only call for update if the local statsigUser is different from the one saved in the SDK instance.

If you set waitForInitialization off, you should get the uninitialized check, and then once SDK initialization completes (within 3 seconds), you should get another check with the actual value (assuming the network request was successful for initialization).

Statsig is working on the metadata around these cases to make it easier to debug. In new SDK versions, it will be possible to differentiate them a bit more. There are also plans to make a change to React that won't even render the children until initialization has at least started so there won't be uninitialized checks at first. This is due to the ordering of effects, where SDK hooks will run before the SDK initialization path in the provider.

When deploying to an environment in Statsig, if you encounter an issue where the RULE is always Default and the REASON is Unrecognized, it typically means that the SDK was initialized, but the config or feature gate you're trying to evaluate did not exist in the set of values. This could be due to a few reasons:

1. The feature gate or dynamic config you're trying to evaluate does not exist or is not correctly spelled in your code. Please double-check the spelling and case-sensitivity.

2. The SDK might not have been able to fetch the latest rules from the Statsig server. This could be due to network issues or if the SDK initialization did not complete successfully.

3. If you're using a server SDK, it's possible that the SDK is outdated and doesn't recognize new types of gate conditions. In this case, upgrading the SDK might resolve the issue.

Remember, the Unrecognized reason is only given when the SDK is initialized, but the config or feature gate did not exist in the set of values.

It's also important to ensure that you are waiting for initialize to finish before making evaluations. For instance, calling checkGate inside the callback or after you are sure the callback has been triggered.

Additionally, check if the environment you've deployed to is able to make requests to the relevant URLs. In some cases, these requests might be blocked by the client in your production environment.

If you're still having trouble, please provide more details about your setup and the issue, and a Statsig team member will assist you shortly.

If you are using the synchronous provider and Server Side Rendering (SSR), the assignment reasons chart should be 100% with the reason “Bootstrap”. There should not be any “network”/“Cache”/etc. If you are seeing a lot of uninitialized values, it could point to a potential implementation issue.

One possible cause could be changes in the user object that you pass into the synchronous provider. If there are fields on the user object that you are loading asynchronously or in an effect elsewhere, it could trigger the SDK to update the values for the user.

To debug this issue, you should verify each of the render passes on the StatsigSynchronousProvider and ensure that there is only a single render with a static user object. If the user object changes and it rerenders, it could lead to the issue you are experiencing.

If you are calling useExperiment() in your code, make sure that the user object passed into the provider is static and never changes. If the user object changes after you bootstrap the StatsigSynchronousProvider, it will cause the provider to re-fetch initialize values using a network request, effectively discarding the results passed to it through SSR.

It's also important to note that if you are using SSR correctly, you should never have a “network” reason. If you are seeing a “network” reason, it could indicate that your users are only going through the Client Side Rendering (CSR) flow.

When integrating Statsig with Google Analytics 4 (GA4), you might notice that all events are sent under the name 'statsig'. This is expected behavior as Statsig sends all logged events and exposures to GA4 under this name once the integration is enabled. These events include properties such as 'category', 'config', 'group', 'value', 'stable_id', 'user_id', 'user_ip', and 'user_userAgent'.

The GA4 integration works for all experiments, not just the ones created after the integration is complete. However, it's important to note that the events are forwarded to a configured Data Stream and can be filtered with event filtering. If you're not seeing the expected events, it might be worth checking your event filtering settings or the configuration of your Data Stream.

Google Analytics is known to have delayed reports, which can range from 24 to 48 hours. Therefore, if you have recently enabled the integration, it might just be a matter of waiting for the events to appear in GA4.

Also, ensure that in the integration configuration, you have selected the necessary options to send all events, including any new events. If you're still having issues, please reach out to the Statsig team for further assistance.

The "Uninitialized" assignments in Statsig could be due to the initTimeoutMs parameter in the Provider. This parameter determines how long the Statsig client waits for the initial network request to respond before proceeding. If this timeout is too short, it could lead to an "Uninitialized" status if the initialization network request doesn't complete within the specified time.

In some cases, a 400ms timeout might be too short, causing the initialization to not complete in time, especially if network conditions are not optimal. It is recommended to consider increasing this timeout value to allow more time for initialization.

However, for Server Side Rendering (SSR) pages using the StatsigSynchronousProvider, there is no concept of initTimeoutMs. Therefore, the initialization happens on the server and there should be minimal uninitialized cases. If you are seeing a notable number of uninitialized cases for an experiment on an SSR page, it could be due to other factors not related to the initTimeoutMs parameter.

Please note that the initialization process involves a network request, and depending on the network condition and location of the users, a short timeout like 400ms might not be sufficient for many users.

When conversions are not showing up in Statsig test results, there are several troubleshooting steps you can take to resolve the issue. First, verify that the events sent from your application include the same identifier (either userID or stableID) as the one used in your client SDKs for the experiment. This ensures that Statsig can correctly join the events with the appropriate experiment groups.

Additionally, be aware that Statsig computes custom metrics on a daily basis for your Metrics dashboard and aggregates them for the duration of the experiment in your Pulse Results. Custom metrics will not populate until the day after creation and will not retroactively apply to previous days.

Statsig will calculate them from the creation date forward. If you are using precomputed metrics, confirm that Statsig is receiving your precomputed metrics. If the Metrics Stream is not visible at the bottom of your Metrics Catalog, there may be a connection issue or an invalid schema.

If these steps have been checked and the issue persists, it is advisable to contact a Statsig team member for further investigation.

When a feature gate is set to Launched or Disabled, it will always return the default value (TRUE for Launched, FALSE for Disabled) and stop generating billable exposure events. However, disabling a feature gate will not stop checks against it - it'll just always return false. Removing it from your codebase will stop checks - assuming you don't have any old versions of your code still running anywhere.

If you're still seeing checks after that, it's possible that there might be some instances where the gate is still being referenced. Double-check your codebase to ensure that all references to this gate have been removed. If the issue persists, it is recommended to flag this issue to the Statsig team for further assistance.

In some cases, you might still see exposure events in Stitch for gates you've launched/disabled (and all are presumably False). This could be due to a bug in the system. The Statsig team is always ready to identify and fix such issues.

Please note that fixes for such bugs are usually rolled out promptly and should resolve over a short period of time. However, if you're still not seeing the issue as resolved on your end, it is advisable to reach out to the Statsig team for further investigation and assistance.

When comparing event counts between Statsig and GA4, it is important to consider the potential for discrepancies due to time zone differences.

Statsig operates on a default day that starts at GMT-8 (Pacific Standard Time) and maintains this standard throughout the year, without adjusting for daylight savings time. This can lead to differences in daily event counts when compared to other platforms that may use different time zones for their reporting.

Additionally, the methodologies for logging and processing events can vary between Statsig and GA4, which might contribute to the observed discrepancies. It is also crucial to check for any filters or exclusions that may be applied within either platform, as these can affect the event counts.

If discrepancies continue to be significant, a thorough review of the event logging code is recommended to ensure that events are being logged accurately and consistently across both platforms. In cases where the issue persists, reaching out to the Statsig support team for further assistance is advised.

When user interactions with a feature, such as clicks on a banner, are not reflected in the Statsig exposure export, it is important to understand that Statsig exports all exposures without exception. If a user has been exposed to a feature gate, they will be included in the analysis. However, discrepancies can occur for several reasons.

For instance, if the user's attributes have changed post-exposure or if the feature gate rules were modified, the user might not qualify anymore but would still be included in the analysis.

To verify exposures, users can log into the Statsig console, select the feature gate, and click on 'View Pulse'. The 'Cumulative Exposures' panel will display the total exposures, including a breakdown of users exposed to the feature (treatment or Pass group) and users not exposed to the feature (control or Fail group).

If issues persist, it is recommended to utilize the debugging tools provided in the Statsig documentation to understand why a certain user received a specific value. In cases where the problem cannot be resolved, reaching out to the Statsig team for assistance is advised.

In the scenario where config.getGroupName() returns null in your SSR page using NextJS and Statsig, it's important to understand that the getGroupName() function is expected to return null if the dynamic config does not have a group name associated with it. This could occur if the dynamic config is not part of an experiment group.

If you believe this is an error and the dynamic config should be part of a group, it's recommended to double-check your configuration in the Statsig console. If the issue persists, it might be a good idea to flag this for further investigation by the Statsig team.

In the case where you're calling getGroupName on an experiment config, such as statsig.getExperiment('test').getGroupName(), and it returns null or "undefined", it's possible that the groupName is not included in the initializeValues payload.

This issue might require a fix to the NodeSDK. The NodeSDK downloads the groupname via Statsig on initialization, but it doesn't include it in the payload generated by Statsig.getClientInitializeResponse.

As a workaround for now, you can create a group name parameter in your experiment or config, and get that from the payload of the dynamic config. This should help you bypass the issue until a fix is implemented.

Yes, the experiment buckets will be preserved for the Stable ID experiments when the user transitions from the Server Side Rendered (SSR) page to the Client Side Rendered (CSR) page or vice versa. This is because the Stable ID is a unique identifier that persists on the device and is used to ensure that your users have the same experience or group across different states, including logged out to logged in states.

Even though the user object (userId) changes between these pages, as long as the Stable ID remains the same, the user's experiment group will remain consistent. This is because the Stable ID is used for bucketing in the experiment, not the userId.

However, it's important to ensure that the Stable ID is correctly implemented and persisted across different pages and sessions. If you're facing issues with Stable ID changing across subdomains, one possible solution is to store the Stable ID in a cookie and reuse it. You can store and retrieve your own id and use the overrideStableID method on the SDK to send it to Statsig. Your overridden id will always take preference.

Remember to always follow the best practices as outlined in the documentation when implementing experiments with Statsig.

When integrating Statsig with Vercel's Edge Config, it is important to understand the behavior of user allocation during updates to experiment configurations.

According to our expert, increasing the allocation percentage in an experiment will not affect users who have already been assigned to the test. However, it is the decrease in allocation that could potentially remove users from the test upon their revisit.

For users who were not allocated during their first visit at a certain percentage, they can indeed be allocated if they revisit after the exposure allocation has increased. This is applicable when using a user ID as the identifier without considering sessions. It is also noted that the Statsig Client SDKs offer a keepDeviceValue argument with getExperiment that enforces stickiness via localStorage, although this does not apply to Server SDKs.

During an update delay, there might be a temporary mismatch in user allocation results when polling from the edge versus polling from the client SDK. The edge configuration might not reflect the new allocation immediately, leading to a period where the user could be allocated on the client side but not yet on the edge.

This mismatch will persist until the edge data is updated to match the new allocation configuration.