Frequently Asked Questions

When configuring alerts for conversion rate changes in experiments, the type of data the alerts are based on is determined by the alert window setting.

If the alert window is set to 'daily', the system will generate alerts based on the data specific to the day the alert is triggered. This approach is particularly useful for monitoring daily fluctuations in metrics.

It is important to note that if a metric tends to exhibit noise throughout the day but remains stable over a 24-hour period, setting a daily alert window may be more beneficial than an hourly one, as it focuses on the daily consistency rather than hourly variations.

In the context of dynamic configurations, the concept of extending configs based on user tiers, while maintaining a base configuration, is a common query. The approach to achieve this functionality is not through dynamic addition of configuration segments, but rather through the use of targeted rules for each tier.

To implement tier-based dynamic configurations, it is recommended to create a complete version of the dynamic config for each tier. This means that instead of having a base config with incremental additions, each tier should have its own full set of configurations. User targeting is then employed to determine which configuration should be applied to a user based on their tier level.

This method ensures that the correct and complete set of configurations is delivered to the user, adhering to the tier they belong to. It is important to note that this process does not support the dynamic addition of configuration segments; instead, it relies on the pre-definition of full configurations for each tier.

Feature gates are a powerful tool for controlling the availability and behavior of features within an application. However, their scope of influence is limited to the application's internal features and does not extend to system-level settings or functionalities.

Specifically, feature gates cannot be used to hide custom keyboard extensions in system settings, as this would require direct interaction with the system's configuration, which is outside the purview of feature gates.

To achieve such functionality, developers would need to write code that interacts with the system settings directly.

Once such code is in place, feature gates can then be employed to control the execution of this code, effectively enabling or disabling the hiding of custom keyboard extensions based on the feature gate's configuration.

In the context of adding filters to metrics within a funnel, it has been confirmed that this functionality is indeed supported. Users are able to apply filters to Event_count and User_count metrics, which can then be utilized in the creation of a funnel.

This provides the flexibility to tailor metrics to specific needs and answer targeted questions effectively. An example provided includes the creation of a “Product View Sale” metric with an applied filter, which is then visible and operational within the funnel creation workflow.

It is important to note that while filters can be added, they are limited to the aforementioned metric types.

Currently, there isn't a feature to 'freeze' a gate in the way described. When a feature gate's status is changed to 'Launched' or 'Disabled', it will always return the default value (TRUE for Launched, FALSE for Disabled) for all users, regardless of whether they are new or old. The gate will stop generating billable exposure events, and you'll stop incurring costs. The gate reference is likely safe to be cleaned up in the codebase.

Implementing a feature to remember the previous evaluation for a gate would be challenging. However, if you want this sort of behavior, you could implement it yourself. If you have a client-side app, it would be fairly easy to store someone's evaluations in local storage, and check for a value there before checking Statsig.

For server-side implementation, you would need a database and a bit more work. This would allow you to store the evaluations and check for a value before checking Statsig, effectively creating a 'freeze' behavior.

Please note that these are workarounds and not built-in features of Statsig. If you have further questions or need assistance with a specific use case, please reach out for further assistance.

In the context of Autotune experiments, it is important to note that adding a new variation to an existing experiment is not possible without resetting the experiment. If you need to introduce a new variation, the process involves pausing the current experiment, making the necessary changes, and then restarting the experiment.

However, it's crucial to understand that this process will reset the experiment, which means that any results obtained prior to the reset will be lost. This is a fundamental aspect of the Autotune experiment's design and operation.

Please ensure to consider this factor when planning to add new variations to your Autotune experiments.

According to current capabilities, there isn't a feature that allows automatic rule creation for new feature gates based on certain tags. When a new feature gate is created, rules for that gate need to be added manually. However, there is a workaround to use the same rule across multiple feature gates.

This can be achieved by creating a segment with the desired criteria and then applying this segment to the rules of your feature gates.

Additionally, there is another method to simplify the process of creating new feature gates with a specific rule. You can create a template feature gate with your rule in it. Then, you can use the “clone” feature to create new identical gates with that rule.

This method allows you to replicate the rule across multiple feature gates without having to manually add it each time.

In the Statsig SDKs, there is no functionality to retrieve a user's historical gate pass status.

The SDKs are designed to evaluate the current state of a feature gate for a user at the time of the check, rather than providing historical data. As such, it is not possible to use the SDK to directly query if a user has passed a certain gate in their history.

To track a user's interactions with feature gates over time, it is recommended to log an event each time a user passes through a gate using the Statsig.logEvent function available in the SDKs. These logged events can then be analyzed within the Statsig console to gain insights into the user's history with the feature gates.

It is important to note that when checking a gate or logging an event, comprehensive user information should be provided to leverage the full capabilities of the Statsig platform.

At present, there is no direct support for configuring the recipient of feature gate emails to an entire team instead of just the individual who created or edited it. However, there is an alternative way to receive notifications for a gate. Users can "subscribe" to them by clicking on the subscribe button. This will allow them to receive notifications related to the feature gate. Please note that this is the only other way to receive notifications for a gate currently supported.

In the current setup of Statsig, there is no direct method to create a feature gate or a user segment based on a specific event occurrence. Feature gates and user segments are typically created based on user attributes or conditions.

However, you can log custom events using the Statsig SDK and then analyze these events in the Statsig console. While this doesn't directly influence the feature gate or user segment, it can provide valuable insights into user behavior that can inform your decision-making process.

For a more automated solution, you might consider using the Console API. While it doesn't directly support event-based targeting, it can help streamline the process of checking for specific user attributes in gates or segments.

It's important to note that the Statsig SDKs are designed to be performant, meaning they make bucketing decisions based on rules/config they have, and can’t make async calls to look up state about a user as part of making this decision.

If you need to gate based on a user's state, you would need to manage the state of a user property (for example, isSubscriber?) and pass this along in the user object for the SDK to be able to gate based on this state.

For further questions or more specific assistance, it is recommended to reach out to the Statsig team directly. They may be able to provide more detailed guidance or discuss potential future features.

Creating a sequential funnel that includes a step with multiple events is not directly supported as funnel metrics are designed to be linear, with each step corresponding to a single event. However, you can gain similar insights by utilizing Ratio Metrics.

Ratio Metrics allow you to define a metric using other metrics as the numerator and denominator. This means you can create a metric that combines the counts of multiple events, which can provide an analogous understanding to having two events count as the same step in a funnel metric.

To implement this, you would define a new Ratio Metric where the numerator is the sum of the events you wish to combine, and the denominator is the preceding event in the funnel. This approach allows for flexibility in analyzing user behavior across multiple events without altering the linear structure of funnel metrics.

Yes, it is possible to extend the target duration of your A/B experiment after it has been set up. Changing the target experiment length in the advanced settings will not result in data loss or change the assignment to Control/Test. The target experiment length is primarily used for tracking progress against your experiment's duration. It does not impact the data collected or the group assignments within the experiment.

By default, Statsig will compute Pulse Results (i.e., metric lifts) for your experiment for the first 90 days. If you need to extend beyond this, you will be able to do so as you approach the 90 days cap. Please note that extending the duration of your experiment does not affect the data collected or the group assignments within the experiment.

When updating log events for a feature gate in Statsig, there is no need to restart the feature gate to see the revised metrics data, as changes should take effect immediately. However, if the updated metrics are not reflecting as expected, it is advisable to verify that the events are being logged correctly.

To obtain cleaner data for metrics lifts after modifying the code for a log event, you can adjust the distribution of users between the control and experiment groups or 'resalt' the feature to reshuffle users without changing the percentage distribution.

For experiments with delayed events, setting the experiment allocation to 0% after the desired period ensures that delayed events still count towards the analytics. It is important to note that the sizes of variants cannot be adjusted during an ongoing experiment to maintain the integrity of the results.

To increase the exposure of a variant, the current experiment must be stopped, and a new one with the desired percentage split should be started.

In the context of managing experiments with Terraform, the status field can be updated to reflect one of four possible values: setup, active, decision_made, and abandoned, aiding in the management of the experiment's lifecycle.

For those utilizing Statsig Warehouse Native, creating an Assignment Source and a new experiment allows for the definition of the experiment timeline and subsequent calculation of results.

Statsig pipelines typically run in PST, with data landing by 9am PST, although enterprise companies have the option to switch this to UTC.

Statsig Cloud calculates new or changed metrics going forward, and Statsig Warehouse Native offers the flexibility to create metrics after experiments have started or finished and to reanalyze data retrospectively.

In the context of managing feature gates within Statsig, the ability to rename an existing feature or experiment is not permitted.

This restriction is in place to avoid potential errors that could arise from renaming a feature that is already being referenced in the codebase, which could lead to the feature gate not returning the correct values.

If there is a need to change the name of a feature gate that has not yet been deployed, the recommended approach is to clone the existing feature gate and assign it the desired new name. This action will result in the creation of a new feature gate with the same configuration rules as the original, but it is important to note that the historical data and metrics associated with the original feature gate will not be transferred to the new gate.

In Statsig, metrics that have been around for a while and contain data can be renamed without losing any historical data. This feature is being rolled out and allows for a more comprehensive approach to renaming metrics while preserving their associated data.

If you have a metric that requires renaming and you wish to retain its data, you can request to have your project included in the rollout of this new feature. Once your project is included, you will be able to rename your metrics accordingly.

It is recommended to provide feedback or report any issues encountered during the use of this new renaming feature to ensure continuous improvement and resolution of potential problems.

In Statsig, you have the ability to both restart and extend an experiment.

To restart an experiment, you can utilize the 'Reset' option. This action will revert the experiment to an unstarted state, and all users will receive the default experience. The "salt" used to randomize a user's group will also be changed. This means that when you start the experiment again, your users will be randomly assigned to a group that is not necessarily the same group they were in prior to the experiment being reset.

When you reset an experiment, it also clears the metrics. However, you will still be able to see the old snapshot in the history, but the experiment itself will start afresh.

To extend an experiment, you can adjust the target duration in the advanced settings. Changing the target experiment length will not result in data loss or change the assignment to Control/Test. The target experiment length is primarily used for tracking progress against your experiment's duration. It does not impact the data collected or the group assignments within the experiment.

Please note that the sizes of variants cannot be adjusted while the experiment is in progress. If you wish to increase the exposure of a variant, you would need to stop the current experiment and start a new one with the desired percentage split.

Yes, it is possible to run feature gates within existing experiments. You can add new features to a variant by creating a feature gate that only passes for users who are in that variant. However, it's important to note that adding new features to a variant while an experiment is running could potentially impact the interpretation of the experiment results. Therefore, it's generally recommended to plan your experiment design carefully before starting the experiment to avoid making changes midway.

If you need to run multiple experiments or add new features, you might want to consider using layers. Layers are a powerful way to run many experiments in sequence, one building on top of the learning from the other. This could be a solution for running multiple experiments on the same users without having to change code.

Remember, once you start an experiment, it will run in all environments unless you set a targeting gate to restrict it to specific environments.

If you want to change the behavior of a variant, the recommended approach is to start a new experiment that is a vs. the new variant with the new feature being part of the new variant.

Currently, Statsig does not provide a direct way to send different types of notifications to different channels in Slack. When setting up the Slack integration, you select a channel to direct all project-level messages. However, once the integration is enabled, you have the option to opt-in to additional notifications such as Health Checks and Statsig system Status Reports. You can also filter change updates by the environment affected.

The notifications will include updates such as when a gate has been edited, which should provide some context about the change. However, the ability to add specific message or rule details is not explicitly mentioned in the documentation.

If you require more specific functionality, it is recommended to reach out to the Statsig team for further assistance. Please note that during the installation process, you instruct the app which channel to send all the messages.

In the context of setting experiment group splits with decimal precision, the system is designed to limit the total number of buckets to 10,000, which allows for one digit of decimal precision.

When dealing with multiple variations, such as 32 in an experiment, it is recommended to assign 3.1% for 31 variants and 3.9% for the control group. This approach is considered acceptable for the experiment, and the slight difference in precision (e.g., 3.1% vs. 3.125%) is deemed negligible, especially in larger sample sizes where the difference might equate to a small number of users.

For experiments requiring more precise splits, alternating between 3.1% and 3.2% for every eighth variant is a suggested strategy. This allocation method is also applicable to more complex experimental designs, such as a 2-level fractional factorial experiment.

The key consideration is the statistical precision, which may be more critical depending on the user base size. It is important to note that the fixed number of buckets behind the scenes necessitates these allocation strategies.

In the event that an incorrect decision group is selected during an experiment, there are a couple of approaches to address the issue.

Firstly, if the parameters on the control and test groups can be swapped without any hardcoded behavior specific to the control or test, this action could potentially resolve the problem. However, it is important to note that this may lead to corrupted data.

Alternatively, the experiment can be reset, and a decision can be made immediately thereafter. It is important to understand that resetting the experiment and making a new decision will replace the top-level results with the new iteration of the experiment, which will not contain data.

However, data from previous iterations of the experiment can still be accessed through the 'View History' feature. This allows for the preservation of historical data while addressing the issue of the incorrect decision group selection.

Yes, it is possible to use an event value for Segment events. When sending events from Segment to Statsig, anything in the properties field of the Segment event is treated as the event metadata field on Statsig's end. This can include an event value. For instance, you can use this field to specify the type of achievement that was unlocked, or the name of the product that was added to cart, or the price of the item that was purchased. However, the specific implementation details might need to be confirmed with the Statsig team.

When it comes to the Autotune configuration, it's important to note that it currently doesn’t support optimizing for the value on an event. Autotune can only be configured to optimize for the count or occurrence of an event. For example, if you want to pick the variation that will have the most number of success events, you pick that event.

For A/B testing with randomized number distributions, Statsig's experiments feature is the recommended approach.

To achieve this, you should create multiple test groups within an experiment. Each user who interacts with your application will be deterministically assigned to one of these test groups based on their user ID, which could be either a stableID or userID.

This assignment ensures that each user consistently experiences the same test group during the experiment. Test groups can be configured with associated parameters, allowing for dynamic control over the behavior and experience of each test group.

This setup enables you to conduct A/B/n experiments where users are evenly distributed across the different configurations you wish to test.

For detailed guidance on configuring your groups and parameters, refer to the official Statsig documentation on creating and managing experiments.

To view raw events for an experiment group in Statsig, you can utilize the Events Explorer feature. This tool is designed to provide you with sample rows for events, which is particularly useful when you need to familiarize yourself with the data schema or wish to observe the exact data being logged.

The Events Explorer operates on a 14-day rolling window of sampled data, which allows for near real-time data availability. It is an excellent resource for identifying trends, making day-to-day or week-to-week comparisons, and gaining a high-level understanding of event occurrences.

However, it is important to recognize that the Events Explorer is not intended for retrieving all data associated with a specific user or for producing a precise count or export of a particular event. For tracking individual user events and exposures, the Users tab should be utilized instead.

Please note that the Events Explorer's sampling approach may not provide a complete dataset for exhaustive analysis but is suitable for general insights and trend monitoring.

At present, Statsig does not support the use of Sankey charts for visualizing user journeys. However, you can utilize funnel metrics to track user journeys within your application. These funnel metrics can be highly customized to provide insights into how changes influence each step of the user journey, either at the user level or at the session level.

To create funnels in Statsig, navigate to Metrics -> Explore -> select “Funnel” in the dropdown menu. Please note that this is a beta feature and currently does not support having experiment exposures in the steps. However, this is a feature we are considering for future development in our analytics offerings.

While visualizations like Sankey charts can be useful, they often struggle to effectively represent small changes, such as 5% deltas, in experimentation. Therefore, even if the conversion rate doubled (+100%), it might still be challenging to spot in a Sankey chart. We are actively investigating this area and are open to customer feedback to find an ideal visualization method.

Currently, the feature to split L7 (Last 7 days), L14, and L28 metrics by different groups, segments, or cohorts is not available.

This functionality is under active development, and considerations are being made to include it in future updates. As of now, users looking to analyze metrics by these specific time-bound parameters will not be able to use the 'Group By' feature to disaggregate the data by properties such as user type, geographical location, or other custom parameters.

This limitation applies to the built-in metrics within the platform. Users are encouraged to stay tuned for updates on the development of this feature, which aims to provide more granular insights into user behavior and metric performance over specified time frames.

Using multiple SDKs on a site is technically possible, but it can lead to complications. The Statsig JavaScript SDK and the React SDK are designed to be used independently. If you use both on the same site, they will each maintain their own state and won't share information.

This could lead to inconsistencies in feature gate evaluations and experiment assignments.

If your site is primarily in JavaScript but includes a React app, it's recommended to initialize Statsig with the JavaScript SDK and then pass the initialized Statsig object into the React app. This way, you ensure that both parts of your application are using the same Statsig state.

However, if your React app is substantial or complex, it might be more beneficial to use the React SDK for its additional features, like the StatsigProvider and React hooks. In this case, you should ensure that the JavaScript part of your application doesn't also initialize Statsig, to avoid the aforementioned issues.

The React SDK brings in the JS SDK as well, so you don’t need to include it separately. Once that is initialized then you can use the ‘statsig’ object directly.

However, you’ll have to be careful - the react SDK internally keeps track of the state of the user, so if you try to update the user outside the provider, your react component tree won’t rerender with updates. Similarly, if you try to call methods before the react SDK initializes the js sdk instance internally, you could run into issues.

The best approach depends on the specifics of your application and its usage of Statsig.

In a subscription-based business model, understanding the long-term impact of experiments on Customer Lifetime Value (CLV) is crucial.

Statsig provides the capability to analyze the impact of an experiment on CLV over extended periods, such as 6 months. To facilitate this, Statsig allows for the setup of an experiment to run for a specific duration, such as 1 month, and then decrease the allocation to 0%, effectively stopping new user allocation while continuing to track the analytics for the users who were part of the experiment.

This tracking can continue for an additional 5 months or more, depending on the requirements. It is important to note that the experience delivered to users during the experiment will not continue after the allocation is set to 0%. However, there are strategies to address this, which can be discussed based on specific concerns or requirements.

Additionally, Statsig experiments by default expire after 90 days, but there is an option to extend the experiment duration multiple times for additional 30-day periods. Users will receive notifications or emails as the expiration date approaches, prompting them to extend the experiment if needed.

This functionality is available on the Pro plan, ensuring that businesses can effectively measure the long-term impact of their experiments on CLV without direct integration with a data warehouse and by updating CLV through integrations such as Hightouch.

Statsig currently does not offer the capability to filter inbound events based on specific userIDs or stableIDs. However, there is a plan to introduce a 'Global Targeting Rule' which will allow users to exclude certain types of users from all tests and feature gates.

It is important to note that this 'Global Targeting Rule' is designed to omit users from tests and gates, and not for the purpose of optimizing billing. Users who wish to exclude specific traffic for billing optimization will need to implement logic to discard such traffic upstream, either within their data collection tools or by adding a wrapper around Statsig log event calls.

In scenarios where you wish to modify the default behavior of a feature gate in Statsig, which typically fails (returns false) when conditions are not met, there are alternative strategies you can employ.

One approach is to use negation operators within your conditions, such as 'not in' or 'not equal to', to effectively invert the logic of the gate. This allows you to create a condition that passes by default unless specified criteria are met. Another method is to adjust the 'fail' percentage to 100%. By doing so, you ensure that the feature gate fails for all users, effectively reversing the default 'off' state to an 'on' state.

These techniques allow for greater flexibility in controlling feature gate behavior and can be tailored to meet the specific needs of your experiment or feature rollout.

When you click on an event in the Event Log Stream, a popup with detailed information, including the stableID, should appear. This is the default behavior when the ID type configured on the gate/experiment is shown in the list view.

If you want to make a gate or experiment a stableID based gate/experiment, the most relevant column to show to ensure you have configured your gate/experiment and checks in code correctly is the unit type of the gate/experiment itself. If you change the unit type to “stableID”, that column will change to stableID.

It's important to note that Statsig allows you to experiment on userID, stableID, and up to 5 customIDs. Therefore, the table can't really scale to show all the IDs inline. However, once the unit type is changed to "stableID", the Statsig Stable ID will appear in the exposure stream.

Yes, it is possible to run two landing page type of experiments simultaneously with different layerIDs. Each experiment will have its own layerID, which allows you to run multiple experiments without needing to update the code on the website for each experiment.

The first experiment can run on the landing page, and the second experiment can run on the ThankYou page. Even though the flow from the landing page leads to the ThankYou page, the two experiments are independent because they have different layerIDs.

The code for setting up these experiments is as follows:

<script  src="https://cdn.jsdelivr.net/npm/statsig-landing-page-exp?apikey=[API_KEY]&layerid=[LAYER_NAME]"> </script>

Remember to replace [API_KEY] and [LAYER_NAME] with your actual API key and the layer name for each experiment. Also, update the script to specify the layerid instead of expid when you run experiments as part of a layer.

However, if you are trying to measure an impact after the thank you page, you probably want both of these experiments to be in the same layer.

Lastly, ensure that the two experiments are not in the same layer, because in that case, you will mutually exclude the same users from each other experiment.

In the context of updating experiments and managing permissions for editing monitoring metrics, it is important to understand the current capabilities of role-specific permissions.

According to the expert, there is a setting within the 'Members & Roles' section under 'Project' that allows for role-specific permission adjustments. By navigating to the 'Roles' tab and selecting a 'Member' role, users can access the 'Permissions' tab where they can find a setting called 'self-approve review.' This setting enables a role to bypass the review structure entirely.

However, it is noted that there is no current permission setting that allows for the customization of permissions to the extent of enabling a user to edit monitoring metrics specifically without granting them the ability to bypass all reviews.

The expert acknowledges the validity of this use case and has indicated that they will discuss the possibility of implementing such a feature with their team. As of the expert's last statement, permissions for general metric management and verify metrics exist, but not for editing monitoring metrics specifically.

Currently, the capability to backfill results for a metric that was added midway through an experiment is not available. This limitation is recognized, and plans are in place to address this issue in the future. However, there is no estimated time of arrival for this feature.

As of now, if a new metric is introduced, it will only be calculated from the date of its creation and cannot retrospectively apply to data collected before that point.

In the context of using Statsig for feature management, it is important to understand how to identify all the feature gates that utilize a specific user segment.

While there is no direct method or function provided to automatically list all feature gates that use a particular segment, users can manually check the rules of their Feature Gates within the Statsig console.

Each rule within a Feature Gate specifies the user attributes it operates on, and by examining these rules, one can determine if a specific segment is being used. For those seeking to automate this process, the Statsig Console API offers a solution.

The segment endpoint of the API exposes the attributes that are in use, which can be leveraged to programmatically identify the use of specific user attributes, including segments, across various gates. This approach can significantly streamline the process of checking for the usage of specific segments in the configuration of Feature Gates.

The tutorial on setting up an A/B/n test using Statsig's Experiments feature uses two examples to illustrate different variants you might test in an experiment.

The first example, enable feature, is a boolean type feature gate that can be either true or false. This represents the feature being on or off and is a simple A/B test scenario where you're testing the impact of a feature being enabled vs. not enabled.

The second example, Circle/Square, is a multi-variant scenario where you're testing more than two variants. In this case, you're testing the impact of showing a circle vs. a square shape in your product.

The screenshots in the tutorial are meant to illustrate the process of setting up these tests in the Statsig console. If you're finding the tutorial confusing, it is recommended to follow along in the Statsig console as you read. This should help clarify the process.

The tutorial is designed to use an A/A test as an example to show you how to set up any A/B test easily, of which A/A test is a special case. However, we acknowledge that it can seem a little confusing and we are committed to improving it for the future.

If you have any specific questions about the tutorial or any other aspect of using Statsig, feel free to ask. We're here to help.

When integrating metrics ingestion with Statsig, it is not necessary for developers to apply their own date filters within their SQL queries. Statsig is designed to manage date filtering autonomously during the data backfill process.

Upon setting up data ingestion, developers provide a SQL query that generates a view for Statsig to ingest. This query is then utilized by Statsig to incrementally load data on a daily basis. For the purpose of backfilling, Statsig will inspect the data for the past three days from the current date to determine if there has been a significant change, defined as a variation exceeding 5% in the number of rows.

If such a change is detected, Statsig will automatically initiate a backfill. For data outside of this three-day window, it is expected that the developer will manually trigger a backfill for the desired date range. Therefore, the SQL query provided for data ingestion should not be constrained by row limits, as Statsig will handle the necessary date filtering to ensure efficient and accurate data ingestion.

When you add a list in the override in Statsig, the system ensures that the users in the override list are assigned to the specified group. This action, however, does not affect the 50/50 split for the rest of the users who are not in the override list. The assignment of users is deterministic based on their ID, so your overrides won't affect other users.

Furthermore, it's important to note that overrides don't affect assignment for users outside of the overridden users. Overrides are typically meant for testing and by default, they are excluded from analysis. This means that even with the override assigned to a specific group, the test/control group split will still maintain a 50/50 ratio.

When using checkGateWithExposureLoggingDisabled at launch, it will not log an exposure event. This is particularly useful when you don't want to log exposures for users who may not reach the feature you're testing.

Later, when the user reaches the feature, calling checkGate will indeed log an exposure event. This is because checkGate by default logs an exposure event when it is called. Therefore, you do not need to call manuallyLogGateExposure in this case.

However, it's important to be aware that manually logging exposures can lead to an imbalance in exposure events if not done correctly. For example, only triggering exposures for users in the Test group of an experiment could imbalance the experiment.

Both checkGate and checkGateWithExposureLoggingDisabled will return the same result for a given user & gate. The thing to watch out for is if there are any changes between checkGateWithExposureLoggingDisabled and checkGate (e.g. gate is updated or user is updated).

In A/B testing, the inclusion of non-interacting users in the sample size does not inherently affect the p-values, provided that the metric being evaluated is relevant to the feature being tested. When non-interacting users are included in the sample, it can dilute the experiment results by adding a number of zeros to both the control and test groups.

This dilution can decrease the experimental power, meaning that a larger sample size or a longer experiment duration may be required to reach statistical significance, given the underlying effect size.

However, the directionality of the results should remain correct, and the zeros should be evenly distributed between the groups, maintaining the integrity of the experiment results.

It is recommended to expose users to the experiment as close as possible to the point where their experiences diverge, to avoid overexposure and to increase the efficiency of the experiment.

f the overexposed users do not contribute to the metric, they do not introduce any noise and therefore do not affect the experimental power.

It is important to note that if the treatment affects the rate at which people reach the relevant step in the process, manual filtering to users who reached that step could introduce bias and should be approached with caution.

When an experiment is reset in Statsig, it should not alter the overrides configured for it. The expected behavior is that resetting an experiment does not modify layer settings.

However, an issue was identified where resetting an experiment resulted in the overrides for that experiment in the layer being removed. This was not the intended functionality, and a fix has been implemented to ensure that resetting an experiment will no longer affect the layer overrides.

If any unexpected behavior is observed related to experiment resets or layer overrides, users are encouraged to report it for further investigation.

When a project member, who is part of a single review group, creates a new Feature Gate within a project that enforces reviews, the review group will not be automatically added to the ALLOWED REVIEWERS for the new Feature Gate.

The creation of a new Feature Gate does not require a review by the review group. However, any updates to the configuration of the Feature Gate will automatically include the review group as part of the review process.

Statsig does not automatically capture any system events, including the app launch event. However, you have the ability to manually log custom events using the logEvent function provided by the SDK. This function allows you to track any event you deem relevant, including the launch of your app.

Here's an example of how to use it:

For Android:

kotlin Statsig.logEvent("app_launch")

For iOS:

swift Statsig.logEvent("app_launch")

Remember to call this function at the appropriate place in your code to track the app launch event.

Statsig has implemented measures to track bot traffic, including search engine crawlers, which can significantly inflate the number of stable_ids, making your user base appear larger. However, these bots typically do not trigger any events and therefore do not contribute to metrics. As a result, this behavior should not impact the computation of lift or the estimate of the confidence interval in most cases.

In some instances, bots may cause SRM issues. If you observe this, it could potentially be bot-related. To mitigate this, you can manually add filters to exclude bots using a targeting gate. For instance, a filter that excludes traffic where the browser contains "bot" can be quite effective.

While Statsig is actively working on a more comprehensive approach to handle bot traffic, it is currently recommended to use any deterministic detection on your side to filter out bot traffic. This can help align the exposure data more closely with other analytics tools and ensure that bot traffic does not affect the Stable ID assignments or the experiment results.

Please note that the impact of bot traffic should be evenly distributed between control and test, so any impact should cancel out. However, if you have further questions or need more specific advice, please reach out for assistance.

Statsig currently does not offer a native integration with Klaviyo. However, the possibility of such an integration is being explored.

Statsig users looking to incorporate Klaviyo events such as email opens, subscriptions, and email clicks into Statsig for the purpose of personalizing their sites may have to wait for future updates regarding this integration.

Additionally, Statsig is considering whether users would like to run experiments directly within Klaviyo on various channels like email and push notifications, or prefer to conduct experiments within their own stack, including web, mobile, and backend platforms.

Statsig does not provide any folder-like features for managing feature gates. However, you can effectively manage your feature gates through the Statsig console. This platform allows you to create, edit, and archive feature gates as per your requirements.

In case you have a large number of feature gates, you can utilize the search and filter options available on the console. These features will assist you in managing your feature gates more efficiently.

Additionally, Statsig offers the use of Tags to help categorize and search for gates within the console. This feature can be particularly useful when you need to manage a large number of feature gates. By tagging your feature gates, you can easily search and filter them, making the management process more streamlined and efficient.

When executing the updateUser function in Statsig, the new user object does not merge with the previous one. Instead, it completely overwrites the previous user object. This is because Statsig operates on the user object passed in each call and does not maintain or update user objects. Therefore, any new information you want to include must be present in the new user object you pass to updateUser.

Statsig does not provide an inherent mechanism to distinguish between new and returning users within its SDK or API.

To differentiate between these user types, developers must implement their own logic to identify whether a user is new or returning. This can be achieved by setting a flag or property within the user's browser or device storage, such as a JavaScript cookie or a local storage flag for web applications, or leveraging device storage for mobile platforms.

Once this identification is made, the information should be included as a user attribute when logging events or checking feature gates with Statsig.

It is important to ensure that this user attribute is passed along with the user object to Statsig to maintain the distinction between new and returning users within the platform's analytics and feature management.

Switching to the native Statsig event logger does not inherently change the synchronization time to Pacific Standard Time (PST), as the synchronization schedule is consistent regardless of the logging method. However, the Statsig client SDK does facilitate more real-time data capture by automatically flushing logged custom events to the Statsig servers every 10 seconds.

For experiment results, Statsig has implemented a more frequent refresh window. During the first day of an experiment, results are computed hourly. After the initial 24-hour period, the results are then refreshed on a daily basis.

It is important to note that these computations are based on a day defined from midnight to midnight PST. For businesses requiring synchronization in Coordinated Universal Time (UTC), Statsig offers the ability to switch to UTC, but this is an option available exclusively for Enterprise customers. This is due to the additional costs associated with maintaining a separate pipeline for UTC synchronization.

The Statsig React.js SDK does not automatically capture app launch events. However, you can manually log custom events, including an app launch event, using the Statsig.logEvent() function.Here's an example of how you might do this:

javascript import { Statsig } from "statsig-react";   // Log an app launch event Statsig.logEvent("app_launch");

Please remember to initialize the SDK before logging events. For more detailed information or assistance, refer to the Statsig documentation.

If you are using the Native Android and ios SDKs of Statsig, you would still need to manually capture the app launch event using statsig.logEvent('app_launch').

In Statsig, user allocation to a holdout group is a predetermined process that occurs automatically when the holdout group is created. This allocation is based on a cryptographic technique using a salt, which is applied to the unit_id (userID or stableID) of the user being evaluated.

As a result, the assignment to the holdout group is stable as long as the user retains the same unit_id. There is no specific trigger required for this assignment to take place. However, it is important to note that while the assignment is predetermined, the actual exposure of a user to a holdout group only occurs when the user encounters a feature gate or experiment that is part of the holdout.

This means that Statsig only becomes aware of the users when they are exposed to a gate or experiment. It is also crucial to understand that if you are managing the assignment for holdout yourself, you will need to manage exposure logging. This approach is not commonly recommended for holdouts due to the risk of polluting the holdout data.

In the context of hypothesis testing, the p-value is the probability of observing an effect equal to or larger than the measured metric delta, assuming that the null hypothesis is true. A p-value lower than a pre-defined threshold is considered evidence of a true effect.

The calculation of the p-value depends on the number of degrees of freedom (ν). For most experiments, a two-sample z-test is appropriate. However, for smaller experiments with ν < 100, Welch's t-test is used. In both cases, the p-value is dependent on the metric mean and variance computed for the test and control groups.

The z-statistic of a two-sample z-test is calculated using the formula: Z = (Xt - Xc) / sqrt(var(Xt) + var(Xc)). The two-sided p-value is then obtained from the standard normal cumulative distribution function.

For smaller sample sizes, Welch's t-test is the preferred statistical test due to its lower false positive rates in cases of unequal sizes and variances. The t-statistic is computed in the same way as the two-sample z-test, and the degrees of freedom ν are computed using a specific formula.

While the normal distribution is often used in these calculations due to the central limit theorem, the specific distribution used can depend on the nature of the experiment and the data. For instance, in Bayesian experiments, the posterior probability distribution is calculated, which can involve different distributions depending on the prior beliefs and the likelihood.

It's important to note that it's typically assumed that the sample means are normally distributed. This is generally true for most metrics thanks to the central limit theorem, even if the distribution of the metric values themselves is not normal.

In order to change the directionality of a custom metric in Statsig, you would typically navigate to the Metrics Catalog tab, select the metric you want to edit, and then click on the "Edit" button. However, there was a known issue where the edit icon was hidden until you mouse over it due to a design change. This issue has been resolved and you should now be able to see the edit icon without having to mouse over it.

If you still encounter any issues with changing the directionality of a custom metric, please refresh your page as the fix should be applied. If the problem persists, please contact the Statsig support team for further assistance.

The request body for feature gates in the React client SDK is obfuscated and encoded in a specific way that is intended to be processed on the server side by Statsig. This is a security measure implemented to protect the integrity of the data. As such, it is not designed to be decoded on the client side and there is no supported method for doing so.

If you need to inspect the values of your feature gates, it is recommended to do so through the Statsig console. This is the most reliable and secure method to access and manage your feature gates. Please refrain from attempting to decode the obfuscated content on the client side as it may compromise the security and integrity of your data.

To edit the metrics included in a tag, you can follow these steps:

1. Navigate to your Metrics Catalog.

2. Use the filter in the upper right-hand corner of the table to filter to the tag you want to modify.

3. Remove the tag from the relevant metrics.

This process allows you to easily manage and modify the metrics associated with a specific tag directly from the Metrics Catalog, without the need to go into Project Settings. Please note that any changes you make will only apply moving forward and will not retroactively affect previously calculated metrics.

To enable a holdout group within Statsig, you will need to navigate to the Experiments section of the Statsig console.

Once there, you will find holdouts as a tab within this section. If you encounter any difficulties or have further questions regarding the creation or management of holdout groups, please feel free to reach out for additional support.

To manually test different experiment variants without the necessity of opening a new tab, you can utilize overrides to control which variant is exposed to a QA user or developer. Overrides can be implemented in two primary ways to facilitate manual testing within the same context.

Firstly, you can add the QA user ID to the overrides in the experiment setup. This method allows you to manage the experiment assignment behavior directly from the Console. Secondly, you can use a local override API to set specific overrides via code.

This approach enables you to change the behavior on the fly, similar to modifying a value in local storage. For instance, calling Statsig.overrideConfig in a browser's developer tools window can alter the experiment's behavior immediately.

These methods ensure that manual testing of different variants can be conducted efficiently and effectively within the same browser tab, providing a seamless experience for QA testers and developers alike.

To filter by a specific user segment within a dashboard or chart, you can utilize the Metrics Explorer feature. Here, you can create or edit a chart and apply filters to individual metrics based on User dimensions.

When you need to filter by a particular user segment, you can select the 'Filter with User ID Segment' option. This feature allows you to define a user segment based on an ID list or set conditions that users must meet to be included in the segment. After applying the necessary filters and configuring your chart, you can export it to your dashboard.

Any modifications made to the chart in the Metrics Explorer will be reflected in the dashboard once saved. Additionally, you can adjust an existing dashboard by entering the edit mode and adding filters or cohorts through the settings panel on the left. This approach enables you to apply user segment filters directly within the dashboard environment.

To create your own StableID for server-side experiments, follow these steps:

1. Generate a unique identifier on your server. This could be a UUID or any other form of unique ID that you prefer.

2. Store this ID in a cookie or another method of your choice to be used on the client side as well.

3. Set this ID as the customID on the StatsigUser object when generating the initialize response from the SDK. Here's an example of how you can do this:

javascript const userObj = {      customIDs: {          stableID: '<generated in your code or passed via cookie or via db lookup>'      }   }

4. Get that ID from the cookie, and set it as the customID on the StatsigUser object when using the client SDK, so all event data and exposure checks tie back to the same user.

If you wish to use the stableID field rather than a custom ID, you still need to do step (2) above. Then:- Override the stableID in the client SDK by getting the value from the cookie and setting the overrideStableID parameter in StatsigOptions. - Set the stableID field on the StatsigUser object in the customIDs map when generating the initialize response from the SDK.

Please note that this information is based on the documentation's opinion.

Statsig differentiates data from different environments by the environment tier you specify during the SDK initialization. You can set the environment tier to "staging", "development", or "production". By default, all checks and event logs are considered "production" data if the environment tier is unset.

Experiments and metrics primarily factor in production data. Non-production events are visible in diagnostics, but they are not included in Pulse results. This is because most companies do not want non-production test data being included. If you want to include these, you can log them as regular events. However, non-production data is filtered out of the warehouse and there is no other way to include it.

When initializing, if you add { environment: { tier: "production" } }, this would set your environment to "production", not "staging" or "development". If you want to set your environment to "staging" or "development", you should replace "production" with the desired environment tier.

Pulse results are only computed for “Production” tier events. To see Pulse results, you need: • An experiment that is “Started” (aka, enabled in production) • Exposures & events in production tier • Exposures for users in all test groups • Events/metrics associated to users in test groups

As long as you have initialized using { environment: { tier: "production" } }, your Pulse will compute. This means that even if your code is deployed to staging, as long as you initialize with the production tier, you will be able to see Pulse results.

When utilizing User Count metrics in Statsig for analyzing reports, the handling of metadata is contingent upon the configuration of the metric. If a User Count metric is created based on an event, it will count as one for the user for that day.

However, the metric can be further refined to consider specific metadata values. By configuring the User Count metric to filter for a particular metadata value, only events that contain that specified metadata value will contribute to the count. This allows for a more granular analysis where the focus is on events with certain characteristics as defined by their metadata.

It is important to note that the configuration of the metric determines how metadata is used in the analysis, providing flexibility in how events are counted and reported.

Statsig processes metrics based on a full day in the PST timezone. The system waits until a full day is marked as ready before processing that day's data. This means there is typically a delay of up to 24 hours for the processing of metrics.

This delay is inherent to Statsig's data processing system and is not related to the tier of your plan. Therefore, it cannot be reduced by moving to a higher tier plan. The processing time is designed this way to ensure the accuracy and integrity of the data.

Please note that this delay is a standard part of Statsig's data processing and is not indicative of any issues with the system. It is important to factor this delay into any data analysis or decision-making processes that rely on Statsig metrics.

The current functionality of the "contains" operator allows for only one target in the condition. If you want to filter on multiple tags, you will need to add more events to your configuration. This can be done by using the "+ Add Another Event" link.

Here's how you can do it:

1. Use the "Filter" option in the Custom Query fields for a metric. 2. Add an individual filter for each tag you want to include in your analysis. 3. To add multiple values in one filter, use the "+ Add Another Event" link to add more events to your configuration.

Please note that the current documentation does not provide a method for adding multiple values in one filter in a more streamlined way. If you need further assistance, please reach out to the support team.

When a vulnerability alert is triggered in Snyk for the Statsig Ruby SDK due to the MPL-2.0 license, it is important to understand the implications of the license and how it may affect your organization. The Mozilla Public License 2.0 (MPL-2.0) is a widely accepted open-source license that allows the code to be used in both open-source and proprietary projects.

Under MPL-2.0, if modifications are made to the original code, those changes must be disclosed when the software is distributed. However, the MPL-2.0 license permits the combination of the licensed code with proprietary code, which means that the SDK can be used in closed-source applications without requiring the entire application to be open-sourced.

It is important to note that while the MPL-2.0 license is generally compatible with other licenses, it may be flagged by security tools like Snyk because it requires review and understanding of its terms. Organizations should consult with their legal team or open-source compliance experts to ensure that the use of MPL-2.0 licensed software aligns with their policies and legal obligations.

If your organization has made no modifications to the original code, there should be no concerns regarding the need to disclose changes. Ultimately, the decision to use software under the MPL-2.0 license should be made by the decision-makers within the organization after careful consideration of the license terms and compliance requirements.

In Statsig, you can manually change the default value of a parameter in a layer. This can be done from the layer's homepage. Here's how:

1. Navigate to the layer's homepage.

2. Locate the parameter whose default value you want to change.

3. Update the default value as needed.

Please note that parameters in a layer are designed to exist across multiple experiments in the layer. Therefore, the layer's homepage is the place where they live.

Remember, the ability to change the default value of a parameter in a layer allows you to manage your Statsig integration code more efficiently. It's one of the benefits of using default parameters in Statsig.

If you're looking for a way to change the default value at the level of an experiment in the layer, where the parameter is first defined, please note that this is not currently possible. The parameters are defined at the layer level, not at the experiment level.

We hope this information helps you in managing your parameters in Statsig. If you have any more questions, feel free to reach out.

In scenarios where there is a need to check feature flags for entities rather than individual users within the Statsig SDK, a workaround can be implemented using dynamic configs.

Dynamic configs in Statsig allow for the evaluation of different values for different IDs, which can be used as a substitute for feature flags by substituting user-specific checks with entity-specific checks. While dynamic configs do not support pulse, they share a common definition and evaluation method with feature flags, returning a JSON value instead of a boolean at the end of the evaluation process.

This approach can be particularly useful when dealing with long-lived sessions or when there is a need to frequently update the cache to reflect the latest configurations.

It is important to note that implementing a new SDK to handle this use case would be a more substantial undertaking, and utilizing dynamic configs as described may provide a viable solution without the need for such an investment.

When an experiment decision has been made and there is a need to continue collecting metrics without resetting user assignments, the platform does not offer a native solution to directly address this scenario.

Once a decision is made on an experiment, the platform does not support undoing that decision. The 'Reset' option is available; however, it will revert the experiment to an unstarted state, and all users will receive the default experience. If the experiment is reset or a new experiment is started, users will be randomly assigned to a group, which may differ from their original assignment.

It is important to note that resetting an experiment will not preserve the initial user group assignments, which may be critical for long-term metric collection and analysis. To monitor the performance of long-term metrics after an experiment has ended, the Metrics Explorer can be utilized. While it does not provide confidence intervals and p-values, it does offer absolute numbers for each metric across all groups in the experiment.

For future experiments, it is recommended to enable Reviews in the project settings to provide better oversight and prevent similar situations. Reviews offer guardrails that can help in managing changes that impact production environments.

To create a custom metric to track the sum of your "Donation" event values in Statsig, follow these steps:

1. Navigate to Metrics from the left-hand navigation panel, then to the Metrics Catalog tab. Click on the Create button.

2. In the Metric Type, select "Aggregation". This type allows you to compute the Sum or Average of the value or metadata property of an event type.

3. For the Event Type, select "Donation".

4. For the ID Type, select "userID" or "stableID" depending on your use case.

5. For the Aggregation Type, select "Sum".

6. For the Field, select "value". This will sum up the values you logged with your "Donation" events.

7. You can also add filters if you want to limit this metric to specific conditions, such as a certain currency.

8. Finally, give your metric a name and save it.

Please note that after you create a custom metric, it will not populate until the next day and will not backfill to previous days. Statsig will only calculate it moving forward from the creation date.

To edit the returned JSON for a dynamic config after adding a targeting rule, locate and click on the 'Edit' link adjacent to the 'Returns JSON' text. This action will bring up the JSON configuration editor.

Within this editor, you can freely modify the configuration parameters and their corresponding values as required by your application. After making the necessary changes, ensure to save your updates by clicking on the 'Confirm' button.

This process allows you to tailor the dynamic config to suit the specific needs of your application, even after targeting rules have been applied. If you encounter any difficulties or the editor does not load as expected, please reach out for further assistance.

Enabling a metrics ingestion pipeline in Statsig involves a series of steps to ensure that your data is correctly integrated and utilized for analysis.

Begin by connecting your data warehouse to Statsig, which will require read access to metric and exposure data, write access for results and exposures, and the ability to run jobs and query data. Once connected, define your metric and assignment sources according to Statsig's expected schemas to map the data accurately.

Metrics can then be ingested through various mechanisms, including Server and Client SDKs, HTTP API, Integrations, Data Warehouse Ingestion, or the Data Warehouse Native Solution for enterprise contracts.

If you encounter any issues during the ingestion process, utilize the Metrics Dashboard in Statsig for debugging. After successful data import, you can proceed with feature gate or experiment implementation.

For detailed guidance on connecting your warehouse and defining sources, refer to the Statsig documentation.

Ensuring consistent experiment results across Client-Side Rendered (CSR) and Server-Side Rendered (SSR) pages when using Statsig involves establishing a single source of truth for user assignments. The challenge arises when independent network requests on different pages result in varying initialization times, potentially leading to different experiment variants being presented to the user.

To address this, it is recommended to have one consistent source of truth for the user's assignment. This can be achieved by using the assignments from the first page (CSR) and carrying them over to the second page (SSR), rather than initializing a new request on the second page.

This approach ensures that the user experiences a consistent variant throughout their session, regardless of the page they navigate to. It is important to note that the user's decision for any getExperiment call is determined at the time of initialization, not when getExperiment is called. The getExperiment call simply returns the value fetched during initialization and logs the user's participation in the experiment and their assigned group.

To implement this, developers should ensure that the assignments are fetched from the network request on the first page and then passed to the second page, avoiding the need for another initialization request and the management of an ever-growing list of parameters.

When utilizing the HTTP API to log events in real-time, it is crucial to ensure that the events are properly formatted to be recognized by the system. If events are not appearing in the Metrics Logstream, it may be necessary to wrap the events in an array using the events:[] syntax. This is particularly important because events are typically sent in bulk, and the correct formatting is essential for them to be processed and displayed in the stream.

Additionally, it is recommended to use the SDKs provided for various languages, as they handle a lot of the heavy lifting, including performance and reliability improvements. For cases where the SDKs are not an option, such as with certain desktop applications or specific frameworks like Qt, it may be possible to use language bindings to integrate with the SDKs of supported languages. This can potentially save significant development effort and ensure more reliable event logging.

Furthermore, for the best practice regarding experiments and configurations, it is advised to fetch the experiment config at the time when it is needed to ensure accurate exposure logging. The SDKs typically fetch all evaluations for a single user in one shot and cache it for the session duration, allowing for local and instantaneous experiment config or gate checks. If using the HTTP API directly, it is possible to use an 'initialize' endpoint, which is typically reserved for SDK use.

This endpoint fetches all evaluated results for a given user without generating exposure logs at the same time, thus avoiding overexposure issues.

To ensure that employee IP ranges are excluded from all Statsig events and A/B tests, you can utilize holdouts. Holdouts are a feature that allows you to compare a group of users who are not exposed to any experiments against the broader population that is.

By creating a targeting gate with a regex that encapsulates the IP addresses of your employees, you can then use this targeting gate to establish a holdout. Setting this holdout to global will effectively exclude the specified IP ranges from any experiments, gates, or other features.

Additionally, if you maintain a list of user IDs for your employees, you can create a segment using these IDs instead of relying on IP addresses. This segment can then be referenced in the holdout to achieve the same exclusion effect.

It is important to note that holdouts are subject to analysis, which may lead to additional billing due to the insights they provide on the overall impact of experimentation efforts.

When you want to exclude users from a holdout group using a targeting gate, the users should pass the targeting gate. The targeting gate is used to define the users who are included in the holdout group. Therefore, if a user passes the targeting gate, they are included in the holdout group.

If you want to exclude certain users, you would set up your targeting gate such that these users fail the gate. This will effectively exclude them from the holdout group.

In other words, you should create a gate that qualifies users to be in the holdout. If the gate evaluates to true for a user, they will be held out or excluded.

Remember, the key is to design your targeting gate in a way that the users you want to exclude will fail the gate, thus ensuring their exclusion from the holdout group.

In the event that an experiment in Statsig has been restarted and there is a need to export user data from the previous iteration of the experiment, the process is not as straightforward as using an 'Export Report' button.

However, it is still possible to retrieve the necessary data. To do so, the user should contact Statsig support directly with a request for the data export. The Statsig team can generate the report and send it to the email address associated with the user's Statsig account and project.

It is important to note that this process is manual and requires intervention from the Statsig team.

To retrieve currently active experiments using JavaScript in the browser console, you can use the Statsig.getExperiment or Statsig.getLayer methods. However, these methods require you to know the name of the experiment or layer. There is no documented method to retrieve all active experiments without knowing their names.

For group overrides, there is no documented method to retrieve group IDs directly. However, you can use the Statsig.overrideConfig method to override the value of a specific experiment for a specific user. The value you provide should match the structure of the experiment's parameters. Here is an example:

javascript Statsig.overrideConfig("a_config_or_experiment_name", { key: "value" }, "a_user_id"); In this case, { key: "value" } should match the structure of the parameters for the experiment you're trying to override. There is no documented way to override groups directly using group IDs.

If you're unsure about the structure of the experiment's parameters or need further assistance, it is recommended to reach out to the Statsig team for more specific guidance.

Please note that it may not be possible to override the group. You can only override the config object that's returned by a getExperiment call.

In order to implement StatSig tracking on Shopify Custom Pixels, you need to ensure that the StatSig SDK is properly initialized and that events are logged correctly. Here's a step-by-step guide on how to do this:

1. First, create a script element and set its source to the StatSig SDK. Append this script to the document head.

javascript const statsigScript = document.createElement('script'); statsigScript.setAttribute('src', 'https://cdn.jsdelivr.net/npm/statsig-js/build/statsig-prod-web-sdk.min.js'); document.head.appendChild(statsigScript);

2. Next, initialize the StatSig SDK. This should be done within an asynchronous function that is called when the StatSig script has loaded.

javascript const statsigInit = new Promise(async (resolve) => {    statsigScript.onload = async () => {      await statsig.initialize("client-sdk-key", { userID: "user-id" });      resolve();    } })

3. Subscribe to the event you want to track and log it with StatSig. Make sure to wait for the StatSig SDK to initialize before logging the event.

javascript analytics.subscribe("checkout_started", async () => {    await statsigInit;    statsig.logEvent("checkout_started"); });

Remember that statsig will be the global, you want to wait for initialization before logging, no need to require.Please note that the SDK accumulates events within, and flushes periodically. Ensure that Statsig.logEvent() was ever called, and that the application didn’t exit right away after that so the SDK has enough time to flush the events.

Inspect the network requests and see what requests were made and what their responses were to verify if the events are being sent correctly.

Incorporating an app version check into Statsig experiment variants can be achieved using Feature Gates or Segments. Here are the steps you need to follow:

1. Capture the Initial App Version: The first step is to capture and store the initial app version that a user starts using. This information is crucial as it will be used to determine whether a user is new and started using the app from a specific version onwards.

2. Use Custom Fields for Targeting: The next step is to use Custom Fields for targeting. This will require some code on the client-side that passes in user upgrade/create timestamps as a custom field.

3. Pass the Initial App Version as a Custom Field Key: You need to pass the value for users who started using the app as new users from a specific version as the Custom Field Key.

4. Configure the Custom Field Key: Once you create the key, configure it using the "Greater than or equal to version" operator. This operator checks the current version of the user's app.

For your specific case, you can create two separate experiments or feature gates. One for users on app version 1, where the variants are v1 and v2, and another for users on app version 2, where the variants are v1 to v4. You can then use the app version as a custom field for targeting.

Please note that the Key set in the Custom Field will be included in the events called by Feature Gates. However, it's not a chargeable event. It's just attributes that will be in the payload for events you're already tracking.

Integrating the Statsig SDK server-side in a Next.js application without a custom server or Vercel deployment can be achieved by utilizing the register API to initialize the SDK on server startup.

This method allows for the SDK to be set up before serving requests, ensuring that feature flags and configurations are available for server-side rendering. However, it is important to note that the register API may only trigger on the first request received by the server.

To ensure the SDK is initialized, a health check or similar request may be necessary to activate the register process.

Additionally, depending on the hosting solution for the Next.js app, writing a data adapter to read Statsig configurations from a cache could be a viable alternative. This approach would involve creating a custom adapter that interfaces with a caching mechanism to retrieve the necessary configurations for Statsig.

For more detailed guidance on implementing feature flags with Next.js, refer to the official Statsig documentation.

In Statsig, feature gates are designed to control access to features based on user attributes or conditions, not on the number of times a user has accessed a feature. However, you can achieve the desired functionality by implementing a custom solution in your application code.

To limit the number of times a user can pass through a feature gate based on a count, you would need to maintain the state and count the times a user has accessed a feature. This can be done by incrementing a count stored in local storage or any other suitable storage system in your application.

Once you have the count, you can use a Custom Field targeting rule on the Gate that uses a numeric comparison operator. This rule should be set up to compare the stored count with the desired limit. When initializing the SDK, pass your stored count into the user object.

Please note that this approach requires you to manage the count state in your own system, as Statsig does not provide built-in support for this type of use case.

In Statsig, there is no built-in feature to make gate evaluations sticky for users. However, developers can achieve a similar effect by storing the result of a gate evaluation and then applying local overrides based on the stored result.

This method allows the gate value to remain consistent for the user, even if the conditions that originally allowed the user to pass the gate have since changed. To implement local overrides, developers can refer to the Statsig documentation on local overrides for guidance on how to set this up within their application.

The documentation provides the necessary steps and code snippets to help developers override gate values locally, ensuring that the user experience remains consistent as intended.

When optimizing landing page loading without waiting for experiment configurations, it is recommended to use a custom script approach if you need to pass in specific user identifiers, such as Segment's anonymous ID.

This is because the standard landing page script provided by Statsig does not allow for the initialization with a different user ID. Instead, it automatically generates a stableID for the user, which is used for traffic splitting and consistent user experience upon revisits. This stableID is stored in a cookie and is used by Statsig to identify users in experiments.

However, if you need to synchronize with metrics from Segment using the Segment anonymous ID, you may need to deconstruct the landing page tool and implement a custom version that allows you to set the userObject with your Segment anonymous ID.

Additionally, you can enrich the Segment User with an additional identifier, such as statsigStableID, which can be obtained in JavaScript using statsig.getStableID(). This ID can then be mapped from the Segment event payload to Statsig's stableID. If performance is a concern, you can bootstrap your SDK with values so you don't have to wait for the network request to complete before rendering.

This can help mitigate performance issues related to waiting for experiment configurations. For more information on bootstrapping the SDK, you can refer to the official documentation on bootstrapping.

Yes, it is possible to preserve the Statsig Stable ID across different platforms such as mWeb and mobileApp. The Stable ID is generated for anonymous use cases and is persisted in local storage. For signed-in users, consistency is ensured across different surfaces, such as mobile app vs desktop, as long as there's a userID used across them.

If you're experiencing issues with StableID changing across subdomains or platforms, one possible solution is to store the StableID in a cookie and reuse it. You can store and retrieve your own id and use the overrideStableID method on the SDK to send it to Statsig. Your overridden id will always take preference. This way, you can set the StableID for not logged in users and your in-house id for people that are logged in.

Here's an example of how you can override the StableID:

javascript await Statsig.initialize(  "client-sdk-key",  null,  {    overrideStableID: '<passed via cookie or via db lookup>'  } );

This should ensure that both your marketing site and main app use the same StableID, allowing you to run experiments that span both sites.

To reconfigure or disable the Slack integration in the Statsig Console, you must manage the integration directly from Slack. There is no option within the Statsig Console to disable the Slack integration.

To reset the channel for notifications, you can attempt to click 'Add to Slack' again, which should allow you to reconfigure the settings. If 'Add to Slack' is not clickable, or if you encounter issues with event filtering not working correctly, you may need to remove the Statsig Slack App from your Slack workspace and then re-add it.

This can be done through the Slack app management area. Once removed, you can re-add the integration via the Statsig Console to set up the desired notifications and configurations.

If you are experiencing issues with resetting your password or need to delete your Statsig account, follow the steps below for resolution. First, if you are not receiving the password reset email, please attempt the reset process once more.

There may have been a temporary issue that has since been resolved. If the problem persists, contact support for further assistance. In the event that you wish to delete your Statsig account, you should navigate to the Project Settings. Within the Members & Roles section, you will find the option to leave the project.

It is important to note that once all members have left the project, the project and associated organization/account will be automatically deleted. This process ensures that all user data is removed in compliance with privacy standards.

If you're experiencing a continuous network error while running Statsig 0.23.0 in a Python 3.9 environment, it's important to first understand the environment in which you're running the Statsig SDK and the version you're using.

If the error is occurring locally and continuously, the Statsig SDK will keep retrying and returning the same error. To debug this, you can send a curl command to our events API from your local environment. Here's an example of how to do this:

bash curl  -X POST \  'https://events.statsigapi.net/v1/log_event' \  --header 'statsig-api-key: <YOUR SDK KEY>' \  --header 'Content-Type: application/json' \  --data-raw '{"events": [{"user": { "userID": "1" }, "time": 1667512336, "eventName": "test_event"}]}'

This command sends a test event to the Statsig events API. If this command works, it means your local environment can successfully send requests to the Statsig API. If it doesn't, you may need to check your local network settings, such as DNS or ad-blockers, which could be blocking requests to the API.

Remember, the Statsig SDK is designed to retry sending events if errors are encountered. If you're still experiencing issues after trying the above steps, please provide more details about your environment and the error you're encountering for further assistance.

Retrieving all experiment values for a specific user in a single server-side call can be achieved using the Statsig server SDKs.

The SDK allows for the evaluation of experiments on the fly for a given user without incurring a network roundtrip. This is particularly useful for server-side rendering scenarios, such as with Next.js, where you may have both server-side rendered (SSR) and client-side rendered (CSR) pages and wish to avoid wrapping your entire application with a CSR provider.

By using the server SDK, you can download the entire configuration and evaluate experiments for each user server-side. This approach scales well as it does not require a network call for each experiment evaluation, ensuring efficient performance even as user traffic increases.

For more information on how to implement this in your application, you can refer to the Node.js server SDK documentation provided by Statsig.

To roll out and monitor multiple new features simultaneously using Statsig, you can utilize the platform's Feature Gates and Experiments.

For each new feature you plan to roll out, create a corresponding Feature Gate. This approach automatically converts a feature roll-out into an A/B test, allowing you to measure the impact of the roll-out on all your product and business metrics as the roll out proceeds.

If you wish to test hypotheses between product variants, create an Experiment. An Experiment can offer multiple variants and returns a JSON config to help you configure your app experience based on the group that the user is assigned to.

To show features randomly to different sets of customers, use the 'Pass%' in a Feature Gate and 'Allocation%' in an Experiment. This allows you to control the percentage of users who see each feature.

Statsig's Experiments offer more capabilities for advanced experiment designs. For example, you can analyze variants using stable IDs for situations when users have not yet signed-up (or signed-in), or using custom IDs to analyze user groups, pages, sessions, workspaces, cities, and so on. You can also run multiple isolated experiments concurrently.

Remember to define your company goals and key performance indicators (KPIs) to measure the success of your features. You can break down these strategic goals into actionable metrics that you can move with incremental, iterative improvements.

If you use three different feature gates, you will find out how each feature, individually, performed against the baseline. If you want combinatorial impact analysis, like, A vs B vs C vs AB vs BC vs AC vs ABC, then you will need to setup an experiment with 7 variants and specify the combinations via parameters and measure.

However, in practice, this level of combinatorial testing isn’t always fruitful and will consume a lot of time. A pragmatic recommendation would be to use feature gates to individually launch and measure the impact of a single feature, launch the ones that improve metrics and wind down the ones that don’t.

When running experiments with both anonymous and logged-in users using Segment and Statsig, it is crucial to have a common identifier to track user interactions accurately.

For anonymous users, you can either pass the stable ID generated by Statsig to Segment or generate a stable ID and pass it to Statsig. This ensures that there is a consistent identifier across both platforms.

For logged-in users, you can use the Segment userID, which should be included in the userObject during initialization. This setup allows Statsig to recognize both anonymous and logged-in users in the experiment. To facilitate this, you may need to provision a new custom ID, such as segmentAnonymousId, and map Segment's anonymousId to this new custom ID in the integration settings.

This approach ensures that exposure events sent via the Statsig SDK and custom events sent via Segment are associated with the same user, regardless of their logged-in status.

To schedule a feature gate roll out based on the environment in Statsig, you need to specify the environment during initialization and in your feature gate rules.

During initialization, set the environment tier as follows:

javascript options = {  environment: { tier: `${env}` }, }

After setting the environment tier, you need to specify the environment in your feature gate rules. Follow these steps:

1. Navigate to the Statsig Console and create or edit a Feature Gate.

2. When creating a rule for your Feature Gate, check the "Specify Environments" checkbox.

3. Choose the environments (like 'development' or 'staging') you want your rule to be enabled for.

4. Save your changes.

By default, unless otherwise specified, a rule is enabled for ALL environments. Therefore, it's crucial to specify the environment in your feature gate rules.

If your feature gate is not evaluating as expected, double-check the environment value you're passing during initialization. Ensure it matches exactly with the environment specified in your feature gate rules.

Remember, the order of rules in the Statsig console matters. If you have multiple rules, the first rule that matches the conditions will be applied. Therefore, if you have a specific rule for a certain environment, make sure it is placed before any general rules that could also apply.

In order to notify users before they are exposed to a new feature, you can create a separate feature gate to control the rollout percentage and a segment to contain the account IDs that can be exposed to the feature. This approach allows you to effectively manage the rollout process and ensure that users are notified in advance.

The main feature gate will only pass for the accounts in the segment that also pass the separate feature gate. This provides a clear distinction between the users who are eligible and those who have been exposed to the feature.

Here's a brief overview of the process:

1. Create a main feature gate (rollout_feature_gate). The users that pass this gate will be exposed to the feature.

2. Create a separate feature gate (exposure_eligibility_gate) to control the rollout percentage. The users that pass this gate are the ones eligible to be exposed to the feature.

3. Create a segment (allegeable_accounts) that contains all the account IDs that can be exposed to the feature.

The rollout_feature_gate will return pass only for the accounts in the allegeable_accounts segment, while questioning the exposure_eligibility_gate. After a certain amount of time, export all account IDs in the exposure_eligibility_gate to the allegeable_accounts segments, and increase the exposure_eligibility_gate percentage.

This approach allows you to have a distinction between the eligible users that were exposed to the feature (allegeable_accounts segment) to the ones that are allegeable but potentially not yet exposed to it (exposure_eligibility_gate). You can manage additional rules and environment conditions in the main feature gate (rollout_feature_gate).

Remember to test your setup thoroughly in a pre-production environment before rolling it out to ensure everything works as expected.

When setting up the Statsig SDK in an Electron/Vite(React) app, it's important to note that no user properties are required. You do not need to pass a userID or an email. If you are just getting the SDK setup and running, an empty object will suffice for now.

If you're using React in your application, the React SDK would be a good fit. However, if you're not using React, the JS SDK will be the best choice. Even if your application is running on Electron, using the React SDK should not pose any issues.

In case you encounter issues with the analytics onboarding page, such as a blank page or undefined keys, please report it to our support team. We strive to resolve such issues as quickly as possible.

To display an indicator on your UI for features that are enabled only for specific segments, such as 'internal beta testers', you can utilize the Statsig Console API. This API allows you to programmatically retrieve detailed information about the rules associated with a specific feature gate. By using the Console API, you can determine which segments are currently receiving traffic for a given feature and display an appropriate indicator in your UI to reflect the rollout status to your beta testers or other user segments.

It is important to note that this approach requires special authorization via a new token, as it involves accessing potentially sensitive configuration data. The Console API is designed to be separate from the core SDK to ensure security and prevent unauthorized users from manipulating feature access.

For more information on how to use the Statsig Console API to fetch gate configurations, please refer to the official documentation provided in the Statsig Console API guide.

To start a new version of an existing experiment in Statsig, follow the steps below:

1. Log into the Statsig console.

2. Once logged in, navigate to the Experiments section. You can find this in the left-hand navigation panel.

3. Click on the Create button to start a new experiment.

4. Enter the name and description for your new experiment. Remember, the name of the experiment should be unique. If you're creating a new version of an existing experiment, consider including a version number or date in the name to distinguish it from the original.

5. Configure your experiment's allocation, targeting criteria, and groups and parameters. This step is crucial to ensure your experiment runs as expected.

6. Specify your hypothesis and add the Primary and Secondary metrics for your experiment. These metrics will help you measure the success of your experiment.

7. After all the details are filled in, press Save and Analyze to start the experiment. You'll be prompted to finalize the dates, experiment status, and optionally set up a schedule to reload experiment results.

By following these steps, you can easily start a new version of an existing experiment in Statsig.

To test whether a metric based on a custom event is tracked in non-production environments, you can follow a specific process. While non-production events are visible in diagnostics, they are not included in Pulse results.

To include these events, you can log them as regular events. It is important to note that non-production data is intentionally not included in metrics to prevent non-production test data from being included in the metrics. When testing your event logging, you can suffix your events with '_test' and use the production environment. Once you are ready to finalize your event logging, you can remove these suffixes.

If you encounter issues where the Metrics Logstream is not visible, it may indicate a connection or schema issue, and Statsig is not receiving your precomputed metrics. In such cases, you should reach out for assistance, and a team member will provide support.

If you encounter an error message stating This cohort has failed to sync because the API key that was provided no longer has access to the partner. Please check the access level on the partner's website concerning this integration. while integrating Amplitude with Statsig, it suggests that there might be an issue with the Server Secret Key you're using.

Here are a few steps you can take to troubleshoot this:

1. Double-check the Server Secret Key: Ensure that you've copied the correct Server Secret Key from the API keys page in your Statsig Project Settings.

2. Check the key's access level: Make sure that the Server Secret Key you're using has the appropriate access level for the integration.

3. Re-enter the key: Try re-entering the Server Secret Key in Amplitude and save the changes.

If you've checked these and the issue persists, it might be a more complex issue. In this case, it is recommended to reach out to the Statsig support team for further assistance.

In TypeScript React-Native, the get<T> method in the DynamicConfig typing class is used to retrieve a specific parameter from a Dynamic Config. Here's how to use it:

1. **Passing the Key**: The key you should pass in is the name of the parameter you want to retrieve from the Dynamic Config. This parameter is defined in the Statsig Console when you create or edit a Dynamic Config.

2. **Understanding the Return Value**: The get method will return the defaultValue you provide if the parameter name does not exist in the Dynamic Config object. This can happen if there's a typo in the parameter name, or if the client is offline and the value has not been cached. If you're always getting the defaultValue, it's likely that the parameter name doesn't match what's in the Dynamic Config, or the client hasn't successfully fetched the latest config values. Please double-check your parameter names and your network status. If the issue persists, it might be a good idea to log this issue for further debugging.

3. **Retrieving the Entire Object**: If you want to get the entire object, you can use the getValue() method. For example, if you call config.getValue(), you will get the entire object.

4. **Example Usage**: The argument passed into getConfig will be the dynamic config key itself which returns a config object that implements the get method. You pass a top-level object key into that get method. For instance, if the top level key is clothing, you would pass that into the get method accordingly, like so: statsig.getConfig('max_discount').get('clothing');

Remember, the get<T> method is designed to access individual properties of the dynamic config, not the entire object. For accessing the entire object, use the getValue() method.

To view statistics for multiple landing pages separately in Statsig when analyzing a test, you need to set up separate experiments for each landing page. This involves adding a unique script to each landing page you're testing. The script should look like this:

<script src="https://cdn.jsdelivr.net/npm/statsig-landing-page-exp?apikey=[API_KEY]&expid=[EXP_NAME]"></script>

Replace [API_KEY] with your Client API Key and [EXP_NAME] with the unique Experiment Name for each landing page.

Once you've set up and published your landing pages, you can monitor the setup by navigating to the Diagnostics Tab in your experiment and looking at the 'Exposure Stream' at the bottom of the page. This shows a real-time stream of the page loads along with the variant they were allocated.To interpret the results, go to the Pulse Results tab in the Experiment page. Here, you can add metrics you want to monitor and verify which variant is doing better.

Remember, you can extend this setup to run ABC or ABn tests by adding more variants in the Experiment Setup tab. If you want to run multiple landing page experiments without needing to update the code on the website for each experiment, consider using layers.

Additionally, you can find group level stats in the hover card. Just hover over the metric and click "view details".

In the context of event-based metrics, particularly the event_dau metric, it is important to understand how users are counted over multiple days. The event_dau metric is designed to count the number of distinct users or devices that have triggered a specific event on a given day.

This means that if an individual user triggers the same event on consecutive days, they will be counted as a unique user for each day that the event is triggered. The metric is focused on daily activity, and as such, it does not aggregate users over multiple days but rather provides a snapshot of unique interactions for each day separately.

Yes, it is possible to view the scorecard metrics of an experiment for a custom date range that starts after the experiment start date. This can be achieved by creating an Explore report with a specific date range. The report will filter the events and metrics to be within that range. However, the users included in the report will also encompass those exposed before the specified timeframe. This is designed to capture the downstream effects of the experiment treatment.

Here are the steps to do this:

1. Navigate to the Explore tab.

2. Fire off a custom query with your desired date range.

If you're having trouble selecting a date range that starts after the experiment start date, it might be a technical issue. Check your inputs and try again. If the issue persists, please flag it for assistance.

Currently, it is possible to add custom criteria for feature gating. You can create user targeting rules based on user attributes such as email and user ID, and additional ID types can be added in project settings. These can be used for all new experiments, allowing control over the experience a test user sees.

However, there isn't a feature to set these custom criteria as default options for a project. The ability to add these as default criteria options with a dropdown menu is not mentioned in the current documentation.

That being said, we are working on Templating features, and any default configuration criteria, including custom field defaults, are being considered. This includes the possibility of having default custom config for metrics so when the feature gate is created, the custom default metrics are selected/not selected.

In terms of the level at which these metrics are custom to, we are considering project-level defaults. This would be particularly useful for organizations with multiple teams utilizing different repositories, allowing for defaults to be set based on the specific project.

Please note that these features are currently in the scoping phase, and we are actively seeking input on feature requirements.

When utilizing the checkGate function within Statsig, it is imperative to provide a userID. This identifier is crucial as it ensures a consistent experience for each user. The userID is a mandatory field, as it is utilized to maintain uniformity in user experiences and is essential for the randomization process during partial rollouts.

It is important to note that even if an email is provided as a top-level field on StatsigUser, it does not substitute the requirement for a userID. In instances where there is no logged-in user, a stable identifier should be established, which could be stored as a cookie or in local storage, and used consistently with each call to Statsig.

This practice is vital for the system to function correctly and provide reliable results.

When considering the removal of preexperimental bias from a feature that was rolled out with limited initial data, resalting the feature gate is not specifically recommended for this purpose.

Pre-experimental bias can occur by chance and may not always significantly impact the results of an experiment. If the overall difference in metrics due to this bias is small, it might not meaningfully affect the interpretation of the results.

In such instances, the bias can be acknowledged as a cautionary note, and the affected metrics should be treated with a degree of skepticism. To address concerns about pre-experimental bias, it is advisable to utilize the 'Days Since Exposure' view.

This tool assists in identifying any novelty effects and pre-existing experiment effects. Should bias be detected, users will receive a notification, and a warning will be displayed on the relevant Pulse results. This approach allows for a more informed analysis of the feature's impact without the need to resalt the feature gate.

Statbot is specifically trained on the Statsig documentation. It does not have access to the codebase. The bot is designed to provide accurate and reliable information based on the documentation. It's important to note that the documentation may lag behind the codebase.

In addition to the documentation, Statbot is also constantly learning from the answers provided by humans in public channels. This allows it to continually improve its responses and provide more accurate and specific coding recommendations. However, it's crucial to remember that the bot's knowledge is based on the documentation and human-provided answers, not the codebase itself.

The Pro tier is structured on a usage-based pricing model, specifically focusing on the volume of events. It is not a flat rate per user but rather a cost that is determined by the number of events generated. The Pro tier is priced at $150 per month for up to 5 million events.

Beyond this threshold, additional events are billed at a rate of $50 per million events. Metered events are the key factor in this pricing structure and include any instance where a user is assigned to a feature gate or experiment, which is recorded as an 'exposure event.' These exposure events are deduplicated within certain time windows to ensure accurate billing For client-side SDKs, the deduplication window is one hour, and for server-side SDKs, it is one minute.

Additionally, custom events logged through the SDK, inbound integrations, or data warehouse rows are also counted as metered events.

At present, we do not provide a downloadable JSON version of our OpenAPI/Swagger documentation for the Console API. However, we appreciate your suggestion and have added it to our backlog for consideration. We will prioritize this feature request along with the rest. Please stay tuned for updates on this feature.

In Statsig, the "Days since exposure" graph in the View Details section can be utilized to understand if an experiment is pulling forward an event. This graph displays the cumulative lift of a metric over time since a user's first exposure to an experiment. If you observe a significant lift immediately after exposure that diminishes over time, this could suggest that the experiment is pulling forward events.

However, it's important to note that there isn't a built-in tool specifically designed to identify pulled forward events. The closest feature available is a toggle in the top right of the scorecard that allows you to select “Days Since Exposure” instead of “Cumulative”. This feature enables you to look across all metrics inline.

Remember, the interpretation of the "Days since exposure" graph and the use of the toggle feature require a good understanding of your metrics and the potential impact of your experiments.

When transitioning from event import to metric import, it is essential to understand the ingestion status and the steps required to archive metrics that are no longer needed. After switching to metric import, if you have precomputed metrics, ingesting these metrics directly can be more efficient than importing raw events.

Once the change is made, there may be a need for adjustments on the backend to accommodate the new import method. This initial setup is a one-time speedbump, and once resolved, metrics should start appearing as expected. If you notice that some metrics have complete data for a specific date range while others do not, it may be due to the ingestion process still being underway or a need for a backfill to import historical data.

For metrics generated from the previous event ingestion job, such as event_count and event_dau types, consider archiving them if they are no longer relevant to your current data structure and needs. Archiving these metrics can help maintain a clean and efficient data environment.

In Statsig, the Daily Participation Rate and One Time Event metrics are used to track user behavior in experiments. The choice between these two metrics depends on the nature of the event you're tracking.

1. Daily Participation Rate: This metric is calculated as the total number of days that a user has the selected event, divided by the number of days the user is in the experiment. This is done for each user in the experiment. The mean event_dau, or the average active days per user, is then calculated by aggregating this average event_dau for each user in the experiment, with each user weighted equally. This metric is more suitable for events that are expected to occur repeatedly for a given user.

2. One Time Event: This metric is ideal for events that are only expected once per user, such as booking events. If the event is expected to occur only once per user during the experiment or holdout period, then the One Time Event metric would be suitable.

For longer experiments and holdouts, the choice of metric would still depend on the frequency of the event. If the event is expected to occur approximately once a month or less frequently, the One Time Event metric should be appropriate. However, if the event is expected to occur approximately weekly or more frequently, the Daily Participation Rate metric might be more appropriate as it captures recurring behavior.

When reviewing experiments, consider all related metrics:

- One-time events best capture the number of unique users who participated in the event. - Daily participation rate is an effective proxy for "how much" people are participating in the event. - Total events (event_count) is a better proxy for revenue or downstream metrics.

For holdouts, it can be helpful to use different rollups. For example, looking at one-time metrics for the 7-day or 28-day rollup would tell you what % of users participated (at all) within the last 7-day or 28-day window. This can be an effective way to get past the history issue.

The statsig::app_error event is an automatic log created by the Statsig SDK when it encounters an uncaught exception on your webpage. This event is not necessarily related to your Statsig implementation, but rather, it is associated with any error on your webpage.

The statsig::app_error event can be particularly useful when you navigate to the Event Logstream and inspect the details. Depending on the nature of the error, the SDK may be able to capture the line and error message.

It's important to note that the occurrence of this event on iOS devices could be due to a specific condition or issue unique to the iOS environment. However, without more specific information, it's difficult to provide a definitive answer.

For further assistance with this issue, it is recommended to reach out to the Statsig team directly. They may be able to provide more insight into this specific event and why it's being triggered on iOS devices.

When conducting an experiment with Statsig, you may encounter a warning stating 'Event metrics do not have data.' This warning is designed to ensure that there are participants in your gate or experiment who have also had some event logged for them. It serves as a validation that the users in the experiment have corresponding metrics that exist for them.

This warning can be triggered during the initial stages of an experiment, especially when all components of Statsig, such as the SDK and experiment configurations, are deployed simultaneously. It is important to note that there is a waiting period after the gate gets modified, and it may take some time for the system to recognize that users have been exposed to the experiment and have logged events.

The warning is expected to resolve once the system detects overlap between users in the experiment and the logged events, which can take up to an hour after the events have been logged.

For most metrics, computation occurs on a daily basis, and results will be available the following day. However, for 'event_count' type metrics, leading indicators can be computed and may be visible in about an hour.

If you encounter this warning, it is advisable to wait for the system to update and check if the warning persists before taking further action.

When utilizing a client SDK that involves experiments with targeting gates based on dynamic user attributes, there is a mechanism to handle changes in user eligibility. If a user's attributes alter in such a way that they no longer meet the experiment's targeting criteria, the client SDK can be configured to store the initial evaluation results in local storage.

This ensures that the user's experience remains consistent, and they continue to have access to the feature or release associated with the experiment, despite any subsequent changes in their dynamic attributes.

This approach is beneficial for maintaining a stable user experience and ensuring that the integrity of the experiment is not compromised by fluctuating user states.

When integrating the Statsig Node.js SDK, developers often inquire about the initialization time required on the server. The initialization process involves a network request, and the time taken for this process can vary based on several factors.

A key consideration is the geographical distance between the server where the SDK is being initialized and Statsig's data centers. Typically, the server latency for handling initialize calls is in the range of 200 milliseconds to 1 second.

It is important to note that project size can influence initialization time, particularly when using large ID list based segments, which require downloading during the initialization phase. Without such segments, the aforementioned latency range is a reasonable expectation for initialization time.

Statsig currently supports metrics calculation for a single production environment. If you have multiple environments set up, you will only be able to use one as your production environment for metrics and experiments.

When running an experiment, the default behavior is that the experiment will run in all environments. This means that if you try to access the config values of the experiment, you will get a valid value, even if you are in a non-production environment.

If you want your experiment to only run on production, you can set a targeting gate. This will ensure that only users in the production environment will pass the gate and be included in the experiment.

Here is an example of how you might access the config values of an experiment:

javascript useExperiment(XXX).config.get(YYY, false);

In this example, XXX is the experiment you are running, and YYY is the config value you are trying to access. If the experiment is not running in the current environment, you will get the default fallback value, which is false in this case.

Remember, once you start an experiment, it will run in all environments unless you set a targeting gate to restrict it to specific environments.

The best practice is to use the experiment checklist and diagnostics tab to instrument the test, enable it in lower environments, and validate that exposures are flowing through. Then, when you’ve validated these, you click “Start” to go to production. This workflow is typically adequate for most users.

Please note that this is subject to change as Statsig continues to evolve and add new features. Always refer to the official Statsig documentation for the most up-to-date information.

In the context of Pulse results, the cutoff time for data pertaining to the previous day is established at 11:59 PM Pacific Standard Time (PST).

This timeframe ensures that all data collected falls within the standard daily cycle, which begins at 12:00 AM PST and concludes at 11:59 PM PST of the same day. It is important for users to align their data analysis and reporting activities with this cutoff to accurately reflect the results for each individual day.

In the context of Bayesian testing, the credible interval is the Bayesian equivalent of the confidence interval. Specifically, it is reported for the conversation delta for your conversion rate.

As for the interpretation of expected loss, it is computed as the expected value of delta when delta is negative, multiplied by the probability of delta being negative. This can be loosely interpreted as the risk to your metric if you launch the test group.

The waitForInitialization option in Statsig's React SDK is used to delay the rendering of your components until the SDK has finished initializing. This ensures that the correct feature gate values are available when your components are rendered. If you're seeing users in the "uninitialized" group, it could mean that the SDK initialization is not yet complete when the feature gate check is being made. This could be due to a slow network connection or other issues that delay the initialization process.

To resolve this, you could consider increasing the initTimeoutMS value, which gives the SDK more time to complete the network roundtrip before falling back to cached values. You could also use the init callback for debugging to check when it's returning. If the issue persists, it might be worth considering using a server SDK on one of your servers to bootstrap with. This way, if you already have a network request serving stuff to your clients, you can have it evaluate the user and pass those values back without needing a roundtrip to the SDK.

Remember, the initCalled: true value doesn't necessarily mean the initialization succeeded. It's important to check for any errors thrown from the initialization method. If you're trying to avoid unnecessary updateUser calls, consider building a statsigUser and only call for update if the local statsigUser is different from the one saved in the SDK instance.

If you set waitForInitialization off, you should get the uninitialized check, and then once SDK initialization completes (within 3 seconds), you should get another check with the actual value (assuming the network request was successful for initialization).

Statsig is working on the metadata around these cases to make it easier to debug. In new SDK versions, it will be possible to differentiate them a bit more. There are also plans to make a change to React that won't even render the children until initialization has at least started so there won't be uninitialized checks at first. This is due to the ordering of effects, where SDK hooks will run before the SDK initialization path in the provider.

The current Statsig SDK does not provide a method for bulk querying experiments for a large number of users in a single call. The getExperiment function is designed to be called for individual users. If you need to get data about all experiments for a specific user ID, you can use the getExperiment function with different user IDs as many times as you want.

Please note that calling getExperiment will trigger an exposure for that user on that day. If they weren’t in a test or control group, they would be exposed and count towards experiment metrics from that day forward.

If you have a use case where you need to know the evaluation but do not want to enroll the user in an experiment group, you can use the getExperimentWithExposureLoggingDisabled API.

To determine if a user has participated in an experiment, you need to download the exposure report for each experiment to see which users have checked it. The daily reports API is an enterprise feature that can be enabled for your project.

If you're using an SDK to evaluate experiments, it should be pretty simple to just iterate through and check for each user. This is the recommended route for handling large numbers of users.

There are two main methods for performing A/B tests for static pages in Next.js using Statsig.

The first method involves using getClientInitializeResponse and storing the initializeValues in a cookie. This approach is suitable if you want to avoid generating separate static pages for each variant. However, the cookie size is limited to 4KB, so this method might not be suitable if the initializeValues are large.

The second method involves generating a separate static page for each experiment's variant. This approach is suitable if you have a small number of variants and want to avoid the cookie size limitation. However, this method might require more setup and maintenance if you have a large number of variants.

If you're unsure which method to use, you can start with the method that seems easier to implement and switch to the other method if you encounter issues.

If you're concerned about the size of initializeValues, there are a couple of ways to bring down the response size. One way is to use target apps to limit the gates/experiments/etc included in the response. Another way is to add an option to getClientInitializeResponse to specify which gates/experiments/etc to include in the response.

If you plan on stitching together multiple cookies, a simple string splice might be easier. An alternative that doesn't involve stitching together multiple initializeValues is to use multiple client SDK instances. This wouldn't be supported in react, but using the js SDK, you could have multiple statsig instances each with its own set of configs. You would have to keep track of which instance to use for which experiment but this may be a "cleaner" approach.

The JS SDK can be synchronously loaded using initializeValues similarly to how the StatsigSynchronousProvider works. So you should be able to just call statsig.initialize(..., {initializeValues}) without needing to worry about awaiting.

Finally, you can also use the local evaluation SDK to fetch the whole config before the page becomes interactive and then pass it to the synchronous SDK. This is a client SDK, but it solves the "flickering" issues because you don't need to wait for the experiment(s) data to be fetched on the fly.

The daily process that updates experiment metrics for the previous day typically completes by 9am PST. This process ensures that all data for newly added metadata dimensions in your experiment metrics are calculated and updated. Please note that the exact timing may vary, and for more specific information, you may need to consult with a Statsig team member.

If you're having trouble locating the entry point for holdouts in your dashboard, please note that it has been moved to a tab within experiments. This change might require some adjustment, but rest assured, all the functionalities remain the same. If you're unable to locate it within the experiments tab, please reach out for further assistance.

If you have any inquiries regarding billing for your account, please direct your questions to our dedicated support team. They are equipped to provide the necessary assistance or connect you with the appropriate sales representative to address your concerns.

To get in touch, send an email to the support team.

When deploying to an environment in Statsig, if you encounter an issue where the RULE is always Default and the REASON is Unrecognized, it typically means that the SDK was initialized, but the config or feature gate you're trying to evaluate did not exist in the set of values. This could be due to a few reasons:

1. The feature gate or dynamic config you're trying to evaluate does not exist or is not correctly spelled in your code. Please double-check the spelling and case-sensitivity.

2. The SDK might not have been able to fetch the latest rules from the Statsig server. This could be due to network issues or if the SDK initialization did not complete successfully.

3. If you're using a server SDK, it's possible that the SDK is outdated and doesn't recognize new types of gate conditions. In this case, upgrading the SDK might resolve the issue.

Remember, the Unrecognized reason is only given when the SDK is initialized, but the config or feature gate did not exist in the set of values.

It's also important to ensure that you are waiting for initialize to finish before making evaluations. For instance, calling checkGate inside the callback or after you are sure the callback has been triggered.

Additionally, check if the environment you've deployed to is able to make requests to the relevant URLs. In some cases, these requests might be blocked by the client in your production environment.

If you're still having trouble, please provide more details about your setup and the issue, and a Statsig team member will assist you shortly.

If you are using the synchronous provider and Server Side Rendering (SSR), the assignment reasons chart should be 100% with the reason “Bootstrap”. There should not be any “network”/“Cache”/etc. If you are seeing a lot of uninitialized values, it could point to a potential implementation issue.

One possible cause could be changes in the user object that you pass into the synchronous provider. If there are fields on the user object that you are loading asynchronously or in an effect elsewhere, it could trigger the SDK to update the values for the user.

To debug this issue, you should verify each of the render passes on the StatsigSynchronousProvider and ensure that there is only a single render with a static user object. If the user object changes and it rerenders, it could lead to the issue you are experiencing.

If you are calling useExperiment() in your code, make sure that the user object passed into the provider is static and never changes. If the user object changes after you bootstrap the StatsigSynchronousProvider, it will cause the provider to re-fetch initialize values using a network request, effectively discarding the results passed to it through SSR.

It's also important to note that if you are using SSR correctly, you should never have a “network” reason. If you are seeing a “network” reason, it could indicate that your users are only going through the Client Side Rendering (CSR) flow.

When integrating Statsig with Google Analytics 4 (GA4), you might notice that all events are sent under the name 'statsig'. This is expected behavior as Statsig sends all logged events and exposures to GA4 under this name once the integration is enabled. These events include properties such as 'category', 'config', 'group', 'value', 'stable_id', 'user_id', 'user_ip', and 'user_userAgent'.

The GA4 integration works for all experiments, not just the ones created after the integration is complete. However, it's important to note that the events are forwarded to a configured Data Stream and can be filtered with event filtering. If you're not seeing the expected events, it might be worth checking your event filtering settings or the configuration of your Data Stream.

Google Analytics is known to have delayed reports, which can range from 24 to 48 hours. Therefore, if you have recently enabled the integration, it might just be a matter of waiting for the events to appear in GA4.

Also, ensure that in the integration configuration, you have selected the necessary options to send all events, including any new events. If you're still having issues, please reach out to the Statsig team for further assistance.

If you're using the dynamicConfig.get("key", defaultValue) method to fetch a value from a Dynamic Config in Statsig and receiving null, it could be due to several reasons:

1. The key does not exist in the Dynamic Config: Ensure that the key you're trying to fetch is correct and exists in the Dynamic Config.

2. The Dynamic Config has not been initialized or fetched correctly: Check the initialization and fetching of the Dynamic Config. Make sure it has been done correctly as per the Statsig documentation.

3. Issue with the default value provided: There might be a problem with the default value you've provided. Verify that it is correct and compatible with the expected value type.

If you've verified the key, checked the initialization and fetching of the Dynamic Config, and ensured the default value is correct, but you're still receiving null, it might be a more complex problem that requires further investigation. In such cases, we recommend flagging this issue for further assistance from the Statsig team.

The "Uninitialized" assignments in Statsig could be due to the initTimeoutMs parameter in the Provider. This parameter determines how long the Statsig client waits for the initial network request to respond before proceeding. If this timeout is too short, it could lead to an "Uninitialized" status if the initialization network request doesn't complete within the specified time.

In some cases, a 400ms timeout might be too short, causing the initialization to not complete in time, especially if network conditions are not optimal. It is recommended to consider increasing this timeout value to allow more time for initialization.

However, for Server Side Rendering (SSR) pages using the StatsigSynchronousProvider, there is no concept of initTimeoutMs. Therefore, the initialization happens on the server and there should be minimal uninitialized cases. If you are seeing a notable number of uninitialized cases for an experiment on an SSR page, it could be due to other factors not related to the initTimeoutMs parameter.

Please note that the initialization process involves a network request, and depending on the network condition and location of the users, a short timeout like 400ms might not be sufficient for many users.

In order to enable reviews for changes that will impact your production environment, you need to follow certain steps. This includes changes to a dynamic config or gate, which affect rules with the production environment or apply to all environments. It also includes changes to an experiment that has started.

The philosophy behind this approach is to strike a balance between allowing developers to move quickly in a local environment, while ensuring that any changes that will impact end users will require a review.

Here's how it works:

1. Reviews are only required for changes that will have an effect on the production environment. This means that if you're making changes to a dynamic config or gate, reviews will only be required if these changes affect rules with the production environment or apply to all environments.

2. For experiments, reviews are only required for changes to an experiment that has started. This is because these changes could potentially impact your end users.

If you're seeing behavior that doesn't align with these rules, it's possible that there's a technical issue that needs to be addressed. However, in most cases, following these guidelines should ensure that your changes are reviewed appropriately before they're implemented in a production environment.

When conversions are not showing up in Statsig test results, there are several troubleshooting steps you can take to resolve the issue. First, verify that the events sent from your application include the same identifier (either userID or stableID) as the one used in your client SDKs for the experiment. This ensures that Statsig can correctly join the events with the appropriate experiment groups.

Additionally, be aware that Statsig computes custom metrics on a daily basis for your Metrics dashboard and aggregates them for the duration of the experiment in your Pulse Results. Custom metrics will not populate until the day after creation and will not retroactively apply to previous days.

Statsig will calculate them from the creation date forward. If you are using precomputed metrics, confirm that Statsig is receiving your precomputed metrics. If the Metrics Stream is not visible at the bottom of your Metrics Catalog, there may be a connection issue or an invalid schema.

If these steps have been checked and the issue persists, it is advisable to contact a Statsig team member for further investigation.

When a feature gate is set to Launched or Disabled, it will always return the default value (TRUE for Launched, FALSE for Disabled) and stop generating billable exposure events. However, disabling a feature gate will not stop checks against it - it'll just always return false. Removing it from your codebase will stop checks - assuming you don't have any old versions of your code still running anywhere.

If you're still seeing checks after that, it's possible that there might be some instances where the gate is still being referenced. Double-check your codebase to ensure that all references to this gate have been removed. If the issue persists, it is recommended to flag this issue to the Statsig team for further assistance.

In some cases, you might still see exposure events in Stitch for gates you've launched/disabled (and all are presumably False). This could be due to a bug in the system. The Statsig team is always ready to identify and fix such issues.

Please note that fixes for such bugs are usually rolled out promptly and should resolve over a short period of time. However, if you're still not seeing the issue as resolved on your end, it is advisable to reach out to the Statsig team for further investigation and assistance.

If you're not seeing data in the "Exposure Stream" tab, there could be a few reasons for this. Here are some steps you can take to troubleshoot:

1. Check Data Flow: Ensure that the id_type is set correctly and that your ids match the format of ids logged from SDKs. You can check this on the Metrics page of your project.

2. Query History: If your data is still not showing up in the console, check your query history for the user to understand which data is being pulled, and if queries are not executing or are failing.

3. Data Processing Time: If data is loading, it's likely we're just processing. For new metrics, give it a day to catch. If data isn't loaded after a day or two, please check in with us. The most common reason for metrics catalog failures is due to id_type mismatches.

4. Initialization Status: Each hook has an isLoading check, which you can use, or the StatsigProvider provides the StatsigContext which has initialization status as a field as well. This can be used to prevent a gate check unless you know the account has already been passed in and reinitialized.

If you've checked all these and are still experiencing issues, please let us know. We're here to help!

In some cases, the issue might be due to the way you are accessing the layer config and getting the value of the config. When you manually reach into the layer config and get the value of the config, we won’t know when to generate a layer exposure. For example, if you have a line of code like this:

const config = statsig?.getLayer('landing_page_gg')?.value;

While this will give you the value of the config, it won’t generate exposures and your experiment won’t work. In order to generate the correct exposures, you will need to get the Layer object and ask for the values using the “get” method. Like this:

const layer = statsig?.getLayer('landing_page_gg'); const block_1 = layer.get('block_1', '');

And similarly for other parameters you have defined in your layer.

If you are pulling multiple values at once and they control the same experience, it is recommended to pack them into a single parameter. However, if the parameters affect experiences in multiple locations, keep them separate. If you are pulling everything in one shot, treat them as a single object value. Use the json type. You could also just add a new parameter to the current experiment that has all the other values. There's no need to create a new experiment if you don’t want to.

When comparing event counts between Statsig and GA4, it is important to consider the potential for discrepancies due to time zone differences.

Statsig operates on a default day that starts at GMT-8 (Pacific Standard Time) and maintains this standard throughout the year, without adjusting for daylight savings time. This can lead to differences in daily event counts when compared to other platforms that may use different time zones for their reporting.

Additionally, the methodologies for logging and processing events can vary between Statsig and GA4, which might contribute to the observed discrepancies. It is also crucial to check for any filters or exclusions that may be applied within either platform, as these can affect the event counts.

If discrepancies continue to be significant, a thorough review of the event logging code is recommended to ensure that events are being logged accurately and consistently across both platforms. In cases where the issue persists, reaching out to the Statsig support team for further assistance is advised.

When utilizing the Statsig PHP SDK to send gate exposure data, it is expected that the events should appear in the Diagnostics event stream for the corresponding gate once the cron job is executed.

To ensure that the SDK is properly initialized and the cron job is configured correctly, developers should review their implementation. A typical initialization snippet and cron job usage are essential for troubleshooting any issues with event visibility on the Statsig dashboard.

If events are not appearing as expected, developers should verify that the SDK is correctly set up and that the cron job is functioning as intended.

It is important to ensure that the SDK is properly flushed before the process exits to guarantee that all events are sent to Statsig servers. If discrepancies persist after verification, further investigation into the SDK's event logging mechanism may be necessary.

When utilizing a service with a Free tier event limit, reaching this limit does not inherently affect the visibility of non-production logs. Non-production logs are intentionally not displayed in the console by default to ensure they do not impact the results of experiments.

To view non-production events in the log stream for a production event, users can toggle the appropriate setting within the console. For testing purposes, events can be suffixed with '_test' and logged using the production environment. This allows for the testing of event logging without affecting production data. Once testing is complete and the event logging is to be finalized, these suffixes can be removed.

If non-production logs are still not visible after these steps, it is likely that there is an issue unrelated to the Free tier event limit. Users are encouraged to seek further assistance to resolve any such issues.

In the event that some metrics in your ongoing experiment are showing as "unavailable", it could be due to an issue with how your events are being logged or matched with the experiment groups. To resolve this, ensure that the events you're sending include the same identifier (either userID or stableID) as the one used in your client SDKs for the experiment. This will help Statsig join the events with the correct experiment groups.

It's important to note that exposures on the last day (the day you made a decision) are not counted as exposures. Therefore, when analyzing your data, remember to filter out that day.

If you're still experiencing issues, it might be helpful to check your query history for the user to understand which data is being pulled, and if queries are not executing or are failing.

In some cases, metrics may temporarily show as unavailable due to Statsig recomputing the metric for the latest numbers. However, these metrics should become available once the final aggregated data is uploaded and any issues with this process are resolved.

If the issue persists, please let us know and we'll have a Statsig team member assist you further.

If you have integrated Statsig with Segment and are experiencing issues with some experiment events not being sent to Segment, there are a few things you could check:

1. Event Filtering: Statsig allows you to filter which events should be sent to Segment. Please check your Event Filtering settings in the Statsig console to ensure that the missing events are not being filtered out.

2. Segment Integration: Please ensure that your Segment integration is still correctly configured and enabled.

3. Event Data: Ensure that the events you're sending from Segment include the same identifier (either userID or stableID) as the one used in your client SDKs for the experiment. This will help Statsig join the events with the correct experiment groups.

If you've checked all of these and are still experiencing issues, please let us know. We'll flag this issue and a Statsig team member will be by to help shortly.

When user interactions with a feature, such as clicks on a banner, are not reflected in the Statsig exposure export, it is important to understand that Statsig exports all exposures without exception. If a user has been exposed to a feature gate, they will be included in the analysis. However, discrepancies can occur for several reasons.

For instance, if the user's attributes have changed post-exposure or if the feature gate rules were modified, the user might not qualify anymore but would still be included in the analysis.

To verify exposures, users can log into the Statsig console, select the feature gate, and click on 'View Pulse'. The 'Cumulative Exposures' panel will display the total exposures, including a breakdown of users exposed to the feature (treatment or Pass group) and users not exposed to the feature (control or Fail group).

If issues persist, it is recommended to utilize the debugging tools provided in the Statsig documentation to understand why a certain user received a specific value. In cases where the problem cannot be resolved, reaching out to the Statsig team for assistance is advised.

In response to concerns about an unexpected increase in billable events, it is important to note that Statsig has robust mechanisms in place to prevent the duplication of billable events.

For client SDKs, Statsig ensures that only one billable unit is counted within a one-hour window, while for server SDKs, this window is reduced to one minute. This effectively means that multiple exposure events for the same stable ID within these time frames do not lead to additional billable units. An increase in billable events could potentially be attributed to a rise in unique user interactions or an uptick in the usage of feature gates, experiments, or dynamic configs.

Users are advised to monitor their billing through the billing page under project settings, which provides aggregated information at the gate/test level. In cases where discrepancies are observed, reaching out to the Statsig team is recommended for a detailed investigation.

Additionally, it has been noted that for certain plans with unlimited gate checks, the plan may not have been correctly applied to a project. Once rectified, the billing page will reflect a significant number of events left, with most gate checks being free.

It is also clarified that only gate checks on rules that are partially open and have analytics enabled are billed.

If you're unable to adjust the dates in the scorecard for your old experiment, it could be due to the experiment's end date being set in the past. When an experiment is created, it is marked with a specific completion date. If you want to extend the experiment or adjust the dates, you have two options:

1. Clone the existing experiment and restart it, making sure to set the experiment end date to the desired date or mark it as ongoing.

2. Reset the experiment from the same menu and restart it from there.

After resetting the experiment, you can change the end date. However, you might notice that the pulse loading takes a longer time than usual. This is not normal even for this experiment. The Wall time, which is the total "real time" it took to run the job, from when it started to when it finished, might be longer than the job time, which is the actual compute time run on your warehouse. This could be due to transient network errors communicating with your warehouse, which is unusual. If you encounter this issue, it's recommended to flag it for assistance.

In the Explore tab of Pulse Results, you can use user dimensions to drill down and explore results. However, the user properties you can segment on are only derived from exposure events, not custom events. Any overrides that are used to force a user into the test will be ignored from pulse results. If you only have override userIds in the results, nothing will show up in the pulse filters. If you're having trouble finding a specific user property, it might be due to these reasons.

It's also important to note that the availability of User Dimensions can vary depending on the gate/experiment you’re querying on. Some entities have User Dimensions available, some don’t based on whether there are exposures or not.

Regarding the issue of error ranges for Daily Topline Impact and Daily Projected Launch Impact not populating, this was due to a bug that occurred during our internal migration of infrastructure and data. We have fixed this bug and it affected certain metric types. All new custom queries should not have a problem. For some metrics, the topline impact could have a 0 as the CI, because all users can have the value of 1 if the metric is an indicator. However, we'll do an audit to make sure the calculation is consistent with our scorecard.

When attempting to update user attributes in a development environment using the JavaScript SDK and subsequently searching for the user in the Statsig admin UI, it is important to understand the expected behavior and requirements for user data to be logged and visible.

Users initialized as anonymous will only have a stableID. If you assign them to a feature or test, they will appear in the Users tab, searchable by stableID, although there may be a delay as data takes time to reach the Users tab.

To see a user's profile updated with additional information such as userID and email in the Users tab, you must enrich their user object with these details and then trigger a subsequent event. It is important to note that only 'production' data is displayed in the Users tab.

However, you can view assignments in real-time for both production and lower environments in the Diagnostics tab. To log the user object, you must call the logEvent API or alternatively use checkGate or getExperiment, as merely initializing the SDK will not log the user object.

For debugging purposes, you can use the Exposures Log stream for any test or gate under the Diagnostics tab, and the Event logstream under the Metrics page to view any custom logEvent events sent to Statsig.

If you're unable to see the Holdouts feature in your Statsig console, it's possible that it has been relocated. The Holdouts feature has been moved to the navbar under experiments. Please check there to access this feature.

If you continue to experience issues, please contact our support team for further assistance.

In the scenario where config.getGroupName() returns null in your SSR page using NextJS and Statsig, it's important to understand that the getGroupName() function is expected to return null if the dynamic config does not have a group name associated with it. This could occur if the dynamic config is not part of an experiment group.

If you believe this is an error and the dynamic config should be part of a group, it's recommended to double-check your configuration in the Statsig console. If the issue persists, it might be a good idea to flag this for further investigation by the Statsig team.

In the case where you're calling getGroupName on an experiment config, such as statsig.getExperiment('test').getGroupName(), and it returns null or "undefined", it's possible that the groupName is not included in the initializeValues payload.

This issue might require a fix to the NodeSDK. The NodeSDK downloads the groupname via Statsig on initialization, but it doesn't include it in the payload generated by Statsig.getClientInitializeResponse.

As a workaround for now, you can create a group name parameter in your experiment or config, and get that from the payload of the dynamic config. This should help you bypass the issue until a fix is implemented.

In Statsig, the "Users" tab is designed to display common properties and evaluations for a given user ID. However, it may not always show all custom properties. The most reliable way to view custom fields added to the user from the Statsig dashboard is through exposure checks.

If you are able to group by a specific user property when performing a custom query for a gate, it indicates that the property is being logged correctly. The absence of this property in the "Users" tab could be due to the temporary nature of custom properties.

If you encounter any issues or require further clarification, please reach out for assistance.

In Feature Gate and Dynamic Config rules, the "equals" operator is not explicitly supported for numbers. However, there are alternative methods to achieve an equivalent functionality.

One such method is to use the "any of" operator with a single item in the array. This will effectively perform an equality comparison. Importantly, there is no need to cast integers to strings for this operation; the system handles the necessary conversions.

As a best practice, it is recommended to test this functionality to ensure it meets your specific requirements.

Remember, the operators provided in Feature Gate and Dynamic Config rules are designed to offer flexibility and cover a wide range of use cases. For numbers, operators like "greater than", "less than", "greater than or equal to", and "less than or equal to" are often more useful in defining conditions. If you need to check for an exact match, you could potentially use both "greater than or equal to" and "less than or equal to" with the same value to mimic an "equals" operation.

For any further questions or specific functionality needs, please reach out to the support team.

To check if a specific user ID or stable ID has been manually added to any experiments using "Manage Overrides", follow these steps:

1. Log into the Statsig console

2. On the left-hand navigation panel, select "Users".

3. In the User's tab, enter the specific user ID or stable ID you're looking for in the "ID" field and select the appropriate "ID Type" from the dropdown menu.

4. Click "Submit" to load the data for the entered ID.

The resulting information will include a section titled "Overrides", which lists all the entities (Feature Gates, Experiments, etc.) that the given ID has been manually overridden into. Please note that you need to enter both the ID and ID type to submit a query.

If a stable ID is not showing in search results despite being present in many experiments, there are a couple of things to check:

1. Ensure that the box “include in Pulse results” was checked in the override modal.

2. Consider the timing of the assignments/exposures. It may take some time for the system to update and reflect recent changes.

If you're only seeing overridden values in the Exposure Stream for your experiment, there could be several reasons for this. Here are some steps you can take to troubleshoot:

1. Check the Initialization Status: Each hook has an isLoading check, which you can use, or the StatsigProvider provides the StatsigContext which has initialization status as a field as well. This can be used to prevent a gate check unless you know the account has already been passed in and reinitialized.

2. Check the Data Flow: Ensure that the id_type is set correctly and that your ids match the format of ids logged from SDKs. You can check this on the Metrics page of your project.

3. Check the Query History: If your data is still not showing up in the console, check your query history for the user to understand which data is being pulled, and if queries are not executing or are failing.

4. Check the Exposure Counts: If you're seeing lower than expected exposure counts, it could be due to initializing with a StatsigUser object that does not have the userID set, and then providing that on a subsequent render. This causes the SDK to refetch values, but logs an exposure for the “empty” userID first. To prevent this, ensure the userID is set before initializing the StatsigUser object.

If you've checked all these and the issue persists, it might be best to reach out for further assistance.In some cases, users may still qualify for the overrides based on the attributes you’re sending on the user object. This may be due to caching, or it may be due to the user qualifying for other segments that control overrides.

For example, if users have "first_utm_campaign": "34-kaiser-db" being sent on the user object, they would qualify for a segment that’s being used in the overrides.

It's also important to note that overridden users will see the assigned variant but will be excluded from experiment results. We have a way to include users in results for experiments not in layers, but it seems we don’t have that option for experiments in layers.

Lastly, consider why you are using overrides in this scenario instead of a targeting gate. Overrides can be used to test the Test variant in a staging environment before starting the experiment on prod. However, if some of your customers have opted out of being experimented on, a targeting gate might be a more suitable option.

In Statsig, the 'Last daily Event Count' column displays the count of events for each day. If you observe a value of 0 for all events, it indicates that no events were logged on the previous day. The numbers will update when new data is processed in a few hours. You can click on each event to see this.

However, it's important to note that the charts only include production events. Non-production events can be viewed in the logstream by toggling to non-prod logs. These non-production events are not included in the charts to prevent them from polluting experiment results.

If you're running an Autotune experiment and notice that the Success Event is not being counted on the Pulse Results, it could be due to a few reasons.

Firstly, ensure that the events you're sending include the same identifier (either userID or stableID) as the one used in your client SDKs for the experiment. This is crucial for Statsig to join the events with the correct experiment groups.

If you're still not seeing the expected results, you might want to check your query history for the user to understand which data is being pulled, and if queries are not executing or are failing. If data is loading, it's likely we're just processing. For new metrics, give it a day to catch. If data isn't loaded after a day or two, please check in with us. The most common reason for metrics catalog failures is due to id_type mismatches.

In the case of using a Segment event for the success metric, it's important to note that there may be a delay in the Autotune results being updated. Data should be updated hourly but could be a bit later depending on your attribution window.

If you continue to not see the data as expected after more time has passed, please reach out for further assistance.

When you make a change in the Statsig Console, such as updating a targeting rule, it is expected that the new rule may take around 5 seconds to propagate. This is in line with our Service Level Objective (SLO). While an instant update might be desirable, it's important to note that this slight delay typically has no real impact on key metrics and user experience.

In the context of feature gates, for example, if you open a feature for Mac users and later remove access for Mac users and make it available for Windows users, there will be a slight delay before the change is reflected for Mac users. This is due to the Time To Live (TTL) associated with any type of hosted service.

Regarding the capacity of ID lists, each project can handle up to 10 million IDs. This means if you have 3 projects, each can handle its own list of 10 million IDs.

The discrepancy between the experiment allocation counts and the ssr_search_results_page_view (dau) counts could be due to several reasons:

1. **User Activity**: Not all users who are allocated to an experiment will trigger the ssr_search_results_page_view event. Some users might not reach the page that triggers this event, leading to a lower count for the event compared to the allocation.

2. **Event Logging**: There might be issues with the event logging. Ensure that the statsig.logEvent() function is being called correctly and that there are no errors preventing the event from being logged.

3. **Timing of Allocation and Event Logging**: If the event is logged before the user is allocated to the experiment, the event might not be associated with the experiment. Ensure that the allocation happens before the event logging.

4. **Multiple Page Views**: If a user visits the page multiple times in a day, they will be counted once in the allocation but multiple times in the ssr_search_results_page_view event.

If you've checked these potential issues and the discrepancy still exists, it might be a good idea to reach out to the Statsig team for further assistance.

Another possible reason for the discrepancy could be latency. If there is a significant delay between the experiment allocation and the event logging, users might abandon the page before the event is logged. This could lead to a lower count for the ssr_search_results_page_view event compared to the allocation.

When encountering an unexpected spike in the new_dau metric without a corresponding increase in website traffic, there are several potential causes to consider. Firstly, it is important to verify the accuracy and consistency of event logging. Any discrepancies in how events are logged can lead to anomalies in the metrics.

Secondly, if data is being imported from external sources, ensure that the ingestion process is functioning correctly and that data is imported in a timely manner.

Additionally, be aware of timezone differences that may affect the interpretation of metrics; Statsig uses Pacific Standard Time (PST) as its default, which could lead to discrepancies if other systems are set to different timezones.

Lastly, the use of user identifiers should be consistent across all events and sessions to maintain the integrity of the data.

If these checks do not resolve the issue, further investigation into specific events that correlate with the metric increase may be necessary. It is also advisable to reach out directly for support if the cause of the spike remains unclear.

When conducting experiments, it is crucial to ensure that there is no cross-contamination between control and treatment groups and that data is accurately reflected in both funnel and summary views.

Cross-contamination can occur due to implementation issues, such as a race condition with tracking. This happens when users, particularly those with slower network connections, land on a control page and a page-view event is tracked before the redirect occurs.

To mitigate this, it is recommended to adjust the placement of tracking scripts. The Statsig redirect script should be positioned high in the head of the page, ensuring that it executes as early as possible. Meanwhile, page tracking calls should be made later in the page load lifecycle to reduce the likelihood of premature event tracking. This adjustment is expected to decrease discrepancies in tracking and improve the accuracy of experiment results.

Additionally, it is important to confirm that there are no other entry points to the control URL that could inadvertently affect the experiment's integrity. Ensuring that the experiment originates from the correct page and that redirects are functioning as intended is essential for maintaining the validity of the test.

Lastly, it is necessary to have specific calls in the code to track page views accurately. These measures will help ensure that the experiment data is reliable and that the funnel and summary views are consistent.

In order to get the useExperiment hook to work correctly in your code, you need to ensure that the StatsigProvider wraps your entire app or at least the root component that uses Statsig. This is because the StatsigProvider initializes the SDK and passes down its state to the rest of the React application via a StatsigContext.Here's how you can modify your code:

jsx // Initialize Statsig at a higher level, ideally in your app's root component <StatsigProvider  sdkKey=CLIENT_SDK_KEY  waitForInitialization={true}  user={{userID: userData?.id, email: userData?.email}}  options={{    environment: { tier: "development" },  }} >  <App /> {/* Your app's root component */} </StatsigProvider> // Then in your ButtonModal component export function ButtonModal({ ...props }: ButtonModalProps) {  const classes = useStyles();  const [ isButtonModalOpen, setIsButtonModalOpen ] = useState(false);  const { config } = useExperiment("button_test");  // Rest of your code... } By doing this, the StatsigProvider will initialize the SDK once and provide the necessary context for all child components. This should allow the useExperiment hook to work correctly. Remember, you cannot call useExperiment in the same component that is rendering the provider. The provider needs to wrap a component above this in the hierarchy.

In the context of experiments, when a user is attributed to the 'uninitialized' state, it indicates that the user has not been properly initialized within the experiment framework. As a result, the user will not be assigned to a real experimental group but will instead fall into the 'Default' group. It is important to note that the 'Default' group is not included in the analysis of experiment metrics.

Therefore, data from users in the 'uninitialized' state will not be included in the experiment metrics. This ensures the integrity of the experiment results by only including data from users who have been correctly initialized and assigned to a valid experimental group.

Yes, the experiment buckets will be preserved for the Stable ID experiments when the user transitions from the Server Side Rendered (SSR) page to the Client Side Rendered (CSR) page or vice versa. This is because the Stable ID is a unique identifier that persists on the device and is used to ensure that your users have the same experience or group across different states, including logged out to logged in states.

Even though the user object (userId) changes between these pages, as long as the Stable ID remains the same, the user's experiment group will remain consistent. This is because the Stable ID is used for bucketing in the experiment, not the userId.

However, it's important to ensure that the Stable ID is correctly implemented and persisted across different pages and sessions. If you're facing issues with Stable ID changing across subdomains, one possible solution is to store the Stable ID in a cookie and reuse it. You can store and retrieve your own id and use the overrideStableID method on the SDK to send it to Statsig. Your overridden id will always take preference.

Remember to always follow the best practices as outlined in the documentation when implementing experiments with Statsig.

When integrating Statsig with Vercel's Edge Config, it is important to understand the behavior of user allocation during updates to experiment configurations.

According to our expert, increasing the allocation percentage in an experiment will not affect users who have already been assigned to the test. However, it is the decrease in allocation that could potentially remove users from the test upon their revisit.

For users who were not allocated during their first visit at a certain percentage, they can indeed be allocated if they revisit after the exposure allocation has increased. This is applicable when using a user ID as the identifier without considering sessions. It is also noted that the Statsig Client SDKs offer a keepDeviceValue argument with getExperiment that enforces stickiness via localStorage, although this does not apply to Server SDKs.

During an update delay, there might be a temporary mismatch in user allocation results when polling from the edge versus polling from the client SDK. The edge configuration might not reflect the new allocation immediately, leading to a period where the user could be allocated on the client side but not yet on the edge.

This mismatch will persist until the edge data is updated to match the new allocation configuration.

When a user qualifies for an experiment based on certain user properties and is assigned a variant, their allocation to that variant is deterministic and will remain consistent throughout the experiment, even if their qualifying properties change and they temporarily no longer qualify for the experiment.

If a user's properties change such that they no longer meet the experiment's qualifying criteria, they will receive the default parameter values until their properties revert to a state that re-qualifies them for the experiment. Upon re-qualification, the user will be reassigned to the same variant as initially allocated.

This deterministic allocation ensures that a user's experience remains consistent across multiple sessions and that the integrity of the experiment is maintained. The allocation is based on a stable identifier, such as a userID or stableID, which allows for consistent group assignment upon re-qualification.