Frequently Asked Questions

In the context of dynamic configurations, the concept of extending configs based on user tiers, while maintaining a base configuration, is a common query. The approach to achieve this functionality is not through dynamic addition of configuration segments, but rather through the use of targeted rules for each tier.

To implement tier-based dynamic configurations, it is recommended to create a complete version of the dynamic config for each tier. This means that instead of having a base config with incremental additions, each tier should have its own full set of configurations. User targeting is then employed to determine which configuration should be applied to a user based on their tier level.

This method ensures that the correct and complete set of configurations is delivered to the user, adhering to the tier they belong to. It is important to note that this process does not support the dynamic addition of configuration segments; instead, it relies on the pre-definition of full configurations for each tier.

In the context of adding filters to metrics within a funnel, it has been confirmed that this functionality is indeed supported. Users are able to apply filters to Event_count and User_count metrics, which can then be utilized in the creation of a funnel.

This provides the flexibility to tailor metrics to specific needs and answer targeted questions effectively. An example provided includes the creation of a “Product View Sale” metric with an applied filter, which is then visible and operational within the funnel creation workflow.

It is important to note that while filters can be added, they are limited to the aforementioned metric types.

In the Statsig SDKs, there is no functionality to retrieve a user's historical gate pass status.

The SDKs are designed to evaluate the current state of a feature gate for a user at the time of the check, rather than providing historical data. As such, it is not possible to use the SDK to directly query if a user has passed a certain gate in their history.

To track a user's interactions with feature gates over time, it is recommended to log an event each time a user passes through a gate using the Statsig.logEvent function available in the SDKs. These logged events can then be analyzed within the Statsig console to gain insights into the user's history with the feature gates.

It is important to note that when checking a gate or logging an event, comprehensive user information should be provided to leverage the full capabilities of the Statsig platform.

Creating a sequential funnel that includes a step with multiple events is not directly supported as funnel metrics are designed to be linear, with each step corresponding to a single event. However, you can gain similar insights by utilizing Ratio Metrics.

Ratio Metrics allow you to define a metric using other metrics as the numerator and denominator. This means you can create a metric that combines the counts of multiple events, which can provide an analogous understanding to having two events count as the same step in a funnel metric.

To implement this, you would define a new Ratio Metric where the numerator is the sum of the events you wish to combine, and the denominator is the preceding event in the funnel. This approach allows for flexibility in analyzing user behavior across multiple events without altering the linear structure of funnel metrics.

In the context of managing feature gates within Statsig, the ability to rename an existing feature or experiment is not permitted.

This restriction is in place to avoid potential errors that could arise from renaming a feature that is already being referenced in the codebase, which could lead to the feature gate not returning the correct values.

If there is a need to change the name of a feature gate that has not yet been deployed, the recommended approach is to clone the existing feature gate and assign it the desired new name. This action will result in the creation of a new feature gate with the same configuration rules as the original, but it is important to note that the historical data and metrics associated with the original feature gate will not be transferred to the new gate.

In Statsig, metrics that have been around for a while and contain data can be renamed without losing any historical data. This feature is being rolled out and allows for a more comprehensive approach to renaming metrics while preserving their associated data.

If you have a metric that requires renaming and you wish to retain its data, you can request to have your project included in the rollout of this new feature. Once your project is included, you will be able to rename your metrics accordingly.

It is recommended to provide feedback or report any issues encountered during the use of this new renaming feature to ensure continuous improvement and resolution of potential problems.

Currently, Statsig does not provide a direct way to send different types of notifications to different channels in Slack. When setting up the Slack integration, you select a channel to direct all project-level messages. However, once the integration is enabled, you have the option to opt-in to additional notifications such as Health Checks and Statsig system Status Reports. You can also filter change updates by the environment affected.

The notifications will include updates such as when a gate has been edited, which should provide some context about the change. However, the ability to add specific message or rule details is not explicitly mentioned in the documentation.

If you require more specific functionality, it is recommended to reach out to the Statsig team for further assistance. Please note that during the installation process, you instruct the app which channel to send all the messages.

In the context of setting experiment group splits with decimal precision, the system is designed to limit the total number of buckets to 10,000, which allows for one digit of decimal precision.

When dealing with multiple variations, such as 32 in an experiment, it is recommended to assign 3.1% for 31 variants and 3.9% for the control group. This approach is considered acceptable for the experiment, and the slight difference in precision (e.g., 3.1% vs. 3.125%) is deemed negligible, especially in larger sample sizes where the difference might equate to a small number of users.

For experiments requiring more precise splits, alternating between 3.1% and 3.2% for every eighth variant is a suggested strategy. This allocation method is also applicable to more complex experimental designs, such as a 2-level fractional factorial experiment.

The key consideration is the statistical precision, which may be more critical depending on the user base size. It is important to note that the fixed number of buckets behind the scenes necessitates these allocation strategies.

Yes, it is possible to use an event value for Segment events. When sending events from Segment to Statsig, anything in the properties field of the Segment event is treated as the event metadata field on Statsig's end. This can include an event value. For instance, you can use this field to specify the type of achievement that was unlocked, or the name of the product that was added to cart, or the price of the item that was purchased. However, the specific implementation details might need to be confirmed with the Statsig team.

When it comes to the Autotune configuration, it's important to note that it currently doesn’t support optimizing for the value on an event. Autotune can only be configured to optimize for the count or occurrence of an event. For example, if you want to pick the variation that will have the most number of success events, you pick that event.

For A/B testing with randomized number distributions, Statsig's experiments feature is the recommended approach.

To achieve this, you should create multiple test groups within an experiment. Each user who interacts with your application will be deterministically assigned to one of these test groups based on their user ID, which could be either a stableID or userID.

This assignment ensures that each user consistently experiences the same test group during the experiment. Test groups can be configured with associated parameters, allowing for dynamic control over the behavior and experience of each test group.

This setup enables you to conduct A/B/n experiments where users are evenly distributed across the different configurations you wish to test.

For detailed guidance on configuring your groups and parameters, refer to the official Statsig documentation on creating and managing experiments.

At present, Statsig does not support the use of Sankey charts for visualizing user journeys. However, you can utilize funnel metrics to track user journeys within your application. These funnel metrics can be highly customized to provide insights into how changes influence each step of the user journey, either at the user level or at the session level.

To create funnels in Statsig, navigate to Metrics -> Explore -> select “Funnel” in the dropdown menu. Please note that this is a beta feature and currently does not support having experiment exposures in the steps. However, this is a feature we are considering for future development in our analytics offerings.

While visualizations like Sankey charts can be useful, they often struggle to effectively represent small changes, such as 5% deltas, in experimentation. Therefore, even if the conversion rate doubled (+100%), it might still be challenging to spot in a Sankey chart. We are actively investigating this area and are open to customer feedback to find an ideal visualization method.

Currently, the feature to split L7 (Last 7 days), L14, and L28 metrics by different groups, segments, or cohorts is not available.

This functionality is under active development, and considerations are being made to include it in future updates. As of now, users looking to analyze metrics by these specific time-bound parameters will not be able to use the 'Group By' feature to disaggregate the data by properties such as user type, geographical location, or other custom parameters.

This limitation applies to the built-in metrics within the platform. Users are encouraged to stay tuned for updates on the development of this feature, which aims to provide more granular insights into user behavior and metric performance over specified time frames.

In a subscription-based business model, understanding the long-term impact of experiments on Customer Lifetime Value (CLV) is crucial.

Statsig provides the capability to analyze the impact of an experiment on CLV over extended periods, such as 6 months. To facilitate this, Statsig allows for the setup of an experiment to run for a specific duration, such as 1 month, and then decrease the allocation to 0%, effectively stopping new user allocation while continuing to track the analytics for the users who were part of the experiment.

This tracking can continue for an additional 5 months or more, depending on the requirements. It is important to note that the experience delivered to users during the experiment will not continue after the allocation is set to 0%. However, there are strategies to address this, which can be discussed based on specific concerns or requirements.

Additionally, Statsig experiments by default expire after 90 days, but there is an option to extend the experiment duration multiple times for additional 30-day periods. Users will receive notifications or emails as the expiration date approaches, prompting them to extend the experiment if needed.

This functionality is available on the Pro plan, ensuring that businesses can effectively measure the long-term impact of their experiments on CLV without direct integration with a data warehouse and by updating CLV through integrations such as Hightouch.

Statsig currently does not offer the capability to filter inbound events based on specific userIDs or stableIDs. However, there is a plan to introduce a 'Global Targeting Rule' which will allow users to exclude certain types of users from all tests and feature gates.

It is important to note that this 'Global Targeting Rule' is designed to omit users from tests and gates, and not for the purpose of optimizing billing. Users who wish to exclude specific traffic for billing optimization will need to implement logic to discard such traffic upstream, either within their data collection tools or by adding a wrapper around Statsig log event calls.

In scenarios where you wish to modify the default behavior of a feature gate in Statsig, which typically fails (returns false) when conditions are not met, there are alternative strategies you can employ.

One approach is to use negation operators within your conditions, such as 'not in' or 'not equal to', to effectively invert the logic of the gate. This allows you to create a condition that passes by default unless specified criteria are met. Another method is to adjust the 'fail' percentage to 100%. By doing so, you ensure that the feature gate fails for all users, effectively reversing the default 'off' state to an 'on' state.

These techniques allow for greater flexibility in controlling feature gate behavior and can be tailored to meet the specific needs of your experiment or feature rollout.

When you click on an event in the Event Log Stream, a popup with detailed information, including the stableID, should appear. This is the default behavior when the ID type configured on the gate/experiment is shown in the list view.

If you want to make a gate or experiment a stableID based gate/experiment, the most relevant column to show to ensure you have configured your gate/experiment and checks in code correctly is the unit type of the gate/experiment itself. If you change the unit type to “stableID”, that column will change to stableID.

It's important to note that Statsig allows you to experiment on userID, stableID, and up to 5 customIDs. Therefore, the table can't really scale to show all the IDs inline. However, once the unit type is changed to "stableID", the Statsig Stable ID will appear in the exposure stream.

In the context of updating experiments and managing permissions for editing monitoring metrics, it is important to understand the current capabilities of role-specific permissions.

According to the expert, there is a setting within the 'Members & Roles' section under 'Project' that allows for role-specific permission adjustments. By navigating to the 'Roles' tab and selecting a 'Member' role, users can access the 'Permissions' tab where they can find a setting called 'self-approve review.' This setting enables a role to bypass the review structure entirely.

However, it is noted that there is no current permission setting that allows for the customization of permissions to the extent of enabling a user to edit monitoring metrics specifically without granting them the ability to bypass all reviews.

The expert acknowledges the validity of this use case and has indicated that they will discuss the possibility of implementing such a feature with their team. As of the expert's last statement, permissions for general metric management and verify metrics exist, but not for editing monitoring metrics specifically.

In the context of using Statsig for feature management, it is important to understand how to identify all the feature gates that utilize a specific user segment.

While there is no direct method or function provided to automatically list all feature gates that use a particular segment, users can manually check the rules of their Feature Gates within the Statsig console.

Each rule within a Feature Gate specifies the user attributes it operates on, and by examining these rules, one can determine if a specific segment is being used. For those seeking to automate this process, the Statsig Console API offers a solution.

The segment endpoint of the API exposes the attributes that are in use, which can be leveraged to programmatically identify the use of specific user attributes, including segments, across various gates. This approach can significantly streamline the process of checking for the usage of specific segments in the configuration of Feature Gates.

The tutorial on setting up an A/B/n test using Statsig's Experiments feature uses two examples to illustrate different variants you might test in an experiment.

The first example, enable feature, is a boolean type feature gate that can be either true or false. This represents the feature being on or off and is a simple A/B test scenario where you're testing the impact of a feature being enabled vs. not enabled.

The second example, Circle/Square, is a multi-variant scenario where you're testing more than two variants. In this case, you're testing the impact of showing a circle vs. a square shape in your product.

The screenshots in the tutorial are meant to illustrate the process of setting up these tests in the Statsig console. If you're finding the tutorial confusing, it is recommended to follow along in the Statsig console as you read. This should help clarify the process.

The tutorial is designed to use an A/A test as an example to show you how to set up any A/B test easily, of which A/A test is a special case. However, we acknowledge that it can seem a little confusing and we are committed to improving it for the future.

If you have any specific questions about the tutorial or any other aspect of using Statsig, feel free to ask. We're here to help.

When integrating metrics ingestion with Statsig, it is not necessary for developers to apply their own date filters within their SQL queries. Statsig is designed to manage date filtering autonomously during the data backfill process.

Upon setting up data ingestion, developers provide a SQL query that generates a view for Statsig to ingest. This query is then utilized by Statsig to incrementally load data on a daily basis. For the purpose of backfilling, Statsig will inspect the data for the past three days from the current date to determine if there has been a significant change, defined as a variation exceeding 5% in the number of rows.

If such a change is detected, Statsig will automatically initiate a backfill. For data outside of this three-day window, it is expected that the developer will manually trigger a backfill for the desired date range. Therefore, the SQL query provided for data ingestion should not be constrained by row limits, as Statsig will handle the necessary date filtering to ensure efficient and accurate data ingestion.

In A/B testing, the inclusion of non-interacting users in the sample size does not inherently affect the p-values, provided that the metric being evaluated is relevant to the feature being tested. When non-interacting users are included in the sample, it can dilute the experiment results by adding a number of zeros to both the control and test groups.

This dilution can decrease the experimental power, meaning that a larger sample size or a longer experiment duration may be required to reach statistical significance, given the underlying effect size.

However, the directionality of the results should remain correct, and the zeros should be evenly distributed between the groups, maintaining the integrity of the experiment results.

It is recommended to expose users to the experiment as close as possible to the point where their experiences diverge, to avoid overexposure and to increase the efficiency of the experiment.

f the overexposed users do not contribute to the metric, they do not introduce any noise and therefore do not affect the experimental power.

It is important to note that if the treatment affects the rate at which people reach the relevant step in the process, manual filtering to users who reached that step could introduce bias and should be approached with caution.

When an experiment is reset in Statsig, it should not alter the overrides configured for it. The expected behavior is that resetting an experiment does not modify layer settings.

However, an issue was identified where resetting an experiment resulted in the overrides for that experiment in the layer being removed. This was not the intended functionality, and a fix has been implemented to ensure that resetting an experiment will no longer affect the layer overrides.

If any unexpected behavior is observed related to experiment resets or layer overrides, users are encouraged to report it for further investigation.

Statsig does not automatically capture any system events, including the app launch event. However, you have the ability to manually log custom events using the logEvent function provided by the SDK. This function allows you to track any event you deem relevant, including the launch of your app.

Here's an example of how to use it:

For Android:

kotlin Statsig.logEvent("app_launch")

For iOS:

swift Statsig.logEvent("app_launch")

Remember to call this function at the appropriate place in your code to track the app launch event.

Statsig has implemented measures to track bot traffic, including search engine crawlers, which can significantly inflate the number of stable_ids, making your user base appear larger. However, these bots typically do not trigger any events and therefore do not contribute to metrics. As a result, this behavior should not impact the computation of lift or the estimate of the confidence interval in most cases.

In some instances, bots may cause SRM issues. If you observe this, it could potentially be bot-related. To mitigate this, you can manually add filters to exclude bots using a targeting gate. For instance, a filter that excludes traffic where the browser contains "bot" can be quite effective.

While Statsig is actively working on a more comprehensive approach to handle bot traffic, it is currently recommended to use any deterministic detection on your side to filter out bot traffic. This can help align the exposure data more closely with other analytics tools and ensure that bot traffic does not affect the Stable ID assignments or the experiment results.

Please note that the impact of bot traffic should be evenly distributed between control and test, so any impact should cancel out. However, if you have further questions or need more specific advice, please reach out for assistance.

Statsig currently does not offer a native integration with Klaviyo. However, the possibility of such an integration is being explored.

Statsig users looking to incorporate Klaviyo events such as email opens, subscriptions, and email clicks into Statsig for the purpose of personalizing their sites may have to wait for future updates regarding this integration.

Additionally, Statsig is considering whether users would like to run experiments directly within Klaviyo on various channels like email and push notifications, or prefer to conduct experiments within their own stack, including web, mobile, and backend platforms.

Statsig does not provide any folder-like features for managing feature gates. However, you can effectively manage your feature gates through the Statsig console. This platform allows you to create, edit, and archive feature gates as per your requirements.

In case you have a large number of feature gates, you can utilize the search and filter options available on the console. These features will assist you in managing your feature gates more efficiently.

Additionally, Statsig offers the use of Tags to help categorize and search for gates within the console. This feature can be particularly useful when you need to manage a large number of feature gates. By tagging your feature gates, you can easily search and filter them, making the management process more streamlined and efficient.

When executing the updateUser function in Statsig, the new user object does not merge with the previous one. Instead, it completely overwrites the previous user object. This is because Statsig operates on the user object passed in each call and does not maintain or update user objects. Therefore, any new information you want to include must be present in the new user object you pass to updateUser.

Statsig does not provide an inherent mechanism to distinguish between new and returning users within its SDK or API.

To differentiate between these user types, developers must implement their own logic to identify whether a user is new or returning. This can be achieved by setting a flag or property within the user's browser or device storage, such as a JavaScript cookie or a local storage flag for web applications, or leveraging device storage for mobile platforms.

Once this identification is made, the information should be included as a user attribute when logging events or checking feature gates with Statsig.

It is important to ensure that this user attribute is passed along with the user object to Statsig to maintain the distinction between new and returning users within the platform's analytics and feature management.

Switching to the native Statsig event logger does not inherently change the synchronization time to Pacific Standard Time (PST), as the synchronization schedule is consistent regardless of the logging method. However, the Statsig client SDK does facilitate more real-time data capture by automatically flushing logged custom events to the Statsig servers every 10 seconds.

For experiment results, Statsig has implemented a more frequent refresh window. During the first day of an experiment, results are computed hourly. After the initial 24-hour period, the results are then refreshed on a daily basis.

It is important to note that these computations are based on a day defined from midnight to midnight PST. For businesses requiring synchronization in Coordinated Universal Time (UTC), Statsig offers the ability to switch to UTC, but this is an option available exclusively for Enterprise customers. This is due to the additional costs associated with maintaining a separate pipeline for UTC synchronization.

In Statsig, user allocation to a holdout group is a predetermined process that occurs automatically when the holdout group is created. This allocation is based on a cryptographic technique using a salt, which is applied to the unit_id (userID or stableID) of the user being evaluated.

As a result, the assignment to the holdout group is stable as long as the user retains the same unit_id. There is no specific trigger required for this assignment to take place. However, it is important to note that while the assignment is predetermined, the actual exposure of a user to a holdout group only occurs when the user encounters a feature gate or experiment that is part of the holdout.

This means that Statsig only becomes aware of the users when they are exposed to a gate or experiment. It is also crucial to understand that if you are managing the assignment for holdout yourself, you will need to manage exposure logging. This approach is not commonly recommended for holdouts due to the risk of polluting the holdout data.

To edit the metrics included in a tag, you can follow these steps:

1. Navigate to your Metrics Catalog.

2. Use the filter in the upper right-hand corner of the table to filter to the tag you want to modify.

3. Remove the tag from the relevant metrics.

This process allows you to easily manage and modify the metrics associated with a specific tag directly from the Metrics Catalog, without the need to go into Project Settings. Please note that any changes you make will only apply moving forward and will not retroactively affect previously calculated metrics.

To enable a holdout group within Statsig, you will need to navigate to the Experiments section of the Statsig console.

Once there, you will find holdouts as a tab within this section. If you encounter any difficulties or have further questions regarding the creation or management of holdout groups, please feel free to reach out for additional support.

To manually test different experiment variants without the necessity of opening a new tab, you can utilize overrides to control which variant is exposed to a QA user or developer. Overrides can be implemented in two primary ways to facilitate manual testing within the same context.

Firstly, you can add the QA user ID to the overrides in the experiment setup. This method allows you to manage the experiment assignment behavior directly from the Console. Secondly, you can use a local override API to set specific overrides via code.

This approach enables you to change the behavior on the fly, similar to modifying a value in local storage. For instance, calling Statsig.overrideConfig in a browser's developer tools window can alter the experiment's behavior immediately.

These methods ensure that manual testing of different variants can be conducted efficiently and effectively within the same browser tab, providing a seamless experience for QA testers and developers alike.

Statsig differentiates data from different environments by the environment tier you specify during the SDK initialization. You can set the environment tier to "staging", "development", or "production". By default, all checks and event logs are considered "production" data if the environment tier is unset.

Experiments and metrics primarily factor in production data. Non-production events are visible in diagnostics, but they are not included in Pulse results. This is because most companies do not want non-production test data being included. If you want to include these, you can log them as regular events. However, non-production data is filtered out of the warehouse and there is no other way to include it.

When initializing, if you add { environment: { tier: "production" } }, this would set your environment to "production", not "staging" or "development". If you want to set your environment to "staging" or "development", you should replace "production" with the desired environment tier.

Pulse results are only computed for “Production” tier events. To see Pulse results, you need: • An experiment that is “Started” (aka, enabled in production) • Exposures & events in production tier • Exposures for users in all test groups • Events/metrics associated to users in test groups

As long as you have initialized using { environment: { tier: "production" } }, your Pulse will compute. This means that even if your code is deployed to staging, as long as you initialize with the production tier, you will be able to see Pulse results.

When utilizing User Count metrics in Statsig for analyzing reports, the handling of metadata is contingent upon the configuration of the metric. If a User Count metric is created based on an event, it will count as one for the user for that day.

However, the metric can be further refined to consider specific metadata values. By configuring the User Count metric to filter for a particular metadata value, only events that contain that specified metadata value will contribute to the count. This allows for a more granular analysis where the focus is on events with certain characteristics as defined by their metadata.

It is important to note that the configuration of the metric determines how metadata is used in the analysis, providing flexibility in how events are counted and reported.

Statsig processes metrics based on a full day in the PST timezone. The system waits until a full day is marked as ready before processing that day's data. This means there is typically a delay of up to 24 hours for the processing of metrics.

This delay is inherent to Statsig's data processing system and is not related to the tier of your plan. Therefore, it cannot be reduced by moving to a higher tier plan. The processing time is designed this way to ensure the accuracy and integrity of the data.

Please note that this delay is a standard part of Statsig's data processing and is not indicative of any issues with the system. It is important to factor this delay into any data analysis or decision-making processes that rely on Statsig metrics.

To create a custom metric to track the sum of your "Donation" event values in Statsig, follow these steps:

1. Navigate to Metrics from the left-hand navigation panel, then to the Metrics Catalog tab. Click on the Create button.

2. In the Metric Type, select "Aggregation". This type allows you to compute the Sum or Average of the value or metadata property of an event type.

3. For the Event Type, select "Donation".

4. For the ID Type, select "userID" or "stableID" depending on your use case.

5. For the Aggregation Type, select "Sum".

6. For the Field, select "value". This will sum up the values you logged with your "Donation" events.

7. You can also add filters if you want to limit this metric to specific conditions, such as a certain currency.

8. Finally, give your metric a name and save it.

Please note that after you create a custom metric, it will not populate until the next day and will not backfill to previous days. Statsig will only calculate it moving forward from the creation date.

To edit the returned JSON for a dynamic config after adding a targeting rule, locate and click on the 'Edit' link adjacent to the 'Returns JSON' text. This action will bring up the JSON configuration editor.

Within this editor, you can freely modify the configuration parameters and their corresponding values as required by your application. After making the necessary changes, ensure to save your updates by clicking on the 'Confirm' button.

This process allows you to tailor the dynamic config to suit the specific needs of your application, even after targeting rules have been applied. If you encounter any difficulties or the editor does not load as expected, please reach out for further assistance.

Enabling a metrics ingestion pipeline in Statsig involves a series of steps to ensure that your data is correctly integrated and utilized for analysis.

Begin by connecting your data warehouse to Statsig, which will require read access to metric and exposure data, write access for results and exposures, and the ability to run jobs and query data. Once connected, define your metric and assignment sources according to Statsig's expected schemas to map the data accurately.

Metrics can then be ingested through various mechanisms, including Server and Client SDKs, HTTP API, Integrations, Data Warehouse Ingestion, or the Data Warehouse Native Solution for enterprise contracts.

If you encounter any issues during the ingestion process, utilize the Metrics Dashboard in Statsig for debugging. After successful data import, you can proceed with feature gate or experiment implementation.

For detailed guidance on connecting your warehouse and defining sources, refer to the Statsig documentation.

When utilizing the HTTP API to log events in real-time, it is crucial to ensure that the events are properly formatted to be recognized by the system. If events are not appearing in the Metrics Logstream, it may be necessary to wrap the events in an array using the events:[] syntax. This is particularly important because events are typically sent in bulk, and the correct formatting is essential for them to be processed and displayed in the stream.

Additionally, it is recommended to use the SDKs provided for various languages, as they handle a lot of the heavy lifting, including performance and reliability improvements. For cases where the SDKs are not an option, such as with certain desktop applications or specific frameworks like Qt, it may be possible to use language bindings to integrate with the SDKs of supported languages. This can potentially save significant development effort and ensure more reliable event logging.

Furthermore, for the best practice regarding experiments and configurations, it is advised to fetch the experiment config at the time when it is needed to ensure accurate exposure logging. The SDKs typically fetch all evaluations for a single user in one shot and cache it for the session duration, allowing for local and instantaneous experiment config or gate checks. If using the HTTP API directly, it is possible to use an 'initialize' endpoint, which is typically reserved for SDK use.

This endpoint fetches all evaluated results for a given user without generating exposure logs at the same time, thus avoiding overexposure issues.

To ensure that employee IP ranges are excluded from all Statsig events and A/B tests, you can utilize holdouts. Holdouts are a feature that allows you to compare a group of users who are not exposed to any experiments against the broader population that is.

By creating a targeting gate with a regex that encapsulates the IP addresses of your employees, you can then use this targeting gate to establish a holdout. Setting this holdout to global will effectively exclude the specified IP ranges from any experiments, gates, or other features.

Additionally, if you maintain a list of user IDs for your employees, you can create a segment using these IDs instead of relying on IP addresses. This segment can then be referenced in the holdout to achieve the same exclusion effect.

It is important to note that holdouts are subject to analysis, which may lead to additional billing due to the insights they provide on the overall impact of experimentation efforts.

When you want to exclude users from a holdout group using a targeting gate, the users should pass the targeting gate. The targeting gate is used to define the users who are included in the holdout group. Therefore, if a user passes the targeting gate, they are included in the holdout group.

If you want to exclude certain users, you would set up your targeting gate such that these users fail the gate. This will effectively exclude them from the holdout group.

In other words, you should create a gate that qualifies users to be in the holdout. If the gate evaluates to true for a user, they will be held out or excluded.

Remember, the key is to design your targeting gate in a way that the users you want to exclude will fail the gate, thus ensuring their exclusion from the holdout group.

When optimizing landing page loading without waiting for experiment configurations, it is recommended to use a custom script approach if you need to pass in specific user identifiers, such as Segment's anonymous ID.

This is because the standard landing page script provided by Statsig does not allow for the initialization with a different user ID. Instead, it automatically generates a stableID for the user, which is used for traffic splitting and consistent user experience upon revisits. This stableID is stored in a cookie and is used by Statsig to identify users in experiments.

However, if you need to synchronize with metrics from Segment using the Segment anonymous ID, you may need to deconstruct the landing page tool and implement a custom version that allows you to set the userObject with your Segment anonymous ID.

Additionally, you can enrich the Segment User with an additional identifier, such as statsigStableID, which can be obtained in JavaScript using statsig.getStableID(). This ID can then be mapped from the Segment event payload to Statsig's stableID. If performance is a concern, you can bootstrap your SDK with values so you don't have to wait for the network request to complete before rendering.

This can help mitigate performance issues related to waiting for experiment configurations. For more information on bootstrapping the SDK, you can refer to the official documentation on bootstrapping.

Yes, it is possible to preserve the Statsig Stable ID across different platforms such as mWeb and mobileApp. The Stable ID is generated for anonymous use cases and is persisted in local storage. For signed-in users, consistency is ensured across different surfaces, such as mobile app vs desktop, as long as there's a userID used across them.

If you're experiencing issues with StableID changing across subdomains or platforms, one possible solution is to store the StableID in a cookie and reuse it. You can store and retrieve your own id and use the overrideStableID method on the SDK to send it to Statsig. Your overridden id will always take preference. This way, you can set the StableID for not logged in users and your in-house id for people that are logged in.

Here's an example of how you can override the StableID:

javascript await Statsig.initialize(  "client-sdk-key",  null,  {    overrideStableID: '<passed via cookie or via db lookup>'  } );

This should ensure that both your marketing site and main app use the same StableID, allowing you to run experiments that span both sites.

To reconfigure or disable the Slack integration in the Statsig Console, you must manage the integration directly from Slack. There is no option within the Statsig Console to disable the Slack integration.

To reset the channel for notifications, you can attempt to click 'Add to Slack' again, which should allow you to reconfigure the settings. If 'Add to Slack' is not clickable, or if you encounter issues with event filtering not working correctly, you may need to remove the Statsig Slack App from your Slack workspace and then re-add it.

This can be done through the Slack app management area. Once removed, you can re-add the integration via the Statsig Console to set up the desired notifications and configurations.

If you're experiencing a continuous network error while running Statsig 0.23.0 in a Python 3.9 environment, it's important to first understand the environment in which you're running the Statsig SDK and the version you're using.

If the error is occurring locally and continuously, the Statsig SDK will keep retrying and returning the same error. To debug this, you can send a curl command to our events API from your local environment. Here's an example of how to do this:

bash curl  -X POST \  'https://events.statsigapi.net/v1/log_event' \  --header 'statsig-api-key: <YOUR SDK KEY>' \  --header 'Content-Type: application/json' \  --data-raw '{"events": [{"user": { "userID": "1" }, "time": 1667512336, "eventName": "test_event"}]}'

This command sends a test event to the Statsig events API. If this command works, it means your local environment can successfully send requests to the Statsig API. If it doesn't, you may need to check your local network settings, such as DNS or ad-blockers, which could be blocking requests to the API.

Remember, the Statsig SDK is designed to retry sending events if errors are encountered. If you're still experiencing issues after trying the above steps, please provide more details about your environment and the error you're encountering for further assistance.

Retrieving all experiment values for a specific user in a single server-side call can be achieved using the Statsig server SDKs.

The SDK allows for the evaluation of experiments on the fly for a given user without incurring a network roundtrip. This is particularly useful for server-side rendering scenarios, such as with Next.js, where you may have both server-side rendered (SSR) and client-side rendered (CSR) pages and wish to avoid wrapping your entire application with a CSR provider.

By using the server SDK, you can download the entire configuration and evaluate experiments for each user server-side. This approach scales well as it does not require a network call for each experiment evaluation, ensuring efficient performance even as user traffic increases.

For more information on how to implement this in your application, you can refer to the Node.js server SDK documentation provided by Statsig.

To roll out and monitor multiple new features simultaneously using Statsig, you can utilize the platform's Feature Gates and Experiments.

For each new feature you plan to roll out, create a corresponding Feature Gate. This approach automatically converts a feature roll-out into an A/B test, allowing you to measure the impact of the roll-out on all your product and business metrics as the roll out proceeds.

If you wish to test hypotheses between product variants, create an Experiment. An Experiment can offer multiple variants and returns a JSON config to help you configure your app experience based on the group that the user is assigned to.

To show features randomly to different sets of customers, use the 'Pass%' in a Feature Gate and 'Allocation%' in an Experiment. This allows you to control the percentage of users who see each feature.

Statsig's Experiments offer more capabilities for advanced experiment designs. For example, you can analyze variants using stable IDs for situations when users have not yet signed-up (or signed-in), or using custom IDs to analyze user groups, pages, sessions, workspaces, cities, and so on. You can also run multiple isolated experiments concurrently.

Remember to define your company goals and key performance indicators (KPIs) to measure the success of your features. You can break down these strategic goals into actionable metrics that you can move with incremental, iterative improvements.

If you use three different feature gates, you will find out how each feature, individually, performed against the baseline. If you want combinatorial impact analysis, like, A vs B vs C vs AB vs BC vs AC vs ABC, then you will need to setup an experiment with 7 variants and specify the combinations via parameters and measure.

However, in practice, this level of combinatorial testing isn’t always fruitful and will consume a lot of time. A pragmatic recommendation would be to use feature gates to individually launch and measure the impact of a single feature, launch the ones that improve metrics and wind down the ones that don’t.

When running experiments with both anonymous and logged-in users using Segment and Statsig, it is crucial to have a common identifier to track user interactions accurately.

For anonymous users, you can either pass the stable ID generated by Statsig to Segment or generate a stable ID and pass it to Statsig. This ensures that there is a consistent identifier across both platforms.

For logged-in users, you can use the Segment userID, which should be included in the userObject during initialization. This setup allows Statsig to recognize both anonymous and logged-in users in the experiment. To facilitate this, you may need to provision a new custom ID, such as segmentAnonymousId, and map Segment's anonymousId to this new custom ID in the integration settings.

This approach ensures that exposure events sent via the Statsig SDK and custom events sent via Segment are associated with the same user, regardless of their logged-in status.

In order to notify users before they are exposed to a new feature, you can create a separate feature gate to control the rollout percentage and a segment to contain the account IDs that can be exposed to the feature. This approach allows you to effectively manage the rollout process and ensure that users are notified in advance.

The main feature gate will only pass for the accounts in the segment that also pass the separate feature gate. This provides a clear distinction between the users who are eligible and those who have been exposed to the feature.

Here's a brief overview of the process:

1. Create a main feature gate (rollout_feature_gate). The users that pass this gate will be exposed to the feature.

2. Create a separate feature gate (exposure_eligibility_gate) to control the rollout percentage. The users that pass this gate are the ones eligible to be exposed to the feature.

3. Create a segment (allegeable_accounts) that contains all the account IDs that can be exposed to the feature.

The rollout_feature_gate will return pass only for the accounts in the allegeable_accounts segment, while questioning the exposure_eligibility_gate. After a certain amount of time, export all account IDs in the exposure_eligibility_gate to the allegeable_accounts segments, and increase the exposure_eligibility_gate percentage.

This approach allows you to have a distinction between the eligible users that were exposed to the feature (allegeable_accounts segment) to the ones that are allegeable but potentially not yet exposed to it (exposure_eligibility_gate). You can manage additional rules and environment conditions in the main feature gate (rollout_feature_gate).

Remember to test your setup thoroughly in a pre-production environment before rolling it out to ensure everything works as expected.

If you encounter an error message stating This cohort has failed to sync because the API key that was provided no longer has access to the partner. Please check the access level on the partner's website concerning this integration. while integrating Amplitude with Statsig, it suggests that there might be an issue with the Server Secret Key you're using.

Here are a few steps you can take to troubleshoot this:

1. Double-check the Server Secret Key: Ensure that you've copied the correct Server Secret Key from the API keys page in your Statsig Project Settings.

2. Check the key's access level: Make sure that the Server Secret Key you're using has the appropriate access level for the integration.

3. Re-enter the key: Try re-entering the Server Secret Key in Amplitude and save the changes.

If you've checked these and the issue persists, it might be a more complex issue. In this case, it is recommended to reach out to the Statsig support team for further assistance.

To view statistics for multiple landing pages separately in Statsig when analyzing a test, you need to set up separate experiments for each landing page. This involves adding a unique script to each landing page you're testing. The script should look like this:

<script src="https://cdn.jsdelivr.net/npm/statsig-landing-page-exp?apikey=[API_KEY]&expid=[EXP_NAME]"></script>

Replace [API_KEY] with your Client API Key and [EXP_NAME] with the unique Experiment Name for each landing page.

Once you've set up and published your landing pages, you can monitor the setup by navigating to the Diagnostics Tab in your experiment and looking at the 'Exposure Stream' at the bottom of the page. This shows a real-time stream of the page loads along with the variant they were allocated.To interpret the results, go to the Pulse Results tab in the Experiment page. Here, you can add metrics you want to monitor and verify which variant is doing better.

Remember, you can extend this setup to run ABC or ABn tests by adding more variants in the Experiment Setup tab. If you want to run multiple landing page experiments without needing to update the code on the website for each experiment, consider using layers.

Additionally, you can find group level stats in the hover card. Just hover over the metric and click "view details".

In the context of event-based metrics, particularly the event_dau metric, it is important to understand how users are counted over multiple days. The event_dau metric is designed to count the number of distinct users or devices that have triggered a specific event on a given day.

This means that if an individual user triggers the same event on consecutive days, they will be counted as a unique user for each day that the event is triggered. The metric is focused on daily activity, and as such, it does not aggregate users over multiple days but rather provides a snapshot of unique interactions for each day separately.

Yes, it is possible to view the scorecard metrics of an experiment for a custom date range that starts after the experiment start date. This can be achieved by creating an Explore report with a specific date range. The report will filter the events and metrics to be within that range. However, the users included in the report will also encompass those exposed before the specified timeframe. This is designed to capture the downstream effects of the experiment treatment.

Here are the steps to do this:

1. Navigate to the Explore tab.

2. Fire off a custom query with your desired date range.

If you're having trouble selecting a date range that starts after the experiment start date, it might be a technical issue. Check your inputs and try again. If the issue persists, please flag it for assistance.

When considering the removal of preexperimental bias from a feature that was rolled out with limited initial data, resalting the feature gate is not specifically recommended for this purpose.

Pre-experimental bias can occur by chance and may not always significantly impact the results of an experiment. If the overall difference in metrics due to this bias is small, it might not meaningfully affect the interpretation of the results.

In such instances, the bias can be acknowledged as a cautionary note, and the affected metrics should be treated with a degree of skepticism. To address concerns about pre-experimental bias, it is advisable to utilize the 'Days Since Exposure' view.

This tool assists in identifying any novelty effects and pre-existing experiment effects. Should bias be detected, users will receive a notification, and a warning will be displayed on the relevant Pulse results. This approach allows for a more informed analysis of the feature's impact without the need to resalt the feature gate.

Statbot is specifically trained on the Statsig documentation. It does not have access to the codebase. The bot is designed to provide accurate and reliable information based on the documentation. It's important to note that the documentation may lag behind the codebase.

In addition to the documentation, Statbot is also constantly learning from the answers provided by humans in public channels. This allows it to continually improve its responses and provide more accurate and specific coding recommendations. However, it's crucial to remember that the bot's knowledge is based on the documentation and human-provided answers, not the codebase itself.

The Pro tier is structured on a usage-based pricing model, specifically focusing on the volume of events. It is not a flat rate per user but rather a cost that is determined by the number of events generated. The Pro tier is priced at $150 per month for up to 5 million events.

Beyond this threshold, additional events are billed at a rate of $50 per million events. Metered events are the key factor in this pricing structure and include any instance where a user is assigned to a feature gate or experiment, which is recorded as an 'exposure event.' These exposure events are deduplicated within certain time windows to ensure accurate billing For client-side SDKs, the deduplication window is one hour, and for server-side SDKs, it is one minute.

Additionally, custom events logged through the SDK, inbound integrations, or data warehouse rows are also counted as metered events.

At present, we do not provide a downloadable JSON version of our OpenAPI/Swagger documentation for the Console API. However, we appreciate your suggestion and have added it to our backlog for consideration. We will prioritize this feature request along with the rest. Please stay tuned for updates on this feature.

When transitioning from event import to metric import, it is essential to understand the ingestion status and the steps required to archive metrics that are no longer needed. After switching to metric import, if you have precomputed metrics, ingesting these metrics directly can be more efficient than importing raw events.

Once the change is made, there may be a need for adjustments on the backend to accommodate the new import method. This initial setup is a one-time speedbump, and once resolved, metrics should start appearing as expected. If you notice that some metrics have complete data for a specific date range while others do not, it may be due to the ingestion process still being underway or a need for a backfill to import historical data.

For metrics generated from the previous event ingestion job, such as event_count and event_dau types, consider archiving them if they are no longer relevant to your current data structure and needs. Archiving these metrics can help maintain a clean and efficient data environment.

In Statsig, the Daily Participation Rate and One Time Event metrics are used to track user behavior in experiments. The choice between these two metrics depends on the nature of the event you're tracking.

1. Daily Participation Rate: This metric is calculated as the total number of days that a user has the selected event, divided by the number of days the user is in the experiment. This is done for each user in the experiment. The mean event_dau, or the average active days per user, is then calculated by aggregating this average event_dau for each user in the experiment, with each user weighted equally. This metric is more suitable for events that are expected to occur repeatedly for a given user.

2. One Time Event: This metric is ideal for events that are only expected once per user, such as booking events. If the event is expected to occur only once per user during the experiment or holdout period, then the One Time Event metric would be suitable.

For longer experiments and holdouts, the choice of metric would still depend on the frequency of the event. If the event is expected to occur approximately once a month or less frequently, the One Time Event metric should be appropriate. However, if the event is expected to occur approximately weekly or more frequently, the Daily Participation Rate metric might be more appropriate as it captures recurring behavior.

When reviewing experiments, consider all related metrics:

- One-time events best capture the number of unique users who participated in the event. - Daily participation rate is an effective proxy for "how much" people are participating in the event. - Total events (event_count) is a better proxy for revenue or downstream metrics.

For holdouts, it can be helpful to use different rollups. For example, looking at one-time metrics for the 7-day or 28-day rollup would tell you what % of users participated (at all) within the last 7-day or 28-day window. This can be an effective way to get past the history issue.

When utilizing a client SDK that involves experiments with targeting gates based on dynamic user attributes, there is a mechanism to handle changes in user eligibility. If a user's attributes alter in such a way that they no longer meet the experiment's targeting criteria, the client SDK can be configured to store the initial evaluation results in local storage.

This ensures that the user's experience remains consistent, and they continue to have access to the feature or release associated with the experiment, despite any subsequent changes in their dynamic attributes.

This approach is beneficial for maintaining a stable user experience and ensuring that the integrity of the experiment is not compromised by fluctuating user states.

When integrating the Statsig Node.js SDK, developers often inquire about the initialization time required on the server. The initialization process involves a network request, and the time taken for this process can vary based on several factors.

A key consideration is the geographical distance between the server where the SDK is being initialized and Statsig's data centers. Typically, the server latency for handling initialize calls is in the range of 200 milliseconds to 1 second.

It is important to note that project size can influence initialization time, particularly when using large ID list based segments, which require downloading during the initialization phase. Without such segments, the aforementioned latency range is a reasonable expectation for initialization time.

Statsig currently supports metrics calculation for a single production environment. If you have multiple environments set up, you will only be able to use one as your production environment for metrics and experiments.

When running an experiment, the default behavior is that the experiment will run in all environments. This means that if you try to access the config values of the experiment, you will get a valid value, even if you are in a non-production environment.

If you want your experiment to only run on production, you can set a targeting gate. This will ensure that only users in the production environment will pass the gate and be included in the experiment.

Here is an example of how you might access the config values of an experiment:

javascript useExperiment(XXX).config.get(YYY, false);

In this example, XXX is the experiment you are running, and YYY is the config value you are trying to access. If the experiment is not running in the current environment, you will get the default fallback value, which is false in this case.

Remember, once you start an experiment, it will run in all environments unless you set a targeting gate to restrict it to specific environments.

The best practice is to use the experiment checklist and diagnostics tab to instrument the test, enable it in lower environments, and validate that exposures are flowing through. Then, when you’ve validated these, you click “Start” to go to production. This workflow is typically adequate for most users.

Please note that this is subject to change as Statsig continues to evolve and add new features. Always refer to the official Statsig documentation for the most up-to-date information.

In the context of Pulse results, the cutoff time for data pertaining to the previous day is established at 11:59 PM Pacific Standard Time (PST).

This timeframe ensures that all data collected falls within the standard daily cycle, which begins at 12:00 AM PST and concludes at 11:59 PM PST of the same day. It is important for users to align their data analysis and reporting activities with this cutoff to accurately reflect the results for each individual day.

In the context of Bayesian testing, the credible interval is the Bayesian equivalent of the confidence interval. Specifically, it is reported for the conversation delta for your conversion rate.

As for the interpretation of expected loss, it is computed as the expected value of delta when delta is negative, multiplied by the probability of delta being negative. This can be loosely interpreted as the risk to your metric if you launch the test group.

The daily process that updates experiment metrics for the previous day typically completes by 9am PST. This process ensures that all data for newly added metadata dimensions in your experiment metrics are calculated and updated. Please note that the exact timing may vary, and for more specific information, you may need to consult with a Statsig team member.

If you're having trouble locating the entry point for holdouts in your dashboard, please note that it has been moved to a tab within experiments. This change might require some adjustment, but rest assured, all the functionalities remain the same. If you're unable to locate it within the experiments tab, please reach out for further assistance.

If you have any inquiries regarding billing for your account, please direct your questions to our dedicated support team. They are equipped to provide the necessary assistance or connect you with the appropriate sales representative to address your concerns.

To get in touch, send an email to the support team.

If you're using the dynamicConfig.get("key", defaultValue) method to fetch a value from a Dynamic Config in Statsig and receiving null, it could be due to several reasons:

1. The key does not exist in the Dynamic Config: Ensure that the key you're trying to fetch is correct and exists in the Dynamic Config.

2. The Dynamic Config has not been initialized or fetched correctly: Check the initialization and fetching of the Dynamic Config. Make sure it has been done correctly as per the Statsig documentation.

3. Issue with the default value provided: There might be a problem with the default value you've provided. Verify that it is correct and compatible with the expected value type.

If you've verified the key, checked the initialization and fetching of the Dynamic Config, and ensured the default value is correct, but you're still receiving null, it might be a more complex problem that requires further investigation. In such cases, we recommend flagging this issue for further assistance from the Statsig team.

In order to enable reviews for changes that will impact your production environment, you need to follow certain steps. This includes changes to a dynamic config or gate, which affect rules with the production environment or apply to all environments. It also includes changes to an experiment that has started.

The philosophy behind this approach is to strike a balance between allowing developers to move quickly in a local environment, while ensuring that any changes that will impact end users will require a review.

Here's how it works:

1. Reviews are only required for changes that will have an effect on the production environment. This means that if you're making changes to a dynamic config or gate, reviews will only be required if these changes affect rules with the production environment or apply to all environments.

2. For experiments, reviews are only required for changes to an experiment that has started. This is because these changes could potentially impact your end users.

If you're seeing behavior that doesn't align with these rules, it's possible that there's a technical issue that needs to be addressed. However, in most cases, following these guidelines should ensure that your changes are reviewed appropriately before they're implemented in a production environment.

When utilizing a service with a Free tier event limit, reaching this limit does not inherently affect the visibility of non-production logs. Non-production logs are intentionally not displayed in the console by default to ensure they do not impact the results of experiments.

To view non-production events in the log stream for a production event, users can toggle the appropriate setting within the console. For testing purposes, events can be suffixed with '_test' and logged using the production environment. This allows for the testing of event logging without affecting production data. Once testing is complete and the event logging is to be finalized, these suffixes can be removed.

If non-production logs are still not visible after these steps, it is likely that there is an issue unrelated to the Free tier event limit. Users are encouraged to seek further assistance to resolve any such issues.

If you're unable to see the Holdouts feature in your Statsig console, it's possible that it has been relocated. The Holdouts feature has been moved to the navbar under experiments. Please check there to access this feature.

If you continue to experience issues, please contact our support team for further assistance.