Frequently Asked Questions

In the context of adding filters to metrics within a funnel, it has been confirmed that this functionality is indeed supported. Users are able to apply filters to Event_count and User_count metrics, which can then be utilized in the creation of a funnel.

This provides the flexibility to tailor metrics to specific needs and answer targeted questions effectively. An example provided includes the creation of a “Product View Sale” metric with an applied filter, which is then visible and operational within the funnel creation workflow.

It is important to note that while filters can be added, they are limited to the aforementioned metric types.

In the current setup of Statsig, there is no direct method to create a feature gate or a user segment based on a specific event occurrence. Feature gates and user segments are typically created based on user attributes or conditions.

However, you can log custom events using the Statsig SDK and then analyze these events in the Statsig console. While this doesn't directly influence the feature gate or user segment, it can provide valuable insights into user behavior that can inform your decision-making process.

For a more automated solution, you might consider using the Console API. While it doesn't directly support event-based targeting, it can help streamline the process of checking for specific user attributes in gates or segments.

It's important to note that the Statsig SDKs are designed to be performant, meaning they make bucketing decisions based on rules/config they have, and can’t make async calls to look up state about a user as part of making this decision.

If you need to gate based on a user's state, you would need to manage the state of a user property (for example, isSubscriber?) and pass this along in the user object for the SDK to be able to gate based on this state.

For further questions or more specific assistance, it is recommended to reach out to the Statsig team directly. They may be able to provide more detailed guidance or discuss potential future features.

Creating a sequential funnel that includes a step with multiple events is not directly supported as funnel metrics are designed to be linear, with each step corresponding to a single event. However, you can gain similar insights by utilizing Ratio Metrics.

Ratio Metrics allow you to define a metric using other metrics as the numerator and denominator. This means you can create a metric that combines the counts of multiple events, which can provide an analogous understanding to having two events count as the same step in a funnel metric.

To implement this, you would define a new Ratio Metric where the numerator is the sum of the events you wish to combine, and the denominator is the preceding event in the funnel. This approach allows for flexibility in analyzing user behavior across multiple events without altering the linear structure of funnel metrics.

When updating log events for a feature gate in Statsig, there is no need to restart the feature gate to see the revised metrics data, as changes should take effect immediately. However, if the updated metrics are not reflecting as expected, it is advisable to verify that the events are being logged correctly.

To obtain cleaner data for metrics lifts after modifying the code for a log event, you can adjust the distribution of users between the control and experiment groups or 'resalt' the feature to reshuffle users without changing the percentage distribution.

For experiments with delayed events, setting the experiment allocation to 0% after the desired period ensures that delayed events still count towards the analytics. It is important to note that the sizes of variants cannot be adjusted during an ongoing experiment to maintain the integrity of the results.

To increase the exposure of a variant, the current experiment must be stopped, and a new one with the desired percentage split should be started.

In the context of managing experiments with Terraform, the status field can be updated to reflect one of four possible values: setup, active, decision_made, and abandoned, aiding in the management of the experiment's lifecycle.

For those utilizing Statsig Warehouse Native, creating an Assignment Source and a new experiment allows for the definition of the experiment timeline and subsequent calculation of results.

Statsig pipelines typically run in PST, with data landing by 9am PST, although enterprise companies have the option to switch this to UTC.

Statsig Cloud calculates new or changed metrics going forward, and Statsig Warehouse Native offers the flexibility to create metrics after experiments have started or finished and to reanalyze data retrospectively.

In Statsig, metrics that have been around for a while and contain data can be renamed without losing any historical data. This feature is being rolled out and allows for a more comprehensive approach to renaming metrics while preserving their associated data.

If you have a metric that requires renaming and you wish to retain its data, you can request to have your project included in the rollout of this new feature. Once your project is included, you will be able to rename your metrics accordingly.

It is recommended to provide feedback or report any issues encountered during the use of this new renaming feature to ensure continuous improvement and resolution of potential problems.

Currently, the feature to split L7 (Last 7 days), L14, and L28 metrics by different groups, segments, or cohorts is not available.

This functionality is under active development, and considerations are being made to include it in future updates. As of now, users looking to analyze metrics by these specific time-bound parameters will not be able to use the 'Group By' feature to disaggregate the data by properties such as user type, geographical location, or other custom parameters.

This limitation applies to the built-in metrics within the platform. Users are encouraged to stay tuned for updates on the development of this feature, which aims to provide more granular insights into user behavior and metric performance over specified time frames.

In the context of updating experiments and managing permissions for editing monitoring metrics, it is important to understand the current capabilities of role-specific permissions.

According to the expert, there is a setting within the 'Members & Roles' section under 'Project' that allows for role-specific permission adjustments. By navigating to the 'Roles' tab and selecting a 'Member' role, users can access the 'Permissions' tab where they can find a setting called 'self-approve review.' This setting enables a role to bypass the review structure entirely.

However, it is noted that there is no current permission setting that allows for the customization of permissions to the extent of enabling a user to edit monitoring metrics specifically without granting them the ability to bypass all reviews.

The expert acknowledges the validity of this use case and has indicated that they will discuss the possibility of implementing such a feature with their team. As of the expert's last statement, permissions for general metric management and verify metrics exist, but not for editing monitoring metrics specifically.

Currently, the capability to backfill results for a metric that was added midway through an experiment is not available. This limitation is recognized, and plans are in place to address this issue in the future. However, there is no estimated time of arrival for this feature.

As of now, if a new metric is introduced, it will only be calculated from the date of its creation and cannot retrospectively apply to data collected before that point.

In the context of hypothesis testing, the p-value is the probability of observing an effect equal to or larger than the measured metric delta, assuming that the null hypothesis is true. A p-value lower than a pre-defined threshold is considered evidence of a true effect.

The calculation of the p-value depends on the number of degrees of freedom (ν). For most experiments, a two-sample z-test is appropriate. However, for smaller experiments with ν < 100, Welch's t-test is used. In both cases, the p-value is dependent on the metric mean and variance computed for the test and control groups.

The z-statistic of a two-sample z-test is calculated using the formula: Z = (Xt - Xc) / sqrt(var(Xt) + var(Xc)). The two-sided p-value is then obtained from the standard normal cumulative distribution function.

For smaller sample sizes, Welch's t-test is the preferred statistical test due to its lower false positive rates in cases of unequal sizes and variances. The t-statistic is computed in the same way as the two-sample z-test, and the degrees of freedom ν are computed using a specific formula.

While the normal distribution is often used in these calculations due to the central limit theorem, the specific distribution used can depend on the nature of the experiment and the data. For instance, in Bayesian experiments, the posterior probability distribution is calculated, which can involve different distributions depending on the prior beliefs and the likelihood.

It's important to note that it's typically assumed that the sample means are normally distributed. This is generally true for most metrics thanks to the central limit theorem, even if the distribution of the metric values themselves is not normal.

In order to change the directionality of a custom metric in Statsig, you would typically navigate to the Metrics Catalog tab, select the metric you want to edit, and then click on the "Edit" button. However, there was a known issue where the edit icon was hidden until you mouse over it due to a design change. This issue has been resolved and you should now be able to see the edit icon without having to mouse over it.

If you still encounter any issues with changing the directionality of a custom metric, please refresh your page as the fix should be applied. If the problem persists, please contact the Statsig support team for further assistance.

To edit the metrics included in a tag, you can follow these steps:

1. Navigate to your Metrics Catalog.

2. Use the filter in the upper right-hand corner of the table to filter to the tag you want to modify.

3. Remove the tag from the relevant metrics.

This process allows you to easily manage and modify the metrics associated with a specific tag directly from the Metrics Catalog, without the need to go into Project Settings. Please note that any changes you make will only apply moving forward and will not retroactively affect previously calculated metrics.

To filter by a specific user segment within a dashboard or chart, you can utilize the Metrics Explorer feature. Here, you can create or edit a chart and apply filters to individual metrics based on User dimensions.

When you need to filter by a particular user segment, you can select the 'Filter with User ID Segment' option. This feature allows you to define a user segment based on an ID list or set conditions that users must meet to be included in the segment. After applying the necessary filters and configuring your chart, you can export it to your dashboard.

Any modifications made to the chart in the Metrics Explorer will be reflected in the dashboard once saved. Additionally, you can adjust an existing dashboard by entering the edit mode and adding filters or cohorts through the settings panel on the left. This approach enables you to apply user segment filters directly within the dashboard environment.

Statsig processes metrics based on a full day in the PST timezone. The system waits until a full day is marked as ready before processing that day's data. This means there is typically a delay of up to 24 hours for the processing of metrics.

This delay is inherent to Statsig's data processing system and is not related to the tier of your plan. Therefore, it cannot be reduced by moving to a higher tier plan. The processing time is designed this way to ensure the accuracy and integrity of the data.

Please note that this delay is a standard part of Statsig's data processing and is not indicative of any issues with the system. It is important to factor this delay into any data analysis or decision-making processes that rely on Statsig metrics.

The current functionality of the "contains" operator allows for only one target in the condition. If you want to filter on multiple tags, you will need to add more events to your configuration. This can be done by using the "+ Add Another Event" link.

Here's how you can do it:

1. Use the "Filter" option in the Custom Query fields for a metric. 2. Add an individual filter for each tag you want to include in your analysis. 3. To add multiple values in one filter, use the "+ Add Another Event" link to add more events to your configuration.

Please note that the current documentation does not provide a method for adding multiple values in one filter in a more streamlined way. If you need further assistance, please reach out to the support team.

To create a custom metric to track the sum of your "Donation" event values in Statsig, follow these steps:

1. Navigate to Metrics from the left-hand navigation panel, then to the Metrics Catalog tab. Click on the Create button.

2. In the Metric Type, select "Aggregation". This type allows you to compute the Sum or Average of the value or metadata property of an event type.

3. For the Event Type, select "Donation".

4. For the ID Type, select "userID" or "stableID" depending on your use case.

5. For the Aggregation Type, select "Sum".

6. For the Field, select "value". This will sum up the values you logged with your "Donation" events.

7. You can also add filters if you want to limit this metric to specific conditions, such as a certain currency.

8. Finally, give your metric a name and save it.

Please note that after you create a custom metric, it will not populate until the next day and will not backfill to previous days. Statsig will only calculate it moving forward from the creation date.

The current Statsig SDK does not provide a method for bulk querying experiments for a large number of users in a single call. The getExperiment function is designed to be called for individual users. If you need to get data about all experiments for a specific user ID, you can use the getExperiment function with different user IDs as many times as you want.

Please note that calling getExperiment will trigger an exposure for that user on that day. If they weren’t in a test or control group, they would be exposed and count towards experiment metrics from that day forward.

If you have a use case where you need to know the evaluation but do not want to enroll the user in an experiment group, you can use the getExperimentWithExposureLoggingDisabled API.

To determine if a user has participated in an experiment, you need to download the exposure report for each experiment to see which users have checked it. The daily reports API is an enterprise feature that can be enabled for your project.

If you're using an SDK to evaluate experiments, it should be pretty simple to just iterate through and check for each user. This is the recommended route for handling large numbers of users.

When encountering an unexpected spike in the new_dau metric without a corresponding increase in website traffic, there are several potential causes to consider. Firstly, it is important to verify the accuracy and consistency of event logging. Any discrepancies in how events are logged can lead to anomalies in the metrics.

Secondly, if data is being imported from external sources, ensure that the ingestion process is functioning correctly and that data is imported in a timely manner.

Additionally, be aware of timezone differences that may affect the interpretation of metrics; Statsig uses Pacific Standard Time (PST) as its default, which could lead to discrepancies if other systems are set to different timezones.

Lastly, the use of user identifiers should be consistent across all events and sessions to maintain the integrity of the data.

If these checks do not resolve the issue, further investigation into specific events that correlate with the metric increase may be necessary. It is also advisable to reach out directly for support if the cause of the spike remains unclear.