🕒 Scheduled Reloads

You can now configure default reload schedules for Experiment Results and Metrics and apply them to existing entities. You can continue to also just configure them on each entity.

This feature is relevant only to Statsig Warehouse Native.

✅ Verified Metrics

Enterprises often have a set of curated, centrally managed metrics in addition to team specific metrics. You can now mark the curated metrics as "verified" so experimenters can tell them apart.

Advanced Product Analytics with Event-Based Cohorts in Metrics Explorer

You can now perform detailed analysis on almost arbitrarily specific user segments with our new Event-Based Cohorts feature in Metrics Explorer. Event based cohorts allow you to group users who performed certain events and share specific properties. You can specify the minimum, maximum, or exact number of times users in the cohort performed the given event, and specify the date range within which they performed it. You can also add multiple property filters to the cohort. This is useful in many scenarios:

• Create multiple cohorts of interesting user segments and compare their product usage. You can add multiple cohorts to your group-by, and use it was a way to compare different segments of users. For example, you can use the Distributions chart to find the usage that represents the 90th percentile for some event/feature of interest, and then create a “power user” cohort in a Drilldown chart by setting the event frequency to that 90th percentile. You can then create an “all users” cohort and compare the two.

• Filtering by a Cohort. Define an event based cohort and use it as a way to filter your analysis. For example dig into low engagement users by filtering you cohort who used a feature at most 1 time in the last month.

Get started with this new feature by going to Metrics Explorer (click on the Metrics tab in the left navigation menu), mousing over to the Group-By section and clicking “+” button and selecting “Compare Cohorts” to begin defining your cohort.

Define and compare interesting user segments with event-based cohorts.

Analyze Product Metrics by Experiment Group in Metrics Explorer

One of the most valuable aspects of any analytics product is illuminating how your product is performing for different groups. This is useful for general product understanding (is some key product metric over-performing for one group of users vs another?), debugging (is some key perf metric spiking for a specific group), and detailed segment analysis (what’s going on for a specific product feature for macOS 14.1.0 users in Seattle?). Doing these type of analyses for users in different experiment groups hasn’t really been possible until now.

In our product analytics surface, Metrics Explorer, you can now select any metric and split the metric out by experiment group. This unlocks many powerful scenarios such as getting a general sense of how a metric is performing for different groups in experiment, viewing the long term effect of an experiment on different groups, or monitoring and debugging the performance of different experiment variants.

Try out this feature by navigating to Metrics Explorer and clicking on the “Metrics” tab in the navigation bar on the left. Select the metric you are interested in, add a “Group-By” and select “Experiment Group”. Now choose the experiment of interest and see how the metric performance varies between groups in an experiment. You can do all the analysis you expect from Metrics Explorer like adding property filters, changing views (stacked lines, bar charts, etc), or scoping to a specific event based cohort.

An example metric split-out by experiment group and showing outsized performance for one group.

We’ve started rolling out a new health check on experiments (gates coming soon) to help teams more easily catch any SDK configuration issues that may be impacting experiment assignment.

The new “Group Assignment Health” check surfaces if there are a high percentage of checks with assignment reasons like "Uninitialized" or "InvalidBootstrap" which might indicate experiment assignment is not configured correctly. You can view an hourly breakdown of assignment reason via the View Assignment Reasons CTA.

To read more about what each assignment reason means and how to debug, see our docs here.

Example Assignment Reason Breakdown

New Year, New Us

In prep for starting 2024 on the right foot, our team spent the last few weeks of 2023 cleaning up and polishing some of the most loved surfaces of the Statsig Console. We're excited to debut a set of shipped improvements to you today! 🎉

Here's what's changing:

🏠 Home Tab 2.0

We’ve given the Statsig Home Tab a facelift! A few of the changes we’ve implemented:

• Added a personalized “to-do” list to the top of your feed, enabling you to easily catch up on all the items that need your attention in the Console

• Moved the Velocity charts into a side panel; these are still accessible on-demand when you want to understand how your team’s velocity is tracking, but aren’t as in-your-face every time you log into Statsig

• Made metrics tracker more flexible- now pin any tag (not just ⭐Core) that you’re curious about tracking regularly to see those metrics pinned to your sidebar

☰ Left Nav

You may have noticed your Left Nav is looking a little leaner these days- we moved two tabs (Holdouts and Autotune) into Experiments tab, alongside Experiments and Layers. As we continue to build new experimentation types, we will consolidate them here, under the umbrella “Experiments” tab.

⚙️ Project/ Org Settings Unification

We’ve unified the surfaces that your Account Settings, Project Settings, and Organization Settings live into one “Settings” tab, making admin-related tasks easier from one central spot in the Console.

🔎 Filters Refresh

As your team’s library of metrics, experiments, and new feature launches grows on Statsig, being able to organize and easily find the entity you want at any given time is crucial. To make this even easier, we’ve invested in leveling up our filter UX, improving discoverability and usability, as well as exposing operators such as “any of” and “all of” for fields like Tags.

New Filters UX

✨ New & Improved Custom Queries

Today, we’re starting to roll out a complete refresh of our Custom Queries feature. Here’s what’s changing:

1. Performance-Custom Queries are a key part of analyzing your experiment results. We’ve brought down query time from 10-15 minutes to 2-3 minutes, speeding up your iteration cycle times and ensuring you can get to the answer you’re looking for faster.

2. Query UX- We’ve completely overhauled the Custom Query UX to feel closer to our MEX query UX, so this should feel like a familiar interface to navigate!

3. Filter/ Group by Event Dimensions- In addition to filtering/ grouping by User Dimensions, we’ve added the ability to filter/ group by Event Dimensions.

4. Improved Detail View- Now, you can see the details of your Custom Query Pulse results via a hovercard similar to what you see in the Scorecard, with raw stats, time-series (coming soon), and topline projected impact.

Some aspects of Custom Queries are remaining the same, such as the ability to save, name, and share your historical queries, as well as the ability to schedule a Custom Query to run daily in the “Scheduled” tab.

Read more about the new Custom Queries product here, and let us know if you have any feedback!

Custom Query UI

Enhanced Formulas in Metrics Explorer

We are happy to announce that you can now do math with your metrics! Metrics Explorer now features new formula capabilities, empowering you to delve deeper into your data with ease.

With this new functionality, you can instantly create and visualize dynamic combinations and transformations of one or more metrics. Whether you're calculating event frequency per user, exploring the relationship between different metrics, or requiring a logarithmic perspective of your data, these insights are now just a few clicks away.

Our enhanced formulas include basic mathematical operations, logarithmic functions, square roots, and more (see our full range here). Moreover, you can now effortlessly add trendlines to your analysis, and any metric in your query can seamlessly integrate as a variable in your formula.

Ready to transform your data analysis? Simply hover over the “+” sign in the Metrics Explorer to start adding and experimenting with formulas.

Visualize the relationship between metrics on the fly. For example, we can now visualize purchase events per user in an ad-hoc way.

📣 Interactive Experiment Summaries

We’re thrilled to announce the launch of a new, interactive “Summary” tab for Experiments. With Experiment Summary, you can collect all implementation details and the final metric lift results in one place, note down team discussion and action items, and create an enduring artifact of all the learnings your team is taking away from your recently-run Experiment.

You can add to a draft state of your Experiment Summary at any point during the Experiment and then once a decision has been made, the Experiment Summary will become the default tab. You can also export your Experiment Summary to a PDF to share with the broader team.

Image of an experiment summary document

🧮 Improved Power Analysis Calculator

Today, we’re starting to roll out a set of improvements to the Power Analysis Calculator. Here's what's changing:

1. Improved UX:

   The new Power Analysis Calculator is a full-blown hub for creating, storing, and looking up previous power analysis calculations.

2. Qualifying event audience generation:

   Now you can use an event as a qualifying threshold to define the audience you would like to run a power analysis on. For example, if you’re an ecommerce company and would like to run a checkout experiment you could use a “tap_checkout" event to define the audience you want to calculate power for your experiment on.

3. Past analyses:

   We’ve introduced a new tab “Past Analyses” into the Power Analysis Calculator, where all previous calculations will live. You can rename these analyses for easy lookup/ collaboration, and view the results inline (as well as play with parameters like MDE and target allocation inline without submitting a new calculation). Each past analysis has a Share Link for easy sharing with your team!