Product Updates

Reviews at Various Levels, Metrics Bulk Management, New Users Tab, and More!

Starting the week off strong with a bevy of launches. As always, feel free to message us with questions/ ideas/ feedback. 

Enable Reviews at XP/ Gate/ Config Level

To-date reviews have been enabled at the Project level, with no ability to set more granular controls for specific configs. This week we’re rolling out the ability to require reviews at the entity-level, even if reviews are not required at the Project-level. This capability is controlled via the “…” menu and is available for Experiments, Gates, Segments, and Dynamic Configs. 

Metrics Bulk Management

To make metrics management within the Metrics tab more streamlined, we’ve added bulk actioning on metrics. Bulk actions include tagging, hiding, or comparing all selected metrics. 

Using Tags in Experiment Scorecard Metrics

In addition to adding individual metrics to your Primary and Secondary Metrics sections within the Experiment Scorecard, you can now add metric tags directly. Adding a tag to the Scorecard will automatically add all metrics in the tag to your Pulse results. 

Exposures in Events Explorer

To continue to make debugging easier using Events Explorer, we are adding the ability to attach Experiment/ Gate exposures to events. Exposure annotations are controlled at the data settings level- to specify which Gates/ Experiments you want to see events annotated with exposure logs for, go to the “Data Settings” dialog at the top of Events Explorer and select up to 5 gates/ experiments. You will now be able to see pass/ fail/ group name on every event in Events Explorer for the gate/ experiment. Statsig starts logging these exposures with your application events after you update the Data Settings and doesn’t backfill exposures for past events. 

New Users Tab UX

The User's tab enables customers to diagnose issues for specific users. Previously, the Users tab listed your application's users as of the previous day. Now, you can you can query for a user who used your application within the last hour. To query for a user simply tap on the “+ Load User” CTA in the upper lefthand corner. A history of all queried users enables easy access to be able to go back and find previously-queried users.

Images for Experiment Groups

Just as critical as a good experiment creation experience is a good experiment viewer experience. To that end, we’re launching the ability to add images to each group in an experiment to better convey the changes between Control and Treatment(s).

Adding Group Images

For experiment creators, simply tap on the image icon next to each experiment group. You can add multiple images to each experiment group to convey the full context of that variant experience.

Viewing Group Images

To view the images associated with each group, simply tap on the “View Images for Each Test Group” CTA in the upper right-hand corner of the Metric Lifts unit above the Hypothesis.

Launch Announcements: Metrics Logstream and Experiment Checklist

Metrics Logstream

Late last week, we launched an additional logstream on the “Metrics Catalog” tab within “Metrics” to provide more visibility and easier debugging for pre-computed metrics being ingested via our API or one of our integrations (Snowflake/ Redshift/ Bigquery, etc.) NOTE- this additional logstream will only show up if you're ingesting pre-computed metrics.

This is the first part of a multi-step project to improve our pre-computed metrics ingestion experience, from setup through to ongoing usage and debugging. Stay tuned for a slew of improvements in the coming weeks… (and if you have feedback on this process/ specific pain points, don't hesitate to ping me directly!) 

Experiment Checklist

As usage of the Statsig platform grows within teams, we’re seeing more and more first-time experiment creators. To this end, we’ve improved our “Setup Checklist” in the “Setup” tab of each experiment. The new checklist includes additional functionality to test your experiment variants inline using ID-based overrides, as well as the ability to test your experiment allocations as they will appear in Production before even starting your experiment.

Note that the new checklist is entirely optional and can be collapsed for our pro experimenters who have been around the block a few times. 

Product Updates: Experiment Scorecard and CUPED

Rounding out the week with two exciting product launches! As always, don't hesitate to reach out here or 1:1 with product feedback, ideas, questions, etc. We love to hear from folks! 

Experiment Scorecard

Today, we’re introducing the ability to include an experiment hypothesis and primary/ secondary metrics at experiment creation, which will manifest in the form of an experiment “Scorecard” in your results tab.

While these fields are optional, the hope is that this feature makes it easier to standardize your experiment design process within the Statsig console, as well as improves the experience for non-experiment-creators reading experiments, enabling them to more fully understand key experiment context. 

CUPED (Controlled experiment Using Pre-Experimental Data)

As part of our bigger investment in a true Experiment “Scorecard”, we have implemented CUPED to automatically reduce variance and bias on all Scorecard metrics. CUPED is a statistical technique first popularized for online testing by Microsoft in 2013 that leverages pre-experimental data to reduce variance and pre-exposure bias in experiment results. Tactically, CUPED can significantly shrink confidence intervals and p-values, ultimately reducing the sample size and duration required to run an experiment. Which means you can run more experiments, faster!

CUPED will be applied by default to all Scorecard metrics (both Primary and Secondary), however you can toggle it on/ off directly above your Pulse results in the Scorecard. CUPED will not be applied to non-Scorecard metrics.

To read more about CUPED, check out our data scientist Craig’s awesome blog post here.

Metrics Tab Updates: Custom Metrics, Filtered Search, and more!

Today we are completing the rollout of a Metrics Tab refresh. As always, we love to get feedback & new feature requests from our community, so don't hesitate to reach out on this thread or 1:1! 

Metrics Tab Refresh

This refresh was aimed at streamlining the Metrics tab and increasing flexibility of how you view your Events and Metrics.

Key updates include-

1. Custom metrics now live within the Metrics Catalog - To create a custom metric, tap the “+Create” button. Once the custom metric is created, it will live in the Metrics Catalog and is searchable + taggable inline, alongside all your other metrics.

2. Filtered search - We’ve added filters to Metrics Catalog and Events to enable more easily drilling down to the set of metrics or events you care about most. Filter by Tag, Source (e.g. Statsig SDK vs. ingested metrics via integration), and Type (e.g. event_count, event_dau, funnel, etc.)

3. Different views - Toggle between a listview for easy scanning and chart view to better understand trends in both the Metrics Catalog and Events tabs. The view toggle is in the upper right-hand corner of each tab.

4. Lineage - Understand the “family tree” of any event or metric with the Lineage unit at the top of all Event Detail and Metric Detail pages.

5. Funnels - We’ve improved our funnel UX, moving funnel views onto the funnel metric detail view itself (all funnels will also exist in the “Charts” tab). The Lineage unit at the top of the funnel metric detail view indicates which events are included in the funnel.