Product Updates

We help you ship faster. And we walk the walk
4/28/2023
Permalink ›

Happy Friday, Statsig Community! To cap off a beautiful week here in Seattle ☀️, we have a number of exciting launch updates to share:

🕒 Fast(er) Pulse

Todate, when you launch a new feature roll-out or experiment, you have to wait 24 hours to start seeing your Pulse results. Today, we’re very excited to shorten that time significantly with the launch of more real-time Pulse. Now, you will see Pulse results start to flow through within 10-15 minutes of starting your roll-out or experiment.

faster pulse 2

A few things to consider-

  • For the first 24 hours, results do not include confidence intervals; early metric lifts are meant to help you ensure that things are looking roughly as expected and verify the configuration of your gate/ experiment, NOT make any launch decisions

  • The Pulse hovercard view will look a bit different; time-series and top-line impact estimates will not be available until the first 24-hour daily lift calculation

☁️ Environments in Overrides

At some companies, an user may have a different ID in different environments and hence want to specify the environment to override a given ID in. To enable this, we’ve added the ability to specify target environment for Overrides in Experiments. For Gates, you can achieve this via creating an environment-specific rule.

environments in overrides

⌛ Experiment Duration by # Target Exposures

(vs. Strictly Time Duration)

We’re introducing more flexibility into how you can measure & track experiment target duration. Now, you can choose between setting a target # of days or a target # of exposures an experiment needs to hit before a decision can be made.

experiment duration

To configure a target # of exposures, tap “Advanced Settings” in Experiment Setup tab, then under “Experiment Measured In” select “Exposures” (vs. “Days”). The progress tracker at the top of your experiment will now show progress against hitting target number of exposures.

experiment duration 2

See our docs for more details.

4/17/2023
Permalink ›

Manual assignment for Stratified Sampling

Layers in a Cake

Statsig manages randomization during experiment assignment. In some B2B (or low scale, high variance cases) the law of large numbers doesn’t work. Here it is helpful to manually assign users to test and control to ensure both groups are comparable. Statsig now lets you do this. Learn More

What is Stratified Sampling?

Stratified sampling is a sampling method that ensures specific groups of data (or users) are properly represented. You can think of this like slicing a birthday cake. If sliced recklessly, some people may get too much frosting and others will get too little. But when sliced carefully, each slice is a proper representation of the whole. In Data Science, we commonly trust random sampling. The Law of Large Numbers ensures that a sufficiently-sized sample will be representative of the entire population. However, in some cases, this may not be true, such as:

  • When the sample size is small

  • When the samples are heterogeneous

Warehouse ingestion tab makeover

We gave our Warehouse Ingestion tab a total makeover so that you can have better visibility into your import status! Some key improvements include:

  • simple visual display to track your import progress, with an extended date range

  • Verify your imported data with ease and confidence using our import volume chart and data samples

  • Take actions more easily and stay in control of your imports (use the “…” menu), whether you want to trigger a backfill or edit your daily ingestion schedule

Statsig data warehouse metrics
3/31/2023
Permalink ›

Explore metrics outside just an experiment

We’ve heard from some folks that they want to explore metrics even outside an experiment’s context. We’ve just started adding capabilities to do this. Now, when you’re looking at a metric in the Metrics Catalog you can:

  • compare values to a prior period to look for anomalies

  • apply smoothing to understand trends

  • look at other metrics at the same time to see correlation (or lack thereof) group by metric dimensions

  • save this exploration as a Dashboard to revisit/share with others

  • view current experiments and feature rollouts that impact this metric (also in Insights)

This starts rolling out March 31.

Old Metrics Exploration Page
The New Metrics Exploration Page
3/31/2023
Permalink ›

Incorporating more granular target population in Power Calculator calculations

Now, you can specify which audience you want to calculate experimental power for, by selecting any existing Feature Gate via the Power Calculator.

To do this, go to the Power Calculator (either under “Advanced Settings” in Experiment creation or via the “Tools & Resources” menu) and select “Population”.

This will kick off an async power calculation based on the selected targeting gate’s historical metric value(s), and you will be notified via email and Slack once your power analysis is complete.

power analysis calculator screenshot
3/25/2023
Permalink ›

Left navigation bar auto-collapse

This gives you more free real estate to do your work in console! This will now be the default setting, but you can switch this back to manual collapse by using the “…” menu on the nav bar.

Permanent and stale gates

This can be found under Types filter in your Gates catalog. While these gates indicate helpful information about your flags, they will not change anything about the functionality of the flags.

statsig premanent and stale gates
  • Permanent Gates (set by you) are gates that are expected to stay in your codebase for a long time (e.g. user permissions, killswitches). Statsig won’t nudge you to clean up these gates.

    • You can set gates to be Permanent in the creation flow or by using the “…” menu within each gate page.

  • Stale Gates (set by Statsig) are good candidates to be cleaned up (and will be used to list out gates for email/slack nudges)

    • On Monday morning, you’ll receive your first monthly nudge (email + slack) to take action on stale gates.

    • At a high level, these gates are defined as 0%/100% rolled out or have had 0 checks in the last 30 days (but exclude newly created or Permanent gates).

statsig

Please see the permanent and stale gates documentation for more information.

🚨 Metric Alerts

Today we are introducing the ability to configure thresholds for your metrics that will automatically trigger an alert if breached in the context of any feature rollout or experiment.

This is especially useful for your company’s suite of core Guardrail Metrics, as you can configure thresholds once and rest assured that you’ll be notified whenever a new feature gate or experiment breaches the pre-set threshold. (As a reminder you can also hook up Statsig to Datadog monitors, read more here!)

metric alerts setup

To configure a Metric Alert-

  1. Go to the Metric Detail View page of the metric in question

  2. Tap into the “Alerts” tab and tap “+ Create Alert”

  3. Configure the threshold value and minimum # of participating units required to trigger the alert (even though this second field is optional, we highly recommend configuring it to minimize the noisiness of your alert)

When your metric alert fires, you will be notified via email (and Slack if you’ve configured the Statsig Slack bot) and directed to the “Diagnostics” tab of the offending gate or experiment.

metric alerts

Please note that these alerts are for metric values in the context of a specific gate or experiment and NOT on the top-line value of a metric. See our docs for more details on Metric Alerts, and don't hesitate to reach out if you have questions or feedback!

Composite Sums, Pass Rate Filter, Permanent Gates, and More

New custom metric type -

➕ Composite Sums: You can now create an aggregation (sum) metric using other metrics from your catalog, whereas previously, you were only able to sum up events. You can now do cool things such as: adding up revenue across different categories or user counts across different regions.

Iteration on the previously launched feature gate lifecycle/cleanup toolsets -

☑️ Pass Rate filter: We heard your feedback and have added a Pass Rate filter in your Gates Catalog, in addition to the existing Roll Out Rate filter, to make your launch/disable/cleanup decisions off of!

What’s the difference? Roll Out Rate is strictly based on the gate rules you’ve set, whereas Pass Rate shows what’s happening in practice at the time of evaluation. For example, if you have a set of rules that effectively pass everyone besides a small holdout group, Roll Out Rate would be < 100% but Pass Rate could still be 100% if the holdout is small enough.

♾️ Permanent Gates: Permanent feature gates are expected to live in your codebase for an extended period of time, beyond a feature release, usually for operations or infrastructure control (examples: user permissions, circuit breakers/kill switches).

permanent gates statsig screenshot

You can now mark your gates Permanent on Statsig, telling your team (and Statsig) that they should proceed with more caution if attempting to clean up these gates. This will not change anything functionally about the gate itself, but will allow us to surface and label them differently in the Statsig console for your convenience.

2/27/2023
Permalink ›

Flexible Environment Configuration and Metrics Directionality

Hi everyone! Here are a few launch announcements to kick off our week. 

🌎 Flexible Environment Configuration

Today, we’re introducing a number of updates to Environments on Statsig-

  1. Adding & Customizing Environments

    Whereas previously Statsig only supported two, pre-defined environments, we are now opening up the ability to create as many environments as you want and customize the name and relative hierarchy of these environments. To customize the environments available in your Statsig Project, go to the “Environments & Keys” tab within Project Settings.

  2. API Keys per Environment

    We are also adding support for environment-specific API keys, enabling greater launch security and privacy through more API key granularity. To configure environment-specific API keys, go to the “Environments & Keys” tab within Project Settings.

  3. Improved Environments UX

    You’ll also notice that environments look and feel a bit different in the Console. We now expose which environments a given rule is targeted at on every rule header, with the ability to toggle rule-sets by target environment via the environment filters at the top of the gate.

Statsig Specify Environments
Statsig Manage Environments

Read more about how environments work via our Docs Page. 

↗️ Metrics Directionality

Today, Pulse makes a relatively naïve assumption that the world is simple and a metric increase = good = green, and a metric decrease = bad = red. However for many metrics, it’s actually a good thing when the metric decreases and similarly bad to see an increase (think page load times or many performance metrics!)

With metric directionality, you can set the desired direction you want to see a metric move and Pulse will update the color-coding in metric lifts accordingly. To access this feature, go to any metric detail view page, and tap the “…” menu to see the “Set Metric Directionality” option.

Statsig Set Metric Directionality
Statsig Increase or Decrease Metric Directionality

We hope you enjoy this new functionality! Please get in touch with us with any feedback, and join the Statsig Slack community to receive these updates first!

Datadog Trigger Integration

In the past, we integrated Statsig with Datadog so that you could send events from Statsig to Datadog and use the whole suite of services on Datadog to monitor these events. However, this was only one-directional and any real-time observations you make from Datadog would require manual oversight to take action on Statsig.

🎉 Intoducing the Statsig and Datadog Trigger Integration

Now we’ve made it possible to leverage Datadog’s real-time monitoring to automatically toggle a feature gate on or off in Statsig.

Configuring Datadog and Statsig to monitor events and toggle feature gates is simple:

  1. Simply create a “trigger” on Statsig

    create a datadog trigger in statsig
  2. Create a Datadog webhook using the trigger URL

    create a datadog webhook
  3. Configure your monitor to notify that webhook.

    configure monitor datadog

Example

Imagine you are rolling out a new feature behind a Statsig feature gate. You can set up a Datadog monitor to detect anomalies in your operational metrics in correlation to changes to this gate.

Instead of having it send an alert to you or your team, you can create a trigger to disable this gate automatically. Now if the monitor fires an alert, by the time you are notified, you can rest assured the gate has already been turned off.

a gate being monitored in datadog

With this new integration, Statsig and Datadog customers can take full advantage of the features of both services. With added benefits of:

  • 🔬 Better monitoring of feature rollouts

  • 🚨 Faster response time to metric regression

  • ☮️ Overall, more reliability of their service and peace of mind when launching new features.

We hope you enjoy this new functionality! Please get in touch with us with any feedback, and don’t hesitate to join the Statsig Slack community!

Loved by customers at every stage of growth

See what our users have to say about building with Statsig
OpenAI
"At OpenAI, we want to iterate as fast as possible. Statsig enables us to grow, scale, and learn efficiently. Integrating experimentation with product analytics and feature flagging has been crucial for quickly understanding and addressing our users' top priorities."
Dave Cummings
Engineering Manager, ChatGPT
SoundCloud
"We evaluated Optimizely, LaunchDarkly, Split, and Eppo, but ultimately selected Statsig due to its comprehensive end-to-end integration. We wanted a complete solution rather than a partial one, including everything from the stats engine to data ingestion."
Don Browning
SVP, Data & Platform Engineering
Recroom
"Statsig has been a game changer for how we combine product development and A/B testing. It's made it a breeze to implement experiments with complex targeting logic and feel confident that we're getting back trusted results. It's the first commercially available A/B testing tool that feels like it was built by people who really get product experimentation."
Joel Witten
Head of Data
"We knew upon seeing Statsig's user interface that it was something a lot of teams could use."
Laura Spencer
Chief of Staff
"The beauty is that Statsig allows us to both run experiments, but also track the impact of feature releases."
Evelina Achilli
Product Growth Manager
"Statsig is my most recommended product for PMs."
Erez Naveh
VP of Product
"Statsig helps us identify where we can have the most impact and quickly iterate on those areas."
John Lahr
Growth Product Manager
"The ability to easily slice test results by different dimensions has enabled Product Managers to self-serve and uncover valuable insights."
Preethi Ramani
Chief Product Officer
"We decreased our average time to decision made for A/B tests by 7 days compared to our in-house platform."
Berengere Pohr
Team Lead - Experimentation
"Statsig is a powerful tool for experimentation that helped us go from 0 to 1."
Brooks Taylor
Data Science Lead
"We've processed over a billion events in the past year and gained amazing insights about our users using Statsig's analytics."
Ahmed Muneeb
Co-founder & CTO
SoundCloud
"Leveraging experimentation with Statsig helped us reach profitability for the first time in our 16-year history."
Zachary Zaranka
Director of Product
"Statsig enabled us to test our ideas rather than rely on guesswork. This unlocked new learnings and wins for the team."
David Sepulveda
Head of Data
Brex
"Brex's mission is to help businesses move fast. Statsig is now helping our engineers move fast. It has been a game changer to automate the manual lift typical to running experiments and has helped product teams ship the right features to their users quickly."
Karandeep Anand
President
Ancestry
"We only had so many analysts. Statsig provided the necessary tools to remove the bottleneck. I know that we are able to impact our key business metrics in a positive way with Statsig. We are definitely heading in the right direction with Statsig."
Partha Sarathi
Director of Engineering
"Statsig has enabled us to quickly understand the impact of the features we ship."
Shannon Priem
Lead PM
Ancestry
"I know that we are able to impact our key business metrics in a positive way with Statsig. We are definitely heading in the right direction with Statsig."
Partha Sarathi
Director of Engineering
"Working with the Statsig team feels like we're working with a team within our own company."
Jeff To
Engineering Manager
"[Statsig] enables shipping software 10x faster, each feature can be in production from day 0 and no big bang releases are needed."
Matteo Hertel
Founder
"We use Statsig's analytics to bring rigor to the decision-making process across every team at Wizehire."
Nick Carneiro
CTO
Notion
"We've successfully launched over 600 features behind Statsig feature flags, enabling us to ship at an impressive pace with confidence."
Wendy Jiao
Staff Software Engineer
"We chose Statsig because it offers a complete solution, from basic gradual rollouts to advanced experimentation techniques."
Carlos Augusto Zorrilla
Product Analytics Lead
"We have around 25 dashboards that have been built in Statsig, with about a third being built by non-technical stakeholders."
Alessio Maffeis
Engineering Manager
"Statsig beats any other tool in the market. Experimentation serves as the gateway to gaining a deeper understanding of our customers."
Toney Wen
Co-founder & CTO
"We finally had a tool we could rely on, and which enabled us to gather data intelligently."
Michael Koch
Engineering Manager
Notion
"At Notion, we're continuously learning what our users value and want every team to run experiments to learn more. It's also critical to maintain speed as a habit. Statsig's experimentation platform enables both this speed and learning for us."
Mengying Li
Data Science Manager
Whatnot
"Excited to bring Statsig to Whatnot! We finally found a product that moves just as fast as we do and have been super impressed with how closely our teams collaborate."
Rami Khalaf
Product Engineering Manager
"We realized that Statsig was investing in the right areas that will benefit us in the long-term."
Omar Guenena
Engineering Manager
"Having a dedicated Slack channel and support was really helpful for ramping up quickly."
Michael Sheldon
Head of Data
"Statsig takes away all the pre-work of doing experiments. It's really easy to setup, also it does all the analysis."
Elaine Tiburske
Data Scientist
"We thought we didn't have the resources for an A/B testing framework, but Statsig made it achievable for a small team."
Paul Frazee
CTO
Whatnot
"With Warehouse Native, we add things on the fly, so if you mess up something during set up, there aren't any consequences."
Jared Bauman
Engineering Manager - Core ML
"In my decades of experience working with vendors, Statsig is one of the best."
Laura Spencer
Technical Program Manager
"Statsig is a one-stop shop for product, engineering, and data teams to come together."
Duncan Wang
Manager - Data Analytics & Experimentation
Whatnot
"Engineers started to realize: I can measure the magnitude of change in user behavior that happened because of something I did!"
Todd Rudak
Director, Data Science & Product Analytics
"For every feature we launch, Statsig saves us about 3-5 days of extra work."
Rafael Blay
Data Scientist
"I appreciate how easy it is to set up experiments and have all our business metrics in one place."
Paulo Mann
Senior Product Manager
We use cookies to ensure you get the best experience on our website.
Privacy Policy