Statsig 101

Tore
Fri Jul 16 2021
FEATURE-FLAGS A-B-TESTING ANALYTICS SOFTWARE-DEVELOPMENT

Use Statsig just like the people who built it

We’ve been using Statsig for the past four months across different surfaces: www.statsig.com, our main landing page, console.statsig.com, our main web interface for using Statsig as a tool, and in some internal tools we’ve built for ourselves. Coming from Facebook, it was natural for us to use Feature Gates to “gate off” features under development, Dynamic Configs to decouple configurations from front-end code, and Experiments+ to start optimizing our conversion funnel. We use all of these and more every day. But where did we get started? How can you dip your toes in without knowing what you might use Statsig for?

Step 0. Create a Feature Gate

Whenever you are building a new feature, the first step should be to create a Feature Gate. This will enable you to put a conditional check in your code to check if you should show the new feature.

Gates are always “off” (return false) by default. So you can create a Feature Gate without adding any conditions to it, and it will always return false. Rain or shine, no matter which SDK or what network conditions, that Feature Gate will return false.

Here, I’ve created a gate for the new search ranking model I am working on.

Step 1. Add a Conditional Branch

With the default value = false in mind, add a conditional branch to your code checking the gate. This gate is still going to always return false, so you can put whatever you want inside the “true” block — you could start with a simple print statement if you haven’t built anything new yet.

For my new search ranking feature, it looks like this:

https://medium.com/media/e436e712625ba60afcf187d3051f5db8/href

At this point you can even check in the code, and control the conditional branch just by updating the conditions and rules on the gate in the Statsig console.

Step 2. Open the Feature Gate to “only me”

Now that you are confident the new code is always gated off, how can you test the experience with it turned on? Update your Feature Gate to only pass with a specific ID or custom field, which you set in your StatsigUser object to test. As you integrate with any of our SDKs, you can pass this field the same way.

Once you have created an “only me” rule, save the changes and you can quickly test via the inline “Test Gate” console, or using the conditional you already created.

userID ‘tore’ passes, while anything else (or not providing an ID) will fail

Now you can start to build, integrate an SDK, and merge your code with the confidence that your in-progress features will only be visible to you.

Step 3. Expand access to your teammates or entire company

Once your feature is ready for more eyes, update the Feature Gate’s targeting conditions to turn it into a “dogfooding” gate. Rather than just opening the feature you are working on to yourself, you can open it up to your whole team, organization, or company. For example, you can use a user’s authenticated email domain to check if they have access (we do this for internal Statsig features all the time!):

You could use a list of user IDs instead of emails, a custom field, or any other combination in order to get more visibility. The key difference being it is not just you seeing the feature any more!

Step 4. Open your gate to the everyone

You can roll out your feature to anybody using an “everyone” condition, or any set of custom checks you can think of (based on app version, country, browser, device, etc). Partial rollouts will track metric variations in pulse — so rather than opening a gate to “everyone — 100%”, we suggest you open to 5%, 10%, 50%, or something in between.

Here’s an example with two different partial rollouts — a 10% rollout to the US and Canada, and a 10% rollout for everyone else.

Step 5. Check Pulse

Pulse calculates the statistical significance of changes in metric values between test and control groups, and is updated on a daily basis. Now that you have a test running, you can measure the impact of your new feature on critical metrics like app crashes, DAU, revenue, time spent, etc. You can see an example of what this looks like in our demo company. In this case, we are analyzing only the difference between test and control for users in the US and Canada.

Now that’s a good looking search ranking change!

Step 6. Repeat

Continue to follow that same playbook:

  1. Create a new Feature Gate for every feature you start building.
  2. Gradually update the rules and conditions governing access to expand the audience from yourself to your team and eventually to everyone.
  3. Use partial rollouts to get a Pulse view of the difference on your key metrics between test and control.

Tip #1: Invite others to your Statsig Project

It’s also worth inviting other Engineers, Designers, PMs, Managers, Directors, or anyone else in your team/org/company to follow along by joining the project on Statsig. There are no per-seat fees, so invite as many people to join this process as you want!

Tip #2: Use good names and descriptions!

As you continue to use this development process more and more, you will find it very useful to name your Feature Gates and give them good descriptions. Using a descriptive gate name will help your code read better and make it more clear in the Statsig console what a particular gate controls. If access to each feature is controlled by its own Feature Gate, you can decide independently which gates to open only to yourself or your team, and ultimately the public. Here is an example from our Feature Gate list:

Oh yeah, we are experimenting with dark mode for the Statsig console!

If you want a hand walking through these steps, or have questions about advanced targeting, setting up custom metrics in pulse, etc. feel free to join our slack community to discuss with us and other Statsig users.

Join Statsig Community on Slack


Try Statsig Today

Explore Statsig’s smart feature gates with built-in A/B tests, or create an account instantly and start optimizing your web and mobile applications. You can also schedule a live demo or chat with us to design a custom package for your business.

MORE POSTS

Recently published

My Summer as a Statsig Intern

RIA RAJAN

This summer I had the pleasure of joining Statsig as their first ever product design intern. This was my first college internship, and I was so excited to get some design experience. I had just finished my freshman year in college and was still working on...

Read more

Long-live the 95% Confidence Interval

TIMOTHY CHAN

The 95% confidence interval currently dominates online and scientific experimentation; it always has. Yet it’s validity and usefulness is often questioned. It’s called too conservative by some [1], and too permissive by others. It’s deemed arbitrary...

Read more

Realtime Product Observability with Apache Druid

JASON WANG

Statsig’s Journey with Druid This is the text version of the story that we shared at Druid Summit Seattle 2022. Every feature we build at Statsig serves a common goal — to help you better know about your product, and empower you to make good decisions for...

Read more

Quant vs. Qual

MARGARET-ANN SEGER

💡 How to decide between leaning on data vs. research when diagnosing and solving product problems Four heuristics I’ve found helpful when deciding between data vs. research to diagnose + solve a problem. Earth image credit of Moncast Drawing. As a PM, data...

Read more

The Importance of Default Values

TORE

Have you ever sent an email to the wrong person? Well I have. At work. From a generic support email address. To a group of our top customers. Facepalm. In March of 2018, I was working on the games team at Facebook. You may remember that month as a tumultuous...

Read more
ANNOUNCEMENT

CUPED on Statsig

CRAIG

Run experiments with more speed and accuracy We’re pleased to announce the rollout of CUPED for all our customers. Statsig will now automatically use CUPED to reduce variance and bias on experiments’ key metrics. This gives you access to a powerful experiment...

Read more

We use cookies to ensure you get the best experience on our website.

Privacy Policy