Environments on Statsig

Vineeth Madhusudanan
Fri Jan 07 2022
EXPERIMENTATION FEATURE-FLAGS

In the summer of 2021, HBO accidentally sent an email to millions of past and current subscribers with the body “This message is used by integration tests only”. The internet was gracious about the mistake an intern made (context), but it was an interesting reminder of the challenges of managing environments.

For continuous deployment, teams often create environments for development, staging and production. The development environment optimizes for rapid iteration. Staging is where you discover the known-unknowns. Production optimizes for reliability and security.

Two philosophies : Per Environment Config vs Global Config

There are different philosophies for how to manage multiple environments. Some teams attempt to create hard boundaries between these environments. They create separate, distinct config for each environment. We’ll call this “Per Environment Config”.

Per Environment Config for a Feature Gate: navigate to different instance of each Feature Gate to see state in that environment

Others want all environments to be similar to Production. They only allow differences essential to the environment’s distinct purpose. We’ll call this “Global Config”.

While Statsig can support both approaches, it is optimized for the latter. We believe this is a better way to manage feature gates and experiments across environments. It optimizes for engineering efficiency : reducing errors and making troubleshooting faster.

Global Config for a Feature Gate: 1% rollout everywhere (including staging/production); with a 100% rollout in the Dev Tier.

Advantages

Some advantages of the Global Config approach (with environment specific exceptions) include…

  1. Can look at a feature gate in one place and see all exceptions by environment, without having to navigate across different screens for different environments and diff them in your head.
  2. Feature Gates and Experiments are typically configured with default/fallback values. The absence of configuration for an environment can easily be missed because the app seems to work with the defaults.
  3. An extra bonus is the ability to test feature gates with real data when you’re troubleshooting why a user passed/or failed a gate. Pass in the JSON with the user attributes your app is sending, and we’ll look across all environments and pinpoint the specific rule that is applying to the user and causing the pass/fail result!
Statsig console lets you test feature gates and isolate the specific rule responsible for the result you see

Wrinkles (and mitigation)

A potential issue with the global config approach is when change management approvals are required. Statsig allows teams to require review for changes before they’re committed.

Change Management : When enabled, reviewers can approve changes before they’re committed

This is useful in production where stability is paramount, but can slow down iteration in development and staging environments.

The solution for this is to create rules explicitly for the Dev and Staging environments (like in the global config example above). Dev and Staging environment rules do not require approval to change— allowing engineers to iterate rapidly. Changes to the “All Rules” section can impact Production and will continue to need change approval.

Note that you aren’t required to create an environment specific section. You can choose to use Environment Tier as a condition in regular rules. Adding explicit sections for Dev and Staging rules makes the distinction easier to see and benefits from removing the need for approvals (if enabled).

You can use Environment Tier when configuring rules too

Reach out!

See an issue not addressed by the approach we’re suggesting? Reach out — I’d love to hear from you and learn more!


Try Statsig Today

Explore Statsig’s smart feature gates with built-in A/B tests, or create an account instantly and start optimizing your web and mobile applications. You can also schedule a live demo or chat with us to design a custom package for your business.

MORE POSTS

Recently published

My Summer as a Statsig Intern

RIA RAJAN

This summer I had the pleasure of joining Statsig as their first ever product design intern. This was my first college internship, and I was so excited to get some design experience. I had just finished my freshman year in college and was still working on...

Read more

Long-live the 95% Confidence Interval

TIMOTHY CHAN

The 95% confidence interval currently dominates online and scientific experimentation; it always has. Yet it’s validity and usefulness is often questioned. It’s called too conservative by some [1], and too permissive by others. It’s deemed arbitrary...

Read more

Realtime Product Observability with Apache Druid

JASON WANG

Statsig’s Journey with Druid This is the text version of the story that we shared at Druid Summit Seattle 2022. Every feature we build at Statsig serves a common goal — to help you better know about your product, and empower you to make good decisions for...

Read more

Quant vs. Qual

MARGARET-ANN SEGER

💡 How to decide between leaning on data vs. research when diagnosing and solving product problems Four heuristics I’ve found helpful when deciding between data vs. research to diagnose + solve a problem. Earth image credit of Moncast Drawing. As a PM, data...

Read more

The Importance of Default Values

TORE

Have you ever sent an email to the wrong person? Well I have. At work. From a generic support email address. To a group of our top customers. Facepalm. In March of 2018, I was working on the games team at Facebook. You may remember that month as a tumultuous...

Read more
ANNOUNCEMENT

CUPED on Statsig

CRAIG

Run experiments with more speed and accuracy We’re pleased to announce the rollout of CUPED for all our customers. Statsig will now automatically use CUPED to reduce variance and bias on experiments’ key metrics. This gives you access to a powerful experiment...

Read more

We use cookies to ensure you get the best experience on our website.

Privacy Policy