Introducing Pulse

Stephen Royal
Thu Apr 29 2021
FACEBOOK PULSE EXPERIMENTATION ANNOUNCEMENT

Deltoid, Facebook’s internal AB testing and experimentation tool is arguably its most important tool allowing teams to build, test and ship products at breakneck speed. Its value proposition is simple: know how your features affect your core metrics BEFORE launching. 100s of teams are constantly building 1000s of features at any given time — and even at that scale, Deltoid would diligently track statistical movements between test and control groups, against the company’s core metrics (like DAU, MAU, user retention, engagement, time-spent, transactions, revenue), a team’s most relevant metrics (page views, shares, interactions, time-spent), and key hypothesis and guardrail metrics. Every engineer, data scientist, product manager, designer, researcher, and even business folks have at some point stared at a Deltoid chart and made important product decisions.

Today with Pulse, we’re bringing the same power of data-driven decision-making to everyone.

Many large companies have also built their in-house experimentation platforms like this one from Spotify and this one from Uber.

Today with Pulse, we’re bringing the same power of data-driven decision-making to everyone. For every feature you’re building behind a Feature Gate, you can now see how it is performing along with how it is affecting your company’s critical metrics.

For instance, in the picture above, a new feature was opened to 10% of user traffic. The horizontal bars in the graph indicate confidence intervals. They start out gray, but once a particular metric gathers statistical significance, it turns either red or green depending on if it’s on the negative or positive side of the axis.

In a single glance you can tell whether or not a feature is ship-worthy. In this example, the new feature is negatively affecting the product_view and product_details metrics while no other metrics show a statistically clear lift; it isn’t wise to ship this as it is. As you improve the feature, your experimental metrics will begin moving to the right, and as you increase exposure, the confidence intervals will narrow down. When the metrics show a lift with tight enough confidence intervals, you’ll achieve statistical significance providing assurance that this feature is ship-worthy.

If you want to give this a spin, head on over to https://www.statsig.com to get started!


Try Statsig Today

Explore Statsig’s smart feature gates with built-in A/B tests, or create an account instantly and start optimizing your web and mobile applications. You can also schedule a live demo or chat with us to design a custom package for your business.

MORE POSTS

Recently published

My Summer as a Statsig Intern

RIA RAJAN

This summer I had the pleasure of joining Statsig as their first ever product design intern. This was my first college internship, and I was so excited to get some design experience. I had just finished my freshman year in college and was still working on...

Read more

Long-live the 95% Confidence Interval

TIMOTHY CHAN

The 95% confidence interval currently dominates online and scientific experimentation; it always has. Yet it’s validity and usefulness is often questioned. It’s called too conservative by some [1], and too permissive by others. It’s deemed arbitrary...

Read more

Realtime Product Observability with Apache Druid

JASON WANG

Statsig’s Journey with Druid This is the text version of the story that we shared at Druid Summit Seattle 2022. Every feature we build at Statsig serves a common goal — to help you better know about your product, and empower you to make good decisions for...

Read more

Quant vs. Qual

MARGARET-ANN SEGER

💡 How to decide between leaning on data vs. research when diagnosing and solving product problems Four heuristics I’ve found helpful when deciding between data vs. research to diagnose + solve a problem. Earth image credit of Moncast Drawing. As a PM, data...

Read more

The Importance of Default Values

TORE

Have you ever sent an email to the wrong person? Well I have. At work. From a generic support email address. To a group of our top customers. Facepalm. In March of 2018, I was working on the games team at Facebook. You may remember that month as a tumultuous...

Read more
ANNOUNCEMENT

CUPED on Statsig

CRAIG

Run experiments with more speed and accuracy We’re pleased to announce the rollout of CUPED for all our customers. Statsig will now automatically use CUPED to reduce variance and bias on experiments’ key metrics. This gives you access to a powerful experiment...

Read more

We use cookies to ensure you get the best experience on our website.

Privacy Policy