The Causal Roundup is a biweekly review of the best articles on causality. Covering topics from experimentation to causal inference, the Statsig team brings to you work from leaders who are building the future of product decision-making.
This month Netflix started a blog series about how they make decisions using A/B tests. Instead of restricting decision-making to executives and experts, experimentation gives all their employees the opportunity to vote with their actions.
Now, you might already have a dozen analytics tools to track scores of metrics but without a causal chain, you’re still broadly shooting in the dark. Netflix shows this beautifully in a follow-up post on A/B testing using a hypothetical product launch with upside-down cover art! This blurb made me weak in the knees…
Articulating the causal chain between the product change and changes in the primary decision metric, and monitoring secondary metrics along this chain, helps us build confidence that any movement in our primary metric is the result of the causal chain we are hypothesizing, and not the result of some unintended consequence of the new feature (or a false positive).
If in pursuit of your destination, you plunge ahead, heedless of obstacles, and achieve nothing more than to sink in a swamp…What’s the use of knowing True North? — Lincoln
One of the challenges with improving long-term metrics such as engagement is that these metrics are hard to move and often require long drawn-out experiments. This paper from LinkedIn in 2019 describes how they overcome this challenge.
LinkedIn proposes using a surrogate metric that predicts the long-term (north star) metric. As surrogate metrics rarely predict the north star metric perfectly, the paper discusses how to adjust A/B testing to ensure experiment results are trustworthy. For example, LinkedIn aims to improve its hiring products with a true north metric called confirmed hires (CH), which measures members who found jobs using LinkedIn products.
However, the CH metric suffers from long lag times. To address this, the team introduces a surrogate metric called predicted confirmed hires (PCH), which leverages several signals including job segments, time of application, quality of application, and so on. The paper also neatly provides practical guidelines for choosing good surrogate metrics such as sensitivity to a wide range of input variables that are worth experimenting.
This past summer, Ujwal Kharel described a great example of how Roblox measured the impact of Avatar Shop on community engagement. They couldn’t run an experiment as it isn’t possible to just turn off the Avatar Shop for some users.
Causal inference was the way to go. They still needed an instrumental variable that’s (i) strongly associated with the treatment variable (Avatar Shop engagement) and (ii) associated with the outcome (community engagement) only via the treatment variable.
The fun part is that they found the instrumental variable from an experiment they ran a year ago. Ujwal’s view is that teams throw away experiment results too easily and begs to dig deeper to find evidence that’s interesting.
I can’t wait to read more from the Roblox Tech Blog for more 🥰
Watch this space for more updates, stories, and practical tips on finding causality in user behavior and growing product adoption. Follow the Statsig blog to get the biweekly update!
Thanks to our support team, our customers can feel like Statsig is a part of their org and not just a software vendor. We want our customers to know that we're here for them.
Migrating experimentation platforms is a chance to cleanse tech debt, streamline workflows, define ownership, promote democratization of testing, educate teams, and more.
Calculating the right sample size means balancing the level of precision desired, the anticipated effect size, the statistical power of the experiment, and more.
The term 'recency bias' has been all over the statistics and data analysis world, stealthily skewing our interpretation of patterns and trends.
A lot has changed in the past year. New hires, new products, and a new office (or two!) GB Lee tells the tale alongside pictures and illustrations:
A deep dive into CUPED: Why it was invented, how it works, and how to use CUPED to run experiments faster and with less bias.