I had the pleasure of chatting with our exceptional Software Engineers, Pierre Estephan and Alex Coleman. Our discussion revolved around metrics and analytics within the context of experimentation.
We delved into real-world examples showcasing how companies like Facebook, and some of our valued Statsig customers harnessed user insights to propel product-led growth. In light of our recent launch of Metrics Explorer, we wanted to discuss how having an integrated platform for experiments and analytics can empower organizations to unlock insights that drive growth. You can watch the recording of the event here:
It's challenging to directly impact high-level KPIs like retention or engagement. Therefore, it's crucial to understand the key drivers that lead to these goals. In all the examples we covered, including the famous Slack example where "messages sent" was the key driver of retention, companies identified what mattered most to their users and worked toward that, ultimately driving growth.
We discussed the concept of novelty effects, using notification fatigue as an example to emphasize the importance of going beyond instant gratification. It's not just about improving DAU in the short-term; it's about observing the long-term impact with metrics like 28-day retention.
Correlation doesn't imply causation, so it's not as simple as blindly pursuing a metric that appears to be correlated with retention. Instead, it's about understanding how users realize value. We emphasized the importance of understanding the qualitative aspects behind a metric, like how Facebook's famous "7 friends in 10 days" wouldn't have worked with automatically added friends for new users; it required relevant suggestions, for example.
Leading PLG companies avoid the biggest operational challenges of running experiments at scale (such as having too many tools in their stack, siloed data, and disagreements between teams on feature results) by maintaining a single platform for consistency. Everyone across the company, including non-technical users, has visibility into metrics. This approach keeps engineering and product teams agile, free from cumbersome processes, and able to build, measure, and learn rapidly.
We discussed examples where companies were able to achieve double-digit growth in core metrics by identifying drop-off points in the customer journey, such as the login or checkout funnel. They used these insights to capture low-hanging fruit by testing features to close these gaps. For more stories about our own customers who achieved remarkable growth, you can visit our customer stories page.
Experimenting with query-level optimizations at Statsig: How we reduced latency by testing temp tables vs. CTEs in Metrics Explorer. Read More ⇾
Find out how we scaled our data platform to handle hundreds of petabytes of data per day, and our specific solutions to the obstacles we've faced while scaling. Read More ⇾
The debate between Bayesian and frequentist statistics sounds like a fundamental clash, but it's more about how we talk about uncertainty than the actual decisions we make. Read More ⇾
Building a scalable experimentation platform means balancing cost, performance, and flexibility. Here’s how we designed an elastic, efficient, and powerful system. Read More ⇾
Here's how we optimized store cloning, cut processing time from 500ms to 2ms, and engineered FastCloneMap for blazing-fast entity updates. Read More ⇾
It's one thing to have a really great and functional product. It's another thing to have a product that feels good to use. Read More ⇾