This is a summary of the Medium article authored by Chander Ramesh on 11/6/22.
When Chander joined Motion in late February 2022, there were three critical problems facing the engineering team:
Incredibly long and unstable release processes
No ability to roll back releases
Chander recognized all three issues were stemming from a lack of feature flagging infrastructure.
Like any new Head of Engineering whose team was drowning, Chander decided to buy (vs. build)—especially since feature flags were not their core product. And while he was correct in his diagnosis, his takeaway from the whole experience was "I was horribly incorrect in choosing LaunchDarkly."
LaunchDarkly had been recommended by several peers Chander had worked with at previous startups, and he admittedly didn’t put much thought into platform evaluation. "Everything was horribly broken, and we needed to move fast, so (I thought to myself) let’s just do it."
By April 1st LaunchDarkly was mandatory for every new feature in Motion’s backend, web app, and Chrome extension. "By July 1st it was obvious this was a huge mistake" Chander states. The team began evaluating alternatives and began integrating Statsig into their codebase in August.
By November 1st LaunchDarkly was completely phased out.
Statsig’s pricing model is usage-based, whereas "LaunchDarkly stubbornly insists on seat-based pricing, so every engineer we added had an additional cost"—as Chander points out.
Motion tried to restrict access to a select few, but it added extra burden on those few engineers to keep track of everyone’s launches and feature flags. This included things like reminding them to turn flags on after a release, cleaning them up after a launch, and adding new ones to the test environment.
Ultimately it became too laborious, and the team decided the developer productivity cost wasn’t worth it. They also ran into some issues with how LaunchDarkly was estimating MAUs—something that raised questions and could have caused substantial overcharges.
Motion ran into performance and memory issues using integrating LaunchDarkly in a Chrome extension using the documentation provided. Chander ultimately found a workaround by using the LaunchDarkly JS Client SDK on Github and adapting it for the team’s needs. He states "while it mostly works, I’m 100% sure there are some underlying bugs. I’m using code in a way it wasn’t meant to be used, so please don’t blindly copy paste this code wherever."
We share Chander’s belief that a good feature flagging system should make it as easy to remove a feature flag as it does to add a new feature flag. According to him "LaunchDarkly fails spectacularly yet again in this regard."
The team at Motion found it impossible to tell if a feature flag was actually being used or not. Additionally, they found the insights graph inside LaunchDarkly was also not accurately representing flag evaluation volumes after several updates were made.
Motion ran into many more challenges with LaunchDarkly, including:
No support for Expo in React Native, which meant their mobile apps couldn’t use feature flags. (Note: Statsig supports Expo.)
Inconsistent updates that impacted some users and resulted in getting the wrong values (this all went away after switching to Statsig.)
Issues with mistakenly archiving flags between test and prod environments due to UI/UX design.
Running into an upcharge fee in order to mandate approvals on feature flags (Statsig provides this for free).
Virtual hugs to you Chander. This whole experience sounds frustrating and we're sorry you had to go through it.
Statsig's experts are on standby to answer any questions about experimentation at your organization.
It’s critically important to evaluate vendors and digital infrastructure providers before spending hours of valuable engineering talent implementing (and potentially unwinding/replacing) a solution.
Statsig loves learning from customer experiences. Often, we’re in a position to help fix and clean up what was obviously a poor fit: It's why we have an engineering team dedicated to customer success, and why our product roadmap is prioritized to solve customer needs (we ship new features every two weeks!).
Plus, 75% of Statsig customers use us for feature flagging in addition to our experimentation and analytics capabilities.
Get started by creating a free account or requesting a demo today!
Original post: Feature-flagging via LaunchDarkly — and why we moved to Statsig | by Chander Ramesh | Nov, 2022 | Medium
Thanks to our support team, our customers can feel like Statsig is a part of their org and not just a software vendor. We want our customers to know that we're here for them.
Migrating experimentation platforms is a chance to cleanse tech debt, streamline workflows, define ownership, promote democratization of testing, educate teams, and more.
Calculating the right sample size means balancing the level of precision desired, the anticipated effect size, the statistical power of the experiment, and more.
The term 'recency bias' has been all over the statistics and data analysis world, stealthily skewing our interpretation of patterns and trends.
A lot has changed in the past year. New hires, new products, and a new office (or two!) GB Lee tells the tale alongside pictures and illustrations:
A deep dive into CUPED: Why it was invented, how it works, and how to use CUPED to run experiments faster and with less bias.