In the innovative landscape of software development resides a diverse mix of product management, feature flagging, and experimentation tools.
At face value, the landscape is roughly homogenous: Some newer companies, some older companies, all doing approximately the same thing. Once you enter the buyer’s journey, however, a different picture begins to form:
Each company has its own unique strengths and weaknesses ranging from actual features, ease of setup, billing model, core functions, and so on. With experimentation becoming the current ‘meta,’ all companies from web optimization platforms to feature flagging services are trying to claim it as their own.
But some companies merely adopted experimentation, others were born in it, molded by it…so to speak. So let’s zoom out and analyze which companies are all-in-one platforms, which are good feature flagging services, and so on. Launchdarkly certainly doesn’t do it all…right?
Launchdarkly is a SaaS platform that manages feature flags, feature rollouts, A/B tests, and the like. Founded in 2014, Launchdarkly was one of the first third-party feature flagging tools to gain widespread adoption by tech companies.
On its “about” page, Launchdarkly states that its mission is to create a world wherein software releases are safe and unceremonious. It describes feature flags as “safeguards” against this, which it provides as its core offering: “Feature management gives you [control], Launchdarkly gives you feature management.”
Much like its largest competitor, Optimizely, Launchdarkly has been steadily segueing into the experimentation space to meet user demand. Feature flagging, its original purpose, now largely serves as a means to become embedded within the customer’s codebases. From there, experimentation is an easy upsell.
📖 Related reading: Why Motion's Head of Engineering Replaced Launchdarkly with Statsig.
Pricing model: Pro tier: $16.67/seat/month. Enterprise: Contact sales
Starter tier: $8.33/seat/month
Notable features:
A/B Testing
Scheduled rollouts
Feature flags
Watch out for: Launchdarkly is a feature flagging platform first and foremost. Once users adopt Launchdarkly and have added it to their code, adopting a new platform for experimentation is vastly more complicated than replacing it with a proper experimentation and analytics platform.
⚠️ Additional reading: What migrating experimentation platforms actually entails.
To that end, Launchdarkly is missing critical experimentation features such as global holdouts, no-code dynamic configs, impact analysis, and more. The platform also charges per seat, but doesn’t offer unlimited MAU in conjunction.
Furthermore, Launchdarkly often charges more than competitors, yet users find that their “table-stakes features” are withheld from users who aren’t on an enterprise plan.
⭐ Read the feature breakdown: Launchdarkly VS Statsig Feature Comparison
Pricing model: Unlimited seats, free feature flags, and easily-predictable prices based on usage.
Starter tier: Up to 500 million free events, and a generous program for startups.
Pro tier: Starting at $150/month, unlimited seats
Enterprise and Warehouse-Native: Requires a conversation with sales to determine user needs
Notable features:
Advanced Stats Engine (CUPED and Sequential Testing with Bayesian or Frequentist approaches)
Impact measurement/analyses
Holdouts, Experiment Layers, Dynamic Configs, and more
Warehouse-native or cloud-based implementation
In-console collaboration and discussions
Summary: Statsig is the one-stop shop for product experimentation, feature flagging, and analysis. It offers both cloud-based and warehouse-native solutions to fit its users’ dynamic needs.
Statsig boasts hundreds of favorable reviews from customers. Thousands of companies—from startups to Fortune 500s—rely on Statsig every day to serve billions of end users per month.
Watch out for: Statsig documentation doesn’t always explain why certain metrics are important, or the differences between them. One user writes “the learning curve for people new to the game is sometimes really steep,” and requires the user to get help from the Statsig team.
✅ Feature comparison: Statsig versus Launchdarkly, head-to-head.
Pricing model: Users of Amplitude are billed incrementally for Analytics, Audiences, Experiments, and CDP. Advanced permissions, real-time monitoring, and more, are only available in the highest tier.
Starter tier: Amplitude’s Analytics package is free to use
Notable features:
Behavior cohorts
Clean and clear dashboards
Unlimited users
In-platform collaboration tools
In-depth analytics
Summary: Amplitude serves as an analytics tool aiming to aid teams in enhancing their products. It boasts robust capabilities such as real-time data, tracking across devices, user behavior analysis, and top-tier security.
Watch out for: While Amplitude offers depth, it often comes with a steeper price compared to competitors. Pricing specifics can be elusive, prompting users to "reach out to sales." A source indicates that Amplitude's growth plan is priced at $70,000 for 50 million events. Another review elaborates: “It is very hard to pay that amount of money [upon] starting a project.”
Given its premium pricing and alignment toward larger corporations, Amplitude might not be the best fit for budding startups or smaller groups. Feedback from users frequently raises two specific criticisms: overcomplexity and lack of user-friendliness.
Pricing model:
Pro tier: $20/user/month (cloud-hosted)
Enterprise: Requires a conversation with sales
Starter tier: Free opensource version for up to 3 users, unlimited feature flags and experiments
Notable features:
Feature flags, experiments, projects
Customizable experiment reports
Audit logs and versioning for features
Summary: GrowthBook is an open-source feature flagging and experimentation platform that allows companies to use their existing data infrastructure and business metrics. Users enjoy GrowthBook’s lack of “black box” stats engine, allowing for transparency and visibility into data and metrics.
For more advanced use cases such as large companies and enterprises, GrowthBook offers a cloud-hosted version as an alternative to its scrappy open-source model.
Watch out for: As with most open-source A/B testing and experimentation platforms, the ball is in your court to make sure your infrastructure is scalable.
While GrowthBook is sparse on the subject, larger open-source platforms openly state that all enterprise-sized organizations should use a cloud-hosted solution, and not expect easy scaling beyond 100k events per month.
Pricing model: Optimizely’s pricing is fully custom and not available without conversing with its sales team. However, it is alleged that costs start above $36,000 and can easily scale upwards of $200,000 per year.
Starter tier: Optimizely has no publicly advertised starter tier.
Notable features:
A/B testing
Content management
SDK support
Summary: Optimizely is an online A/B testing and experimentation platform that lets users make website changes, test user interactions, and personalize customer experience. Optimizely was founded in 2010 as an optimization tool for websites, and has recently been building features to compete against newer product experimentation and feature flagging tools.
Watch out for: Users state that Optimizely’s features, being originally designed for web optimization, can be complicated and cumbersome when doing “anything more than a simple headline switch.”
Lastly, watch out for sneaky contracts and sales tactics. Multiple users report being shocked when seeing the bill. One user states they paid $40,000 and their Optimizely version still didn’t include personalization, geo-based targeting, and behavioral experiences. Another user had to drop Optimizely altogether due to unforeseen “catches in the contract and pricing.”
Take an inside look at how we built Statsig, and why we handle assignment the way we do. Read More ⇾
Learn the takeaways from Ron Kohavi's presentation at Significance Summit wherein he discussed the challenges of experimentation and how to overcome them. Read More ⇾
Learn how the iconic t-test adapts to real-world A/B testing challenges and discover when alternatives might deliver better results for your experiments. Read More ⇾
See how we’re making support faster, smarter, and more personal for every user by automating what we can, and leveraging real, human help from our engineers. Read More ⇾
Marketing platforms offer basic A/B testing, but their analysis tools fall short. Here's how Statsig helps you bridge the gap and unlock deeper insights. Read More ⇾
When Instagram Stories rolled out, many of us were left behind, giving us a glimpse into the secrets behind Meta’s rollout strategy and tech’s feature experiments. Read More ⇾