At Statsig, we collaborate with a diverse range of companies scaling their experimentation culture, spanning industry leaders, fast-growing startups, and cutting-edge AI enterprises. Earlier this year, we introduced our Warehouse Native (WHN) offering, granting new customers the flexibility to select their deployment model according to their unique needs and use cases.
Companies exhibit significant variations in their existing experimentation approaches, toolsets, resource allocations, and policies. While some customers enter the picture with a distinct architectural inclination, others are in the midst of evaluating their alternatives. In light of this, we wanted to provide some guidance based on our customer onboarding experiences.
Below is a summary of key criteria to consider when making your decision between the two modes of deployment:
|Warehouse native (WHN)
|Primary source of metrics come from Statsig SDKs or CDPs like Segment. Some metrics can still come from a warehouse.
|Warehouse is the primary source of metrics, making WHN ideal when wanting to reuse existing data pipelines and computation.
|Automated experimentation for every experiment and product launch, especially with metrics derived from event logging.
|Flexible analysis on top of your existing source of truth metric data.
|Data team involvement
|Involvement is optional but recommended for experiment design and readouts.
|Necessary for setting up the warehouse connection and configuring core metrics, but not mandatory for every experiment.
|TCO is slightly lower. No warehouse costs involved.
|TCO includes Statsig license + costs incurred for computation and storage in your warehouse.
|An integrated end-to-end platform that spans SDKs for feature rollout, experiment execution, analysis, and experiment readouts.
|Modular: You can opt for the integrated end-to-end platform or choose to use only a subset of capabilities, such as assignment or experiment analysis.
This framework is designed to provide directional guidance, intended to ease organizations in their decision-making process. Specific requirements can vary, so it's essential to carefully evaluate your situation. That said, the following factors that build on the table above merit consideration:
This is often a crucial factor for companies that have a data warehouse as their primary source of metrics and want to perform experiments using their existing, reliable datasets. In this case, the warehouse-native approach allows for experiment analysis directly within the data warehouse, preserving privacy and data ownership and avoiding data egress.
Building on the preceding point, enterprises with pre-established, trusted metrics in their data warehouse usually seek an experimentation platform that directly interfaces with these metrics.
On the other hand, Statsig's cloud-hosted solution is perfect for organizations seeking a reliable method for creating metrics. It's more engineering-friendly and doesn't necessitate additional data work. You can simply use the Statsig SDK to log events and send them to Statsig, where they can be seamlessly integrated with any precomputed metrics from a data warehouse, if needed. While metrics can be easily imported in the cloud-hosted version, using WHN may be a better choice if all your metrics are derived from the warehouse.
Organizations already leveraging other SaaS tools for customer data and metrics find synergy in adopting a cloud-hosted experimentation platform. For instance, companies already utilizing tools like Segment discover a straightforward path for direct ingestion of event data into the cloud-hosted Statsig version.
Whether an organization already has a feature flagging tool can influence their decision. For instance, customers considering migration from an existing vendor like LaunchDarkly might be more accustomed to cloud workflows. However, as many organizations are transitioning towards maintaining a single source of truth for their metrics and making experiment data available in a warehouse for rapid analysis, they might prefer a warehouse-native tool that operates on top of these metrics.
In general, cloud-hosted solutions typically offer a slightly lower total cost of ownership when factoring in warehouse management expenses related to compute and storage, as well as any initial setup costs.
Organizations enforcing stringent data egress policies pertaining to user-level data from their warehouse may gravitate toward a warehouse-native deployment. However, it's crucial to note that private information such as names and email addresses isn't requisite for experiments, as they operate based on user ID. We have some of our largest enterprise customers trusting to run their workloads on our cloud-hosted offering, leveraging hashing or private attributes to avoid logging IDs or sensitive data.
Considering the diverse stages of cultivating a data-driven culture of experimentation, organizations starting their journey often find the cloud to be a more accessible starting point, as it is so easy for developers to adopt. In contrast, a warehouse-native deployment requires a trusted and reusable set of metrics/events within a warehouse.
The cloud-hosted option places lower demands on data team involvement in experimentation, as all teams, including Engineering and Product, gain access to the console for metrics and analysis. Statsig Cloud-Hosted caters to users who favor automation, minimizing the need for manual data work to compute experiment results. It fosters simplicity and accessibility for the entire organization, encompassing engineering, product, and data teams. Warehouse-native entails some data team support, though not for every experiment.
As a result, Statsig Cloud often presents a seamless initiation point for organizations new to experimentation. It accommodates companies at varying levels of data maturity, offering a "just works" approach. This proves invaluable for organizations lacking data engineers or extensive data management resources, facilitating swift experimentation without the complexity of managing a data warehouse or intricate data processes.
Organizations that extensively customize their metrics may discover that defining these metrics within their data warehouse and orchestrating precise experiments using a warehouse-native platform is more convenient. This is because complex data transformations and the integration of new data sources are more straightforward with a data warehouse.
Organizations seeking in-depth metric analysis and insights through built-in analytics (Metrics Explorer) are best suited to Statsig Cloud-hosted. Most other features align between our cloud-hosted and warehouse-native offerings, and we are working diligently to bring the two offerings to parity.
Related: See the full comparison table between Statsig Warehouse Native versus cloud-hosted.
TLDR: The choice between cloud-hosted and warehouse-native isn't about one being superior to the other but rather about selecting based on your current tooling, processes, and objectives. Don't hesitate to reach out if you would like to discuss further what aligns with your goals.
VWO (Visual Website Optimization) is a great platform for website testing, conversion optimization, and more. How does it stack up against competitors like Optimizely?
“Pricing experiments,” once considered a tactic available only to the major online merchants, have been adopted as a core component within the e-commerce playbook.
Hyper-personalization often involves a dual approach: quantitative metrics for robust performance analysis and insights from customer feedback to enhance satisfaction.
Optimizely is a marketing and website experimentation platform that extends into the full-stack development space—but how does it stack up against the competition?
I've seen too many data scientists doing the right thing, yet getting frustrated and giving up because their audience isn't receptive. Here's how I'm going to change that.
How Statsig aims to introduce every San Franciscan to the tools that simplify data-driven product development at fast-growth startups and Fortune 500s alike.