Last week, I hosted Kevin Anderson (Product Manager, Experimentation at Vista) for a candid fireside chat filled with interesting anecdotes and tips for building an experimentation-driven product culture. You can view the full recording below:
Having been in the domain for nearly two decades and currently working as a Product Manager, liaising with internal and external stakeholders, and running a weekly newsletter (Experimental Mind), Kevin has a deep understanding of how the space is evolving. Below is a teaser of five interesting topics that were discussed in our chat:
Kevin noted how experimentation has evolved from focusing solely on conversion rate optimization on websites to influencing the entire product development culture. This shift has been facilitated as vendors have entered the space, aiming to democratize big-tech tooling, which has made it easier to adopt experimentation.
As a PM, Kevin focuses on identifying and eliminating points of friction to facilitate more high-quality experiments across the organization.
He shared various tactical approaches, such as maintaining a public Slack channel where anyone can post questions about experimentation and consistently speaking with stakeholders to gather insights on their hesitations towards experimenting.
What stood out to me was Vista’s commitment to transparency in the experimentation process — everyone has access to the Statsig project and can view data — so everyone benefits from the learnings.
Kevin highlighted the importance of measuring the success of experimentation initiatives—a priority he set when he joined the company. He shared both qualitative and quantitative ways to measure progress but emphasized focusing on quantity initially—i.e., running more experiments. He argued that you can't improve quality unless you're already running numerous experiments.
Kevin noted that companies often underestimate the effort required to build feature flagging and experimentation in-house. While it may be feasible to develop a V1 to handle specific use cases, advancing to the next steps requires significant investment and resources—often limiting progress.
These days, the baseline feature set available from third-party vendors is very high. Mature companies with a robust in-house platform, like Microsoft or Booking, started their development 10-20 years ago to reach their current status. Kevin's advice was to focus on acquiring the best solutions available in the market to accelerate the adoption of experimentation—a strategy he embraced at Vista when deciding to partner with Statsig.
Kevin also shared some hot takes, such as pushing back against the common belief that you can't do experimentation without C-suite support. While it is easier when it comes from the top, it is certainly not a requirement.
Kevin's approach was to identify champions across cross-functional teams, even without an explicit mandate from the top, and achieved significant results in previous roles. Don't wait for a CXO to enforce the culture; just find the right allies (even from other teams in the company) to start with, achieve small wins, and kick off the experimentation flywheel!
These were my favorite highlights, but I'd highly recommend listening to the full conversation.
In this tutorial, we show how to setup Next.JS Feature Flags with the Statsig SDKs. We'll use Next.JS App Router, Statsig for product analytics, and also share how to deploy this app with Vercel.
Statsig has four data tools that are ideal for earlier stage companies: Web Analytics, Session Replay, Sidecar (low-code website experimentation), and Product Analytics.
Boost your site's conversions with Statsig's Web Analytics, low code experimentation Sidecar, and Session Replay. Optimize efficiently with deep insights and testing tools.
An experiment scorecard is more than just a collection of numbers; it's a narrative of your experiment's journey from hypothesis to conclusion.
Statsig and PostHog both offer suites of tools that help builders be more data-driven in how they develop products, but how do they differ?
Product analytics reveals user interactions, driving informed decisions, enhancing UX, and boosting business outcomes.