During our discussion, Joel offered an intriguing look into the current culture of experimentation at Rec Room, detailing its evolution over the past few years and backing his insights with several real-world examples. Here's a concise overview of the key points discussed:
Joel revealed Rec Room's unique approach to product development: a continuous shipping model that prioritizes "ready → fire → aim," in stark contrast to the conventional "ready → aim → fire" method. This approach helps to avoid the pitfalls of analysis paralysis, instead emphasizing rapid deployment and learning from real-world feedback.
Recently, "repeat" has been incorporated into this cycle, embodying the principle that "iteration is quality." This emphasizes refining existing features over constantly seeking to create new ones. Joel underscored the importance of experimentation in this process, as it plays a pivotal role in understanding user needs and shaping product decisions.
A significant insight from Joel was the necessity of leadership support in cultivating an experimental culture.
He emphasized that this backing from leadership is vital for promoting trust and autonomy in teams, enabling them to test every new idea and make data-driven decisions confidently. Achieving early wins can pave the way; Joel shared how early successes in experimentation propelled their journey. He recounted a specific instance where a sleek UI redesign inadvertently led to a drop in key metrics. This was due to the prominent placement of a call-to-action (CTA) message in the older version.
This discovery prompted a strategic repositioning of the CTA, demonstrating the immediate impact of their experimental approach across the company.
Joel highlighted key factors that have been instrumental in scaling Rec Room's experimentation capabilities, enabling the company to conduct hundreds of experiments annually.
A crucial step was simplifying the process for teams to initiate tests. By investing in the right tools and debunking the misconception that integrating a feature into an A/B test was laborious, Joel made experimentation more accessible. He evangelized an enlightening quote from an engineer on their team internally: “It would take about 30 minutes to make this feature an A/B test,” which effectively challenged the belief that testing would add an extra day to the development process.
Furthermore, Joel established a routine of weekly experimentation reviews. These sessions were designed to not only share results but also to foster excitement about learning from user interactions.
He emphasized that the value of an A/B test isn't solely in its deployment; understanding that a feature doesn't positively impact core metrics is equally valuable. This approach helps in safeguarding against metric regression by identifying and withholding changes that don't contribute positively.
Considering that Rec Room integrates gaming, social, and marketplace elements within its platform, defining metrics comprehensively and making informed trade-offs becomes crucial. For instance, how do you boost time spent on the platform without increasing toxic behavior? Rec Room tackled this challenge by effectively turning these 'policy decisions' into data-driven ones.
Joel shared how they utilized experimentation in determining penalties for violations such as the use of inappropriate language. They experimented with various levels of disciplinary actions, ranging from muting a player for a minute to more extended bans, and then analyzed how these interventions correlated with a decrease in toxicity, ultimately arriving at the optimal solution.
Joel’s examples underscore how experimentation at Rec Room goes beyond just UI changes, influencing crucial policy decisions. For each change to the core game logic such as giving tokens as rewards, they set guardrail metrics (such as revenue) to ensure that the changes did not regress a metric of interest.
Joel shared an intriguing example where a slight adjustment made a significant impact. An engineer tweaked the height of their users by 0.1 (essentially reducing it by 10%) to make them slightly shorter. Due to their practice of sending every event to Statsig for analysis, they were able to observe how this minor change affected a range of metrics.
Surprisingly, they noticed a notable decrease in gift box openings. This small change in height had unintentionally made it more difficult for users to open gift boxes, a critical insight that might have gone unnoticed if not for their thorough approach to test, measure, and learn from every alteration.
Throughout our conversation, Joel brought up various instances that highlighted an important lesson: our intuitions are often incorrect more frequently than we'd like to admit. We regularly implement changes anticipating an improvement in core metrics, only to encounter counterintuitive outcomes. Sometimes, we even celebrate an uptick in one metric, not realizing that it might be at the expense of another. These examples underscore the value of placing features behind a feature flag and rigorously testing them through A/B testing.
Watch the full 45-minute recording of the event for more anecdotes and insights from Joel:
VWO (Visual Website Optimization) is a great platform for website testing, conversion optimization, and more. How does it stack up against competitors like Optimizely?
“Pricing experiments,” once considered a tactic available only to the major online merchants, have been adopted as a core component within the e-commerce playbook.
Hyper-personalization often involves a dual approach: quantitative metrics for robust performance analysis and insights from customer feedback to enhance satisfaction.
Optimizely is a marketing and website experimentation platform that extends into the full-stack development space—but how does it stack up against the competition?
I've seen too many data scientists doing the right thing, yet getting frustrated and giving up because their audience isn't receptive. Here's how I'm going to change that.
How Statsig aims to introduce every San Franciscan to the tools that simplify data-driven product development at fast-growth startups and Fortune 500s alike.