Last week, I hosted a virtual fireside chat with Joel Witten, Head of Data at Rec Room. Joel captivated us with his experiences in fostering a data-driven culture within the company, illustrating the journey from a mere handful to an impressive 150 experiments annually.

During our discussion, Joel offered an intriguing look into the current culture of experimentation at Rec Room, detailing its evolution over the past few years and backing his insights with several real-world examples. Here's a concise overview of the key points discussed:

Ready → Fire → Aim → Repeat

Joel revealed Rec Room's unique approach to product development: a continuous shipping model that prioritizes "ready → fire → aim," in stark contrast to the conventional "ready → aim → fire" method. This approach helps to avoid the pitfalls of analysis paralysis, instead emphasizing rapid deployment and learning from real-world feedback.

Recently, "repeat" has been incorporated into this cycle, embodying the principle that "iteration is quality." This emphasizes refining existing features over constantly seeking to create new ones. Joel underscored the importance of experimentation in this process, as it plays a pivotal role in understanding user needs and shaping product decisions.

Leadership buy-in is crucial for successful experimentation

A significant insight from Joel was the necessity of leadership support in cultivating an experimental culture.

He emphasized that this backing from leadership is vital for promoting trust and autonomy in teams, enabling them to test every new idea and make data-driven decisions confidently. Achieving early wins can pave the way; Joel shared how early successes in experimentation propelled their journey. He recounted a specific instance where a sleek UI redesign inadvertently led to a drop in key metrics. This was due to the prominent placement of a call-to-action (CTA) message in the older version.

This discovery prompted a strategic repositioning of the CTA, demonstrating the immediate impact of their experimental approach across the company.

Honing the culture of experimentation

Joel highlighted key factors that have been instrumental in scaling Rec Room's experimentation capabilities, enabling the company to conduct hundreds of experiments annually.

A crucial step was simplifying the process for teams to initiate tests. By investing in the right tools and debunking the misconception that integrating a feature into an A/B test was laborious, Joel made experimentation more accessible. He evangelized an enlightening quote from an engineer on their team internally: “It would take about 30 minutes to make this feature an A/B test,” which effectively challenged the belief that testing would add an extra day to the development process.

Furthermore, Joel established a routine of weekly experimentation reviews. These sessions were designed to not only share results but also to foster excitement about learning from user interactions.

He emphasized that the value of an A/B test isn't solely in its deployment; understanding that a feature doesn't positively impact core metrics is equally valuable. This approach helps in safeguarding against metric regression by identifying and withholding changes that don't contribute positively.

Trade-offs and guardrail metrics

Considering that Rec Room integrates gaming, social, and marketplace elements within its platform, defining metrics comprehensively and making informed trade-offs becomes crucial. For instance, how do you boost time spent on the platform without increasing toxic behavior? Rec Room tackled this challenge by effectively turning these 'policy decisions' into data-driven ones.

Joel shared how they utilized experimentation in determining penalties for violations such as the use of inappropriate language. They experimented with various levels of disciplinary actions, ranging from muting a player for a minute to more extended bans, and then analyzed how these interventions correlated with a decrease in toxicity, ultimately arriving at the optimal solution.

Joel’s examples underscore how experimentation at Rec Room goes beyond just UI changes, influencing crucial policy decisions. For each change to the core game logic such as giving tokens as rewards, they set guardrail metrics (such as revenue) to ensure that the changes did not regress a metric of interest.

Create a free account

You're invited to create a free Statsig account! Get started today, and ping us if you have questions. No credit card required, of course.

Create My Account

Small changes can have a significant, often unexpected effect

Joel shared an intriguing example where a slight adjustment made a significant impact. An engineer tweaked the height of their users by 0.1 (essentially reducing it by 10%) to make them slightly shorter. Due to their practice of sending every event to Statsig for analysis, they were able to observe how this minor change affected a range of metrics.

Surprisingly, they noticed a notable decrease in gift box openings. This small change in height had unintentionally made it more difficult for users to open gift boxes, a critical insight that might have gone unnoticed if not for their thorough approach to test, measure, and learn from every alteration.

And finally, our intuition is wrong more often than we realize

Throughout our conversation, Joel brought up various instances that highlighted an important lesson: our intuitions are often incorrect more frequently than we'd like to admit. We regularly implement changes anticipating an improvement in core metrics, only to encounter counterintuitive outcomes. Sometimes, we even celebrate an uptick in one metric, not realizing that it might be at the expense of another. These examples underscore the value of placing features behind a feature flag and rigorously testing them through A/B testing.

Watch the full 45-minute recording of the event for more anecdotes and insights from Joel:

Check out our sandbox project

Take our “sandbox” project for a spin! In this project, we have generated real-time sample data for you to explore and experiment with.

Test Drive Now