📊 Storytelling with Experiments

Storytelling with Experiments

Good experimenters who bring others along, are often good storytellers. We've heard from many of our customers that they want to offer a narrative arc with experiments that includes the ability to set up context, dive into conflict and offer resolution - a narrative layer around our out-of-the-box Scorecards.

Today we are adding the ability to include live, interactive widgets in Experiment Summary that allow experimenters to craft this narrative for their audience.

A few examples of how customers are using this:

• Embedding the results of a Custom Explore Query to add context from relevant deep dives (e.g. analyzing the experiment time period while removing outliers like Black Friday, or specific days that had data blips)

• Adding rich charts such as the conversion funnel being experimented on to contextualize experiment Scorecards

• Breaking out metrics to match the mental model for good AI experimenters, for example:

  • Direct model measurement : latency, tokens produced, cost

  • Direct user feedback : explicit - thumbs up/thumbs down, implicit - dwell time, regeneration requested

  • Longer-term effects : user activity level over the next week, retention, subscriptions

The original experiment Scorecard is still available as part of Experiment Summary.

Our eventual goal is to have all the context around an experiment - including experiment design, critique, q&a and readouts- centralized in one place. This is the first step toward that.

This feature is rolling out gradually. To embed rich charts in your Experiment Summary, go to the "Summary" tab in your experiment, tap into "Experiment Notes" and select the "+ Add Charts" CTA on the right. Happy storytelling!