Products
Solutions
Resources
“I wasn’t just testing the name of the chart.”
Madelyn Tarr was responding to a question from Cary Schwartzstein about a rumor that she had, in fact, run an A/B test on the name of an analytics chart.
Along with Chad Swenson, I’d asked these three Statsig + Amplitude PMs to chat with me about how they prioritize experiments. This one didn’t sound like a high-priority test to me, but as a words guy, I’ll admit I was curious to hear more.
“I was working on a journey orchestration product, and it needed a name,” Madelyn explained. “Most of us wanted to call it ‘Journeys.’ But there was also an Amplitude chart type called ‘Journeys.’ So we ran a test to see if giving the chart a different name would impact usage.”
“So you could steal ‘Journeys’ if it didn’t?” asked Cary.
“With the other team’s permission!” said Madelyn. “We needed to standardize on a name to move the NPI process forward. This was a critical experiment to get aligned on that.”
“What happened with the test?” I asked.
“Yeah, we stole ‘Journeys.’”
“I’m surprised that name didn’t have an effect,” Chad said. “I love that chart.”
“Well, then the product got deprioritized,” said Madelyn, “so we gave it back.”
Over the next half hour, we discussed the risks, impacts, effort, and statistics that help them decide which product experiments to run. And although only 15–20% of experiments end up in the positive direction, these insights were all wins.
Let’s start with intros. How did you join Statsig + Amplitude, and what’s your area of focus?
Chad: I’ve been at Amplitude since last fall. I was working on Session Replay, a few other things like Zoning, Heat Maps, basically everything under the Experience Analytics umbrella.
When I heard about the Statsig opportunity from Chris Yu, I was instantly like, “OK, let’s do it!” I was bullish on Statsig even pre-acquisition; it’s a strong brand, and I think the previous team’s trajectory was really amazing.
There were all sorts of cool features that had been in development internally for Statsig’s longer-term vision, and that’s what I’m working on now. We call it Statsig Labs. It’s Session Replay, AI Evals, Observability, and a bunch of other internal tools. Pretty much anything outside of experimentation and feature gates.
We just have to figure out what’s the next best thing to make.
Madelyn: You’re the bets guy.
Chad: That’s me!
Madelyn: So I joined Amplitude in November of last year, not far off from when Chad did. I was originally PMing for both the Activation team and the SDK team. That was a lot of fun, but when the Journey Orchestration product got cut, it was time for something new.
When people hear that, they’re always like, “Oh, I’m so sorry.” But I firmly believe, as a business, you have to make prioritization decisions. I think it was the right call for the business. As that was winding down, Chris Yu approached me about Statsig. I similarly feel really bullish on Statsig.
I came from a startup, so I got to experience the excitement of the modern development era; I was shipping a lot of our front-end code as a PM without a technical background. I think in the long term, that’s how all companies will operate, and I really believe Statsig will be critical infrastructure for that. Feature gates, the ability to turn something on or off really quickly, will be important as companies move fast, try things out, or, quite frankly, introduce more slop.
At Statsig, I’m working on the Feature Management section, which is Feature Gates, Dynamic Configurations, and Parameter Stores. It’s all the infrastructure before you get into the actual experimentation.
Cary: So I’m the newest here! I joined about six months ago, directly to Amplitude’s Feature Experimentation team.
Joining Statsig made logical sense. We’d seen Statsig as a strong competitor, and I’d done a lot of competitive analysis to see how Amplitude’s experimentation offerings matched up. It was like a bomb dropping to realize that Statsig was now in our remit.
I’m on the SDK and config delivery; I think of it as the developer experience side of Statsig. I make sure all of the service areas that developers interact with are easy for them to get their hands on, both within the UI, like debugging what’s going on, or in the SDK itself, like the methods they use to get up and running with Statsig.
When teams are new to experimentation, it can be hard to know what to even test. How do you think about that? When do you run an experiment versus just ship a feature?
Madelyn: I think if we’re honest, everything should be an experiment. With the exception of, you know, some minor changes or bug fixes. But any new feature should be experimented on.
I think why teams don’t is often because of capacity. There are a lot of moving pieces when you’re building products, and that’s even more true now with the speed of the AI era. Teams think they don’t have the time to look at their features through the lens of an experiment.
Whatever you do, experimentation is an added step. Depending on your tooling, that might mean you have to go loop in a whole other team, or even your data scientist, just to validate a hypothesis. PMs will be like, “I can’t get those resources; guess I just have to ship the feature.”
Chad: The old, “We’ll test it in production.”
Madelyn: Exactly. It’s important to move quickly, but also to move safely. Experimentation or Feature Gates is how you do that.
In a past product life of mine, I worked on a media brand, and we didn’t have much qualitative feedback from customers. It was really hard to sit customers down and have a conversation. So if I wanted to change something without the risk of dropping our revenue to zero, we had to experiment and root the decision in behavioral data.
That’s usually my barometer for when to experiment, if something is a risk to revenue. But in a perfect world, I’d experiment on everything.
Cary: Building on that a bit, I think we address that capacity issue really well on Statsig, where we blur the line between releases and experiments.
If you look at Feature Gates, Madelyn’s product, everything is inherently an A/B test. You can attach metrics directly to that feature gate and be able to test without having to set up the entire infrastructure of an experiment. You get that out of the box, straight away.
Madelyn: That’s a great call. And I may be a little biased, but I believe feature gates are the best way to build in the modern era.
If every PM and designer now has access to code, and changes are happening as rapidly as they should be, the only way to do that safely is to have nearly everything live behind a feature gate. That way, you can turn it off quickly, but also be alerted if something goes wrong. You run the release as an A/B test baked into the feature.
Say you launch something you think is no big deal, like a button color change. Revenue drops to zero. Turns out you broke the button altogether. I remember fixing site outages like that on my D2C brand. The site would be down for maybe ten minutes, but it’d end up that $40,000 was lost in that short window of lack of conversion capability.
Those are the types of issues where you need to get alerts.
As someone who does things like forgetting to turn the lights off when he goes out, that terrifies me. Is experimenting always a matter of stakes?
Chad: I think it also depends on traffic. Most of my past was in D2C, and we had so much traffic that we were optimizing our onboarding all the time. Every single feature we came up with was an experiment.
But in the B2B context, the amount of things you can run a real experiment on is an order of magnitude smaller, because you just don’t have as much traffic.
It often happens that the next best thing to build, like the feature that one of your biggest customers is requesting, isn’t one you can get statistical significance on. Maybe only a few customers are going to use it.
Cary: That’s a good thought. Like, if you’re getting enough traffic, your experiment can prove statistical significance. If you’re not, it’s going to be underpowered.
Chad: But I think you can still run an underpowered “experiment” on things like that with a feature gate. You might not get statistical significance, but you can still make sure you don’t have a regression. And that you don’t hurt the metrics you care about most. You may even be able to see some directionality.
That’s a muscle I’m trying to build for myself and my engineers. Don’t just give up on a test because it isn’t going to be stat-sig. You’re still going to launch more safely. And using Statsig is helping us lean into that cultural mindset.
Madelyn: Another benefit of that muscle: it helps you be more rigorous about measuring performance overall.
Without experimentation, it’s not uncommon for teams to ship a lot but only measure performance on the number one thing their leadership or business cares about. So you wouldn’t measure something like a small UI change. But if there’s an issue with it, you won’t know until six or seven months later, when weird behavior is impacting your business.
That’s why it’s so important to make it easy to configure experiments. The easier we can make testing for our users, the more they’ll be able to build a rigorous experimentation culture.
Cary: I think it is still important to consider statistical significance, though. Like, be aware that you might see directionality, but it’s not statistically sound.
If I were a PM, I’d say you’ve convinced me; I want to run as many experiments as I can, even if I can’t always get stat-sig. But I probably couldn’t do that practically, at least not at first. How do you prioritize?
Chad: I think about it more from the perspective of the feature. Like, what’s the impact? What’s the likelihood of that impact to the customer? What’s the engineering work for it? Which customers are asking for it?
You can stack rank all that in a spreadsheet, but then you also have to think about how you measure the impact and success of the test. Like, can you actually get statistical significance? Is there a KPI that this is going to drive?
Or if it’s something that’s just for a few customers, maybe you can talk to them and get their qualitative feedback. But in that case, an experiment becomes less about moving a specific metric and more about safety, so you’d switch to testing observability metrics and latency, things like that.
Madelyn: I decide the priority of an experiment based on the information that I need most, or that’s most pressing.
There are different types of experiments; I don’t mean A/B tests vs. multiarm bandits, I mean, like, different scenarios where you’re trying to understand different things. Experiments don’t have to be part of rolling out a feature.
Like with the Journey name test I talked about at the beginning, I was experimenting to get data that I needed to make a case about the product name and move NPI forward. It wasn’t related to launch performance or core product functionality. But it was still pressing information.
Cary: Yeah, you have to make sure that what you’re learning is actually helpful, not just experimenting willy-nilly. Like, don’t worry about running experiments on very small segments, because those won’t give you anything you can run with.
Or if you’re running personalization and testing whether that affects retention, make sure there’s a measurable outcome that would help you improve it, and that you aren’t just confirming something you already know.
What about the effort of designing an experiment? Feature gates make that easier, but there’s still work beyond building a feature to create an effective experiment. Does that play into your decision?
Chad: That definitely plays into it and can be another bottleneck. Engineers have different levels of experience with experiment design. Some naturally think about it as they build a new feature, and with others, I need to document how I think the test is going to work for them to implement it. Honestly, we’re all still learning.
But with something like an A/B test, it’s pretty straightforward: the control is the status quo, the variant is the new feature.
And for marketing tests, maybe you want five or six different treatments, but each one is pretty low-effort. Especially with AI, you can really pump out the variants. It’s fun!
Cary: That’s what I was going to say! With AI, the cost of development has gotten a lot lower, so it’s a lot easier to design and run tests.
It’s also important to make the right decision with those tests. But it doesn’t cost as much to crank them out.
Madelyn: I think that gets back to experimentation culture, and how it’s evolving.
It’s similar to what Amplitude was part of maybe seven or eight years ago, when the industry first started talking about “analytics culture.” Now, behavioral analytics is absolutely required for your product.
I think experimentation will be the next layer to go through that cultural shift. When we talk to different prospects in different markets today, we see that same early pattern, that some people are really comfortable with experimentation and assume it’s a requirement for the future, and that others don’t operate that way.
Experimentation is only going to become more important as development gets cheaper and faster. Not to assume everything is slop. But to move really quickly, you want things to be low stakes and reversible.
The easier we can make it to experiment with Statsig, the better we’ll be able to democratize, evangelize, and share the experimentation culture.