When people think about e-commerce experiments, it's often Amazon that first comes to mind.

Amazon has become famous for strategically enhancing the checkout experience by precisely positioning buttons like "Add to Cart" to propel purchase decisions. In fact, most of the available literature on experimentation—whether it's from testing tool vendors or influential voices in product and data science—tends to center around UI-focused experiments.

These often involve tweaks like altering color schemes, adjusting font sizes, and repositioning elements, all geared toward achieving higher click-through rates.

Undoubtedly, these UI experiments yield significant impacts. Amazon, for instance, has mastered the art of streamlining its checkout process to minimize distractions and maximize completed purchases.

However, confining the discussion to UI experiments alone falls short of capturing the remarkable business potential that experimentation can unlock.

In this blog, we'll delve into examples that cast Amazon as the protagonist, showcasing how experimentation extends far beyond surface-level UI adjustments.

1. Experimenting on core business logic: Balancing upselling and customer satisfaction

Consider the monumental success of Amazon Prime—the subscription service that generates substantial recurring revenue for the company. It's no wonder that a pivotal focus of Amazon's strategy revolves around a key business objective: boosting Prime subscriptions by converting non-Prime members.

Yet, Amazon's commitment to customer satisfaction can sometimes present unintended challenges: The company's dedication to its customer-centric ethos prompts it to allocate its surplus delivery capacity to fulfill orders for non-Prime members expeditiously.

While this enhances the experience for these non-Prime customers, it inadvertently blunts the incentive for them to transition to a Prime membership, as their orders already arrive promptly.

An intriguing hypothesis emerges: Could the allure of early deliveries for non-Prime members be impacting their inclination to convert to Prime, and could optimizing the delivery experience strike a balance between customer delight and driving Prime conversions?

Amazon might explore experimenting with a subset of customers. This test group could experience deliveries with a slight delay, enabling Amazon to assess the impact on their subsequent behaviors—both in terms of orders and conversions to Prime.

This experiment's success hinges on Amazon's ability to weigh the potential loss in Customer Lifetime Value (CLV) among non-Prime members against the gain in CLV from converted Prime customers. Here is what the experiment might look like.

This example underscores a crucial principle: Every facet of core business logic translates into a product feature—an interactive lever that can be pulled to test various business hypotheses.

Experimentation isn't confined to UI modifications; rather, it permeates deep into the very foundations of a business, offering the potential to optimize and innovate.

In our recent AMA with Ronny Kohavi, he discussed Amazon's discovery that a single product detail page, including listings from both third-party sellers and Amazon itself (rather than separate 'merchant pages'), performed significantly better. This insight influenced their strategic decisions, leading to a greater focus on building a marketplace.

Consider this a prompt to reflect on your own core business decisions. How might they correspond to elements of your product or code? How could you harness experimentation to refine, redefine, and elevate your business strategies?

Think about how you've made decisions for these other key aspects of your business and we'll continue using Amazon as an example to discuss how experimentation guides key decisions.

2. Experimenting with pricing strategies

Building a winning pricing strategy doesn't require guesswork. Consider dynamic pricing, which involves recalibrating prices in real time, taking cues from variables such as demand fluctuations, temporal patterns, and competitive dynamics.

However, before incorporating such an algorithmic approach, you'd want concrete insights into its potential impact on core business metrics, particularly revenue generation.

A logical starting point involves comparing the performance of two distinct groups: the treatment cohort exposed to the dynamic pricing algorithm—where prices change based on real-time factors—and the control group that experienced static pricing for identical products.

Extending this mindset, a similar approach can be applied to other pricing paradigms.

Imagine scrutinizing the efficacy of flash sales, deep discounts, surge pricing during peak demand, and even dynamic product bundling with enticing rebates. Each of these strategies has the potential to influence both short-term spurts and long-term trajectories, positioning them as compelling contenders for systematic experimentation.

It's worth noting that while Amazon may have fine-tuned its pricing strategy and some experiments might be replicable, a one-size-fits-all approach won't suffice. It's essential to identify what resonates best with your specific business.

3. Experimenting on recommendation algorithms

Recommendations serve as a vital driver for cross-selling, ultimately amplifying user revenue. However, the key lies in refining the algorithm with optimal weights to steer desired customer actions.

For instance, should the algorithm give precedence to showcasing products from different categories, or should it concentrate more on items within the same category?

An experiment can be run in which the test group can receive recommendations from diverse product categories, tracking the average order value along with key engagement metrics like click-through rate (CTR), bounce rates, and conversions. Depending on the nature of your business, you may expect variations across user segments.

Similarly, algorithms like in-app search can be optimized to enhance accuracy, pertinence, and personalization.

In the AI era, companies are harnessing the capabilities of OpenAI's ChatGPT to augment in-app discovery and engagement. A case in point is Instacart's unveiling of the "Ask Instacart" feature earlier this year, exemplifying this trend. Read our blog about experimenting with generative AI to learn more about how you can conduct your own AI experiments.

Introducing Statsig Warehouse Native

Interested in deploying Statsig directly within your own data warehouse?

Learn More

4. Experimenting on payment and shipping methods

Payment and shipping methods can impact a consumer's purchase decision, and providing the right choices can drive sales. Flexibility in payment options can attract new customers who might not have otherwise considered buying the product.

In markets like India, cash-on-delivery options are highly favored within key segments and can drive substantial sales volumes. Amazon was an early adopter of this model.

Of course, this complexity is accompanied by added intricacies, but experimenting with these options on a smaller scale can verify whether implementing such choices is worthwhile for a specific region.

If the experimental group demonstrates a noteworthy increase in conversion rates and average order value compared to the control group, this indicates that offering diverse payment options is indeed a valid strategy for enhancing sales and promoting larger purchases.

Similarly, exploring various shipping methods might help capture more business. In markets where price sensitivity is paramount, offering lower-cost options like extended shipping times or pickup alternatives can potentially entice consumers seeking cost-effective solutions, even if it means sacrificing some convenience.

Conversely, certain customer segments might be willing to pay a premium for expedited shipping or enroll in services like Prime.

5. Experimenting on customer service

Customer obsession is a leadership principle at Amazon, guiding many pivotal choices they've made. However, balancing customer experience with cost optimization can be challenging. Here again, experimentation offers a way to find that equilibrium.

For starters, you can experiment with various return policies, fine-tuning the balance between customer convenience and abuse prevention. This can enhance customer satisfaction and loyalty.

When tackling business challenges like reducing refunds, consider testing an MVP to uncover solutions. This not only conserves resources but also reveals valuable insights into customer behavior.

Considering a chatbot for your business? Test an AI-powered chatbot with a group to handle common queries. While this may speed up issue resolution, maintaining an acceptable escalation rate is vital for preserving customer satisfaction.

For global giants like Amazon, tailoring services to specific regions has the potential to boost engagement. Test offering multilingual support, allowing customers to select their preferred language. If this drives higher satisfaction and greater use of multilingual options, catering to diverse linguistic preferences can positively impact engagement and loyalty.

Experimentation's potential is vast and extends beyond buttons on the user interface. Often, less obvious experiments yield the most substantial revenue impact.

TLDR:

The key insight is that nearly every major business decision involves a testable product feature. The examples provided encourage viewing your own decisions through the experimentation lens, illustrating the limitless potential for refining your approach.

Join the Slack community

Connect with our data scientists and engineers, and ask questions or just hang out with other cool folks that believe in an experimentation culture!

Join Community