What are experimental units?

Ever wondered why some experiments hit the mark while others miss completely? It often comes down to one critical factor: the experimental units. Getting this right can be the difference between groundbreaking discoveries and misleading conclusions.

In this blog, we'll dive into what experimental units are, why they're so important, and how to identify them in different study contexts. Whether you're designing a medical trial or setting up an A/B test with Statsig, understanding experimental units is key. Let's get started!

Understanding the concept of experimental units

So, what exactly are experimental units? Simply put, they're the fundamental elements that receive treatments in a study. Think of them as the building blocks of your experiment—they're crucial for ensuring your results are valid and accurate. Identifying the correct experimental unit is essential for determining your sample size and the structure of your experiment.

Generally, an experimental unit is the entity that receives a specific treatment independently from others. It could be a person, an animal, or an object. For example, if you're studying how different exercise programs affect cholesterol levels, each patient in your study is an experimental unit.

Misidentifying your experimental units can lead to overestimating your sample size. That can skew your statistical analysis and lead to invalid conclusions. So, correctly identifying the experimental unit is paramount for reliable outcomes. As discussed in this Reddit post, in a scenario testing a new menu item's market viability, the restaurants were actually the experimental units.

In more complex experiments, you might have multiple experimental units. For instance, in split plot experiments where different treatments are applied at different hierarchical levels. Tools like the Experimental Design Assistant (EDA) help represent experimental units using nodes to visually map out relationships and interventions. This structured approach aids in ensuring clarity and precision in your experimental setups.

Identifying experimental units in various study contexts

In many studies, experimental units are the individuals, animals, or objects that receive treatments independently. For example, in our exercise and cholesterol study, each patient is an experimental unit. The exercise program is the treatment, and the cholesterol level is the response variable.

But it can get a bit tricky with animal experiments. The experimental unit could be an individual animal, a litter, a cage of animals, or even a specific part like a patch of skin. In crossover designs, the same animal receives multiple treatments over time, essentially serving as its own control.

Correctly identifying the experimental unit is crucial. If you get it wrong, you might overestimate your sample size and skew your statistical analysis. Remember the restaurant chain example? They mistakenly thought the sandwiches were the experimental units, but it was actually the restaurants.

In complex experiments, you might deal with multiple experimental units at different hierarchical levels. For instance, if you're testing diet and vitamin supplements on mice, the cage might be the unit for diet, while individual mice are units for vitamins. This setup allows you to explore interactions but requires more complex statistical analysis.

At Statsig, we understand that choosing the right experimental unit is key to accurate results. Whether you're running user-level or device-level experiments, correctly identifying your experimental units ensures your data is reliable and your conclusions are sound.

The impact of experimental units on experimental design

Experimental units directly influence how you randomize and apply treatments in your experiments. Proper identification ensures treatments are independently applied to each unit. If you misidentify them, you might fall into the trap of pseudoreplication. That's when multiple observations are mistakenly treated as independent replicates, inflating your sample size and potentially invalidating your results.

Take complex experiments like split plot designs, for example. They involve multiple experimental units at different hierarchical levels. If you're studying diet and vitamin supplements on mice, the cage might be the experimental unit for diet, while individual mice are the units for vitamins. This layered approach lets you explore interactions between treatments but requires careful planning and analysis.

Misunderstanding experimental units can lead to big challenges in your analysis. You might overestimate your sample size, skew your results, and draw invalid conclusions. Remember the restaurant chain example? Mistaking sandwiches for experimental units instead of restaurants could have led to inflated sample sizes and erroneous insights. That's why correctly identifying what your experimental units are is so important for reliable outcomes.

At Statsig, we help you design experiments that get these details right, so you can trust your data and make informed decisions.

Practical considerations and common misconceptions

Misidentifying your experimental units can not only lead to inaccurate conclusions but also waste resources. In the restaurant chain example, treating sandwiches as experimental units instead of restaurants would have resulted in flawed data analysis.

When you're representing experimental units in design tools like the Experimental Design Assistant (EDA), it's helpful to use nodes to map out relationships and interventions clearly. This visual representation ensures your experimental setup is both clear and precise.

To correctly identify your experimental units, consider the following:

• Determine the primary subject of interest: At what level are treatments independently applied?

• Assess the nature of the unit: Is it an individual, a group, or a specific part of a subject?

• Evaluate treatment interactions: Are there potential interactions between treatments? If so, adjust your experimental design accordingly.

In complex experiments, you might have multiple experimental units at different hierarchical levels. Quasi-experiments can be valuable when fully randomized experiments aren't feasible, but they require careful consideration of assumptions and statistical methods.

Choosing the right randomization unit is crucial for accurate results and consistent user experiences, especially in online experiments. Whether it's user-level or device-level experiments, as discussed in this Statsig blog post, the context and goals of your study will guide your choice.

Closing thoughts

Understanding and correctly identifying experimental units is fundamental to the success of any experiment. It ensures your data is valid, your analysis is sound, and your conclusions are reliable. Whether you're a researcher, a data scientist, or running A/B tests with Statsig, keeping a keen eye on your experimental units can make all the difference.

If you're interested in learning more, check out the resources linked throughout this blog, or explore more on the Statsig blog. Happy experimenting!