Stratified Sampling

Stratified sampling is a method of sampling that involves dividing the entire population into homogeneous groups known as strata (plural for stratum). Random samples are then selected from each stratum.

For example, if you had customers of different sizes (XS and XL) and randomized them into two groups - Control and Test, you'd want both Control and Test to be balanced across XS and XL customers.

Read also: Introducing Stratified Sampling

Stratified sampling is particularly useful in B2B scenarios and other relatively low volume or high variance scenarios to ensure this balance.

Automated Stratified Sampling

Automated stratified sampling is a feature that is currently in development. It aims to automate the process of stratified sampling.

Manual Assignment for Stratified Sampling

In manual assignment for stratified sampling, when setting up an experiment, you can configure overrides (e.g., force user X or Segment A into Control, force user Y or Segment B into Test). This is meant for testing; overridden users are excluded from experimental analysis in Pulse results.

If you do want manual assignment for stratified sampling, you should check the Include Overrides in Pulse checkbox. This will include the users you've manually overridden into each variant in all metric lift analyses. You can configure 100% of experiment participants into your test variants manually, or configure some subset of participants into variants manually and randomly assign the rest of your participants.

When you use the Statsig SDK for assignment, it takes care of randomization. When you control assignment of users, you're responsible for making sure users are balanced across experiment groups.

For more information, you can refer to the Stratified Sampling | Statsig Docs.

Join the #1 Community for Product Experimentation

Connect with like-minded product leaders, data scientists, and engineers to share the latest in product experimentation.

Try Statsig Today

Get started for free. Add your whole team!

What builders love about us

OpenAI OpenAI
Brex Brex
Notion Notion
SoundCloud SoundCloud
Ancestry Ancestry
At OpenAI, we want to iterate as fast as possible. Statsig enables us to grow, scale, and learn efficiently. Integrating experimentation with product analytics and feature flagging has been crucial for quickly understanding and addressing our users' top priorities.
Dave Cummings
Engineering Manager, ChatGPT
Brex's mission is to help businesses move fast. Statsig is now helping our engineers move fast. It has been a game changer to automate the manual lift typical to running experiments and has helped product teams ship the right features to their users quickly.
Karandeep Anand
At Notion, we're continuously learning what our users value and want every team to run experiments to learn more. It’s also critical to maintain speed as a habit. Statsig's experimentation platform enables both this speed and learning for us.
Mengying Li
Data Science Manager
We evaluated Optimizely, LaunchDarkly, Split, and Eppo, but ultimately selected Statsig due to its comprehensive end-to-end integration. We wanted a complete solution rather than a partial one, including everything from the stats engine to data ingestion.
Don Browning
SVP, Data & Platform Engineering
We only had so many analysts. Statsig provided the necessary tools to remove the bottleneck. I know that we are able to impact our key business metrics in a positive way with Statsig. We are definitely heading in the right direction with Statsig.
Partha Sarathi
Director of Engineering
We use cookies to ensure you get the best experience on our website.
Privacy Policy