Platform
Resources
Frequently Asked Questions
A curated summary of the top questions asked on our Slack community, often relating to implementation, functionality, and building better products generally.
How can we address differences in errors between test and control pages possibly caused by client-side redirects and internet speed?
Date of slack thread: 7/11/24
Anonymous: Hey <@U01RGJZBTLL> We occasionally observe statistically significant differences in errors between test and control pages. Our developers cannot identify a cause. Since the Autofill error occurs at exact same second with the availability of the Stable ID, and we use client-side redirects, we suspect internet speed might be affecting our experiment results.
- Should StatSig evenly distribute across different browser versions and different page load speed, considering we are using client-side redirects? We haven’t seen SRM in users across browser versions, but not sure about page speed dimension.
- How should we decide if stratified sampling or other techniques are required to ensure that your experiment groups are balanced with respect to page speed?
- Are any computations/statistical techniques you can recommend to check if there could be problems in balanced groups across page speeds.
Timothy Chan (Statsig): Hi Sergei,
- Redirect experiments that are related to logging efficiency tend to be seen in the exposures logging (eg. The Control group is failing to log exposure events at the same fidelity as the Test group). And this shows up as an SRM warning. I looked at your experiment and there is not SRM warning and the groups look well-balanced.
- Stratified sampling works best when you can predict ahead of time, who will be showing up in the experiment (as we pre-balance the experiment). It’s also great for metrics that suffer from extreme skew (eg. revenue, timespent metrics). If you have a lot of repeat users (>80%), this should work well. I don’t know if this is a great way to deal with page speed. I would look closer at the exposure event. This is usually what’s affected the most by redirects. You may be able to drill down on the app_error event by other user properties like country/browser to get some more clues what’s going on.
Join the #1 experimentation community
Connect with like-minded product leaders, data scientists,
and engineers to share the latest in product experimentation.
Why the best build with us
At OpenAI, we want to iterate as fast as possible. Statsig enables us to grow, scale, and learn efficiently. Integrating experimentation with product analytics and feature flagging has been crucial for quickly understanding and addressing our users' top priorities.
OpenAI
Dave Cummings
Engineering Manager, ChatGPT
Brex's mission is to help businesses move fast. Statsig is now helping our engineers move fast. It has been a game changer to automate the manual lift typical to running experiments and has helped product teams ship the right features to their users quickly.
Brex
Karandeep Anand
President
At Notion, we're continuously learning what our users value and want every team to run experiments to learn more. It’s also critical to maintain speed as a habit. Statsig's experimentation platform enables both this speed and learning for us.
Notion
Mengying Li
Data Science Manager
We evaluated Optimizely, LaunchDarkly, Split, and Eppo, but ultimately selected Statsig due to its comprehensive end-to-end integration. We wanted a complete solution rather than a partial one, including everything from the stats engine to data ingestion.
SoundCloud
Don Browning
SVP, Data & Platform Engineering
We only had so many analysts. Statsig provided the necessary tools to remove the bottleneck. I know that we are able to impact our key business metrics in a positive way with Statsig. We are definitely heading in the right direction with Statsig.
Ancestry
Partha Sarathi
Director of Engineering