Frequently Asked Questions

Date of slack thread: 7/7/24

Anonymous: Hi! When creating a scheduled run for a query, the available options for the Time Window (Rolling) range from 1 to 28 days. This limitation poses a challenge as our experiments are expected to extend beyond 28 days. Could you consider adding more options for the Time Window? I suggest including an indefinite option that encompasses the entire duration of the experiment from initiation, which is typically what we need. By the way, in our case, using a query is essential because we cannot rely on the overall result. The overall result includes different groups, each with its own statistics, making the outcome dependent on the random proportion of each group in each experiment. Therefore, we can’t rely on it. Thanks!

Jiakan Wang (Statsig): Even if the experiment runs for longer than 28 days, we typically recommend only looking at the results for the last 7 or 14 days because it reflects the actual impact of the experiment minus any kind of novelty effect. Note that even if you choose last 7 days as the window, we still include ALL users exposed to the experiment for the entire duration, the window only controls during which time period we compare the metrics for between users in different groups. Also, the overall result is essentially the same as if we extend the explore query window to be the experiment’s entire duration.

Anonymous: I understand your recommendation and actually, I agree the shorter the duration is the better. However:

1. The groups are the essential data in our case. The total experiment results are not relevant as I explained.
2. The duration should be determined by statistically significant results. And that can be more time than the arbitrary duration that you picked.
3. I believe that this feature request is very simple to add from your side. For us it might be the relevancy of your product to our needs. Thanks

Vijaye (Statsig): Could you elaborate on #1? Are you not keeping the targeting or variants and their sizes constant over the entire duration of the experiment?

Anonymous: Hi, I’m not sure that I completely understand your question. I’ll try answering: We have several groups each gets a pre-set percentage of the users that participate, and this percentage is constant over the duration of the experiment. Anyway, our need is for an option for a bigger window - I don’t see the downside of adding it - and I can assume it should be relatively very simple. I assume that we aren’t the only ones with this issue. Your tool can be very good and this is just an arbitrary limitation of duration for an essential feature. Thanks

Vijaye (Statsig): Thanks Yoav. We want to help you with the experiment decision you are trying to make, and hence the questions to understand the setup. Doing an arbitrarily large window is not that straightforward. We actually have to non-incrementally recompute the variance for each duration window. And most experimentation experts agree that we should ignore the novelty effects that come with experiments. Hence we chose these standard windows in consultation with customers. For example, if the last 28-day window gives you a different result than the total experiment duration window, it’s prudent to use the 28-day window result over the total experiment duration. When you say “groups” do you mean variants? Could you share the link to your experiment so we can take a look?