Frequently Asked Questions

Date of slack thread: 5/8/24

Anonymous: I have a use case where I want to fetch metadata related to an experiment and do some checks before getting the experiment, is the best way to do this by getting experiment with exposure logging disabled in the options?

Tyler (Engineer - Statsig): Yeah - if you need to know the user’s enrollment status without having it count as them being in the experiment - checkWithExposureLoggingDisabled is the way to go.

Will add a few caveats here though: You’ll still want to make sure that you’re being “fair” in exposing users to the experiment. That is to say - if the experiment treatment causes you to potentially not expose the user (i.e. user A would get exposed if they were control, but I don’t expose them if they are in test for some reason about user A), you’ll pretty quickly introduce Sample Ratio Mismatch (SRM). This can also arise in slightly more subtle ways - e.g. if it takes me an extra 10 seconds to expose test users vs. control users, and some percentage of users leave after 8 seconds - I’m going to have imbalance there as well.

We have a healthcheck on the experiment that will let you know if you are experiencing SRM - but if you’re manually exposing we’re pretty likely to point to that as the likely culprit.

Tyler (Engineer - Statsig): Oh - implied above - but you will want to still expose the user without exposureLoggingDisabled (i.e. a normal exposure) at some point as well to make sure we actually know they do get exposed eventually so we can run analytics on them.

Anonymous: gotcha thanks for replying Tyler! For more context what im trying to do is check if the user has the ID Type being used by the experiment (some users dont apply to the experiment in that case), If the user does not have the ID type then not exposing them to the experiment.

Tyler (Engineer - Statsig): Ah - perfect! No SRM concerns there then. I’ll also flag we exclude {unitId: “”} from the analysis - but will still give them one of the treatments - as what the out-of-the-box behavior there is for that scenario.