Frequently Asked Questions

Date of slack thread: 4/18/24

Anonymous: In a given experiment that’s defined by a user ID type: once a user is exposed to that group, will they always be exposed to that group, or can they be exposed to a different group for a later session?

tore (Statsig): I’ll add a bit more info here, since this is a common point of confusion. A lot of times people assume we track all users by their ID to know who is in which group, but that doesn’t scale well. Instead, we make it deterministic: given a stable state of experiment, targeting criteria, and a stable user object, you will get the same evaluation, always.

Rather than switching groups, the more likely thing to happen is that the user is no longer eligible for the experiment (fails the targeting gate). If you aren’t using a targeting gate, you are good to go. If you are, and the user properties change in a way that causes them to fail the targeting gate, a subsequent check could go from bucketed into an experiment group to not eligible for the experiment.