Jacob Hurwitz
Friday, November 11, 2022 at 10:57 PM
I've read the docs on stability and resalting for partial rollouts of feature gates. We're b2b so we roll out most of our features to entire teams at once, rather than to individual users. We accomplish this by sending the team ID as a custom field; historically most of our rules have been 0% of teams, 100% of teams, or a specific allowlist of teams.
Could we do a partial rollout to (say) 10% of team IDs? This seems to be possible in the UI, but the docs don't cover this case so I'm not sure how it would work in practice.
In particular, the docs say "Gate evaluations are stable for a given gate, percentage rollout, and user ID." Are evaluations also stable for a given gate, percentage rollout, and custom field value?
And then another question not covered anywhere I can find in the docs, relevant to both user ID rollout and custom field rollout: if you increase a rollout from (say) 10% to 15%, does that necessarily include the original 10% plus a new 5%, or is this 15% group independent of the old 10% group? I'd assume the original group is included but I don't see this explicitly aggressed in the docs.
Jiakan Wang (Statsig)
Friday, November 11, 2022 at 11:06 PM
Yes - it’s pretty easy to do partial rollout on IDs other than user ID. You just need to select the corresponding ID type when you create the gate, then whenever there is a partial rollout, it will use that ID to decide which unit should pass/fail.
E.g. if a gate has “company ID” as the ID type, and then the rule is “Everyone pass 50%“, then 50% of all company IDs will pass
Jiakan Wang (Statsig)
Friday, November 11, 2022 at 11:06 PM
For your 2nd question, yes - when you increase the rollout, we just grab additional buckets from those that are failing, so users who were previously passing will continue to pass
Jiakan Wang (Statsig)
Friday, November 11, 2022 at 11:07 PM
We’ll add these details to the appropriate places on docs. Thanks for pointing this out!
Jiakan Wang (Statsig)
Friday, November 11, 2022 at 11:11 PM
https://docs.statsig.com/guides/experiment-on-custom-id-types|This doc talks about running experiments on custom ID types, at the time we wrote it we only supported it on experiments, but we’ve since added the same concept to gates as well
Jacob Hurwitz
Friday, November 11, 2022 at 11:20 PM
Thanks for the pointer. We have not yet set up a custom ID type for this team ID custom field.
To double-check my understanding, this means:
• We will _not_ get stability by team ID if we roll out on this custom field using our existing gate, since the existing gate uses the user ID type.
• If we set up custom ID types and create a new feature gate that uses this ID type, then this new gate _will_ get stability by team ID.
Is that correct?
Jacob Hurwitz
Friday, November 11, 2022 at 11:22 PM
Sorry, one more point:
• This is not possible using custom fields. We also need to modify our codebase to pass this as a custom ID (which is different from custom fields in the API). I see that `StatsigUser` has type:
```export type StatsigUser = {
userID?: string;
custom?: Record<
string,
string | number | boolean | Array<string> | undefined
>;
customIDs?: Record<string, string>;
...
};```
so custom fields and custom IDs are different, despite being similarly named.
Jacob Hurwitz
Friday, November 11, 2022 at 11:23 PM
The other question I’d ask is: can we modify the ID type of an existing feature gate (resalting is allowed, and modifying the ID type feels similarly destructive/no more dangerous), or do we have to create a new feature gate?
tore (statsig)
Friday, November 11, 2022 at 11:31 PM
I think we dont expose that in the UI, but as you transition over I’d be happy to change the ID type on the backend for you if you give us a list of gates. But correct, you would need to register the customID type in the console and start passing it in code
Jacob Hurwitz
Friday, November 11, 2022 at 11:46 PM
Jacob Hurwitz
Friday, November 11, 2022 at 11:46 PM
Also, is this best practice for setting up a partial rollout — selecting “everyone” as the criteria?
tore (statsig)
Friday, November 11, 2022 at 11:48 PM
Correct, the “Everyone” condition is a % of the unit type of the gate. Any others youd like me to migrate for now or just the create_autotag_workflows?
Jacob Hurwitz
Friday, November 11, 2022 at 11:52 PM
Just that one for now, please
Jacob Hurwitz
Friday, November 11, 2022 at 11:53 PM
My hunch is that this is going to become more frequent in the future — so we’ll either be asking you to migrate more as one-offs, would want a self-serve tool for updating ID type, or (potentially) ask for a way to change the default ID type for new gates — but that feels sufficiently far in the future that I don’t think it’s worth solving for now.
tore (statsig)
Friday, November 11, 2022 at 11:55 PM
Yeah that all seem reasonable - I think we were hesitant at first to allow people to change the ID type but seems like a reasonable advanced feature. Perhaps we will expose it behind the “Resalt” option, as both of them change the rollout in similar ways like you said above
Jacob Hurwitz
Saturday, November 12, 2022 at 12:04 AM
Thanks for all your help here — please let me know when the gate’s ID type has been changed!
tore (statsig)
Saturday, November 12, 2022 at 12:11 AM
All done, sorry for the wait
Jacob Hurwitz
Saturday, November 12, 2022 at 12:13 AM
Thanks again! :slightly_smiling_face: