Late Friday, August 25, some of our customers started reporting that our Feature Flags were not resolving correctly with the JS SDK on the latest versions of Chrome (as of this time). They hadn’t changed anything but were noticing they’d occasionally get the default value rather than the value that they expected.
What’s perplexing is that we hadn’t pushed any code change that would affect client-side evaluation. This was an interesting bug that seemingly surfaced out of nowhere.
We identified that the bug happens when we do a simple map lookup of the feature flag name. We typically hash (sha256) our feature flag names for privacy purposes, and for some reason, the flag was just missing from the map. The hash was occasionally—not all the time—incorrect.
From there, it was easy to identify that this happens in a library that we use,
js-sha256, which is a popular library that we’ve used for a while. Interestingly, the library had not changed in over 6 years.
This was a head-scratcher.
Luckily, someone had posted an issue that pointed to this bug showing up on certain Google Chrome versions. The bug garnered more visibility as more people updated to the buggy versions of Chrome.
The repro code (posted to GitHub by @slisson) was super simple and sure enough, we were able to repro it ourselves.
Turns out it’s a bug in certain recent versions of Chrome where the JITter incorrectly optimized away a specific variable.
On Sunday, August 27, we patched our JS SDK so our customers and their end-users will automatically pick up the version that mitigates the Chrome bug.
A/B testing serves to continually enhance product experiences and foster innovation—a process that is beneficial to all, even designers.
Metrics Explorer promises to redefine how you interact with your metrics by providing more analytics power directly within the Statsig platform.
A lack of “fairness” is called SRM or sample ratio mismatch. For instance, flipping a biased coin that doesn't land on heads and tails equally.
Confining experimentation to UI and website optimization falls short of capturing the remarkable business potential that experimentation can unlock.
We conducted a virtual AMA with Ronny Kohavi and our very own Tim Chan. Check out the recording.
Statsig had the pleasure of partnering with the hosts and founders of the Acquired Podcast to hold the first-ever Product Growth Forum in San Francisco.