Date of slack thread: 4/12/24
Anonymous: Dear Statsig Team, I am representative from a typeface(http://www.typeface.ai|www.typeface.ai) company and we have been relying on Statsig for feature management. Recently, we’ve encountered connection timeout errors from the Statsig APIs, as shown in the attached screenshot. This issue is significantly impacting our customers experience with our application. We urgently request your assistance in investigating this issue and providing us with the Root Cause Analysis (RCA) as soon as possible. It’s worth noting that while the errors have been intermittent and we are not currently experiencing any errors, a few users encountered them earlier today and over the past week. Thanks, Anonymous
Vijaye (Statsig): Cc <@U05RM15N0SF>
Vijaye (Statsig): Anonymous, I’ll let our infra lead chime in here. Could you pls share more details? This is on the browser? Have you made sure there is no ad blocker enabled? The only thing that would affect your users is the initialize endpoint. This is served from 12 different AZs across the world, so it’s hard to imagine them going down without us knowing.
Vijaye (Statsig): It’s pretty late in the west coast but someone will be able to help out in a few hours. Meanwhile I have verified that there are no outages currently.
Anonymous: Thanks <@U01RGJ9N4N8>,
Vijaye (Statsig): Anonymous, actually we haven’t recorded any outage on our side in the last couple days. You can see all our incidents here: https://status.statsig.com|https://status.statsig.com Could you please share how you captured these timeouts? When did this happen? What sdk? And are you able to confirm they didn’t have ad block extensions?
Anonymous: <@U01RGJ9N4N8>, If user switches to mobile hotspot or to a different network they are able to post the statsig rgstr request successfully Yes its not related to ad block extensions.
Vijaye (Statsig): Are you saying this is happening consistently on JioFiber network? We haven’t encountered this before. Is it possible they have enabled some ISP side filtering?
Anonymous: its happening consistently for them and they did not enable any ISP filtering…
Vijaye (Statsig): okay, we’ll investigate. This isn’t an issue on our side - it looks like a last mile problem related to a specific ISP. I searched around for the symptom and it looks like this affects other websites too: https://www.reddit.com/r/IndianGaming/comments/pyao7h/cant_acess_many_websites_with_jio_fiber_some/
Vijaye (Statsig): Could you please run the following commands from a JioFiber network and share the results here?
curl <https://api.statsig.com/health>
and
curl <https://featuregates.org/health>
Eric (Statsig): Hey <@U06TKQHEJJ3> - do you have any additional data you can share on scale and time frame as well? If you’d like, you’re welcome to DM me directly with the data as well to avoid publicly sharing.
Anonymous: Hi <@U01RGJ9N4N8> <@U042H7FDZ9B> I have asked two users to run the command and please find the outputs the below for those two users. User #1: curl "https://api.statsig.com/healthhttps://api.statsig.com/health"|" curl: (28) Failed to connect to http://api.statsig.com|api.statsig.com port 443 after 75025 ms: Couldn’t connect to server curl https://featuregates.org/health curl: (28) Failed to connect to http://featuregates.org|featuregates.org port 443 after 75029 ms: Couldn’t connect to server User #2: (28) Failed to connect to http://featuregates.org|featuregates.org port 443 after 132548 ms: Connection timed out cc: <@U06TPBUFG4F>
Vijaye (Statsig): This is something you’ll need to debug with Jio. It looks like your ISP is blocking a bunch of domains. Would you be able to debug this on your side? See if there are any weird DNS issues that could be affecting a simple network request.