False positives due to "context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

[SeanKilleen]

Describe the bug

I think this might be a duplicate of #934, but I'm unsure if the workaround there is still the case.

I've been running Gatus for around a week now in prod. I'm running it as an Azure Container app. I think this is possibly crucial information: I started using Gatus because of random rolling false positives that would happen in another tool (uptime-kuma), also hosted in an Azure Container app.

Suddenly today, 18 alerts for apps on the same 3 servers (hosted by a provider I"ll call "the managed provider" -- not a big cloud provider, more of a hosted VM provider), while all the monitors for other endpoints remained up.

During the apparent outage, I could reach all endpoints. The issue remained until I stopped the Azure Container App for Gatus and restarted it. (no configuration change, etc.) At that time, all alerts resolved themselves.

...however, after a few minutes of starting up again, 14 endpoints were once again marked as down.

In each instance, the error is the same: Get "[the url]": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

What do you see?

Repeated false positives with context deadline exceeded (Client.Timeout exceeded while awaiting headers), only for a certain subset of our alerts pointing to a certain hosting provider.

What do you expect to see?

No false positives.

List the steps that must be taken to reproduce this issue

No response

Version

v5.33.1

Additional information

No response

[TwiN]

I'm going to be honest, I've never heard of something like this happening unless the host running Gatus has connectivity issues.

You mentioned 14 endpoints, is that all the endpoints of the same type you have configured? Are those 14 endpoints running on the same cluster/host?

It might be worth adding a couple of other endpoints of the same type (e.g. HTTP GET requests, if that's what your 14 endpoints are) pointing to other domains hosted outside of your infra, e.g. example.com. That way, the next time it happens, you can confirm whether there's an issue with the 14 endpoints in question (if the other, unrelated endpoints you added do not trigger), or with egress connectivity.

Something you could look at too is the connectivity configuration. It's used to prevent false positives if connectivity to an endpoint does not work.

I don't want to say it's impossible for Gatus to be at fault here, but with the size of Gatus' user base, if a problem like this was happening, there would be a lot more open issues about it, since Gatus' entire job is to monitor endpoints.

[SeanKilleen]

I hear you -- it seems pretty odd to me, too. If I wasn't staring at hundreds of notifications while being able to access the apps, I'd say it wasn't likely either.

I'm testing a theory currently -- my thinking is that because I'm hosting Gatus in an Azure Container App, the SNAT pool might be getting exhausted. As I understand it, Azure Container apps use pooled SNAT, whereas Azure Container Instances does not. I'm leaning towards this theory because 1) I was completely ignorant about SNAT until now, and 2) this would explain why uptime-kuma, which I used before Gatus, had this same issue after time.

So, I moved our Gatus instance to Azure Container Instances just now. I'll update and close this in a few days if I don't see the issue re-occur.

Thanks for your quick response!

[TwiN]

Cheers, and I hope your issue goes away!