Adhoc check 504s give the impression something is wrong #1350
Description
Internal feedback: In private probes, it's common to see logs with 504 (Gateway Timeout) every 5 minutes. The logs looks like this:
{"level":"info","program":"synthetic-monitoring-agent","subsystem":"adhoc","time":1749842581207,"message":"fetching ad-hoc checks from synthetic-monitoring-api"}
{"level":"info","program":"synthetic-monitoring-agent","subsystem":"adhoc","probe id":18626,"probe name":"prd-apps","time":1749842581227,"message":"registered ad-hoc probe with synthetic-monitoring-api"}
{"level":"error","program":"synthetic-monitoring-agent","subsystem":"adhoc","error":"rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type \"text/html\"","time":1749842881240,"message":"receiving ad-hoc check"}
{"level":"warn","program":"synthetic-monitoring-agent","subsystem":"adhoc","error":"rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type \"text/html\"","connection_state":"READY","time":1749842881240,"message":"handling check changes"}
This creates the impression that something is wrong, but it's not actually a problem. There is a 5 minute timeout on the connection if it's idle (on the server side). Since the adhoc "test" button is not used very often, the connection is going to be idle more often that not, which is why these show up in the logs.
There's nothing wrong in the agent, but private probe users may mistake these logs for a problem. Is there a way to separate "normal" gateway timeouts from errors that probe operators should pay attention to?