Skip to content

Adhoc check 504s give the impression something is wrong #1350

@peterschretlen

Description

@peterschretlen

Internal feedback: In private probes, it's common to see logs with 504 (Gateway Timeout) every 5 minutes. The logs looks like this:

{"level":"info","program":"synthetic-monitoring-agent","subsystem":"adhoc","time":1749842581207,"message":"fetching ad-hoc checks from synthetic-monitoring-api"}
{"level":"info","program":"synthetic-monitoring-agent","subsystem":"adhoc","probe id":18626,"probe name":"prd-apps","time":1749842581227,"message":"registered ad-hoc probe with synthetic-monitoring-api"}
{"level":"error","program":"synthetic-monitoring-agent","subsystem":"adhoc","error":"rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type \"text/html\"","time":1749842881240,"message":"receiving ad-hoc check"}
{"level":"warn","program":"synthetic-monitoring-agent","subsystem":"adhoc","error":"rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type \"text/html\"","connection_state":"READY","time":1749842881240,"message":"handling check changes"}

This creates the impression that something is wrong, but it's not actually a problem. There is a 5 minute timeout on the connection if it's idle (on the server side). Since the adhoc "test" button is not used very often, the connection is going to be idle more often that not, which is why these show up in the logs.

There's nothing wrong in the agent, but private probe users may mistake these logs for a problem. Is there a way to separate "normal" gateway timeouts from errors that probe operators should pay attention to?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions