tags with values that contain complicated values are not escaped correctly

[dbern-stripe]

given this configuration:

      {TelemetryMetricsStatsd,
       [
         formatter: :datadog,
         metrics: metrics(),
         # 512 bytes is the default, but including the SQL query is larger than that.
         mtu: 2048
       ]}

# ...
      summary("my_app.repo.query.query_time", tags: [:query], unit: {:native, :millisecond}),

The repo is an Ecto.Repo, using Ecto version 3.7.1 and EctoSQL 3.7.0. Here is where it emits the telemetry

emits this to statsd:

gostatsd1 | time="2021-10-12T19:07:24Z" level=info msg="stats.timers.my_app.repo.query.query_time. i0.\"column_one\". i0.\"data\". i0.\"inserted_at\". i0.\"updated_at\" FROM \"my_table\" AS i0. i0.\"user_id\".environment.development.host.st-dbern1.query.SELECT i0.\"id\".s.172.21.0.1.std 0.000000 1634065644"

From here, it's interpreted incorrectly at a sink like NewRelic, where it looks like the tag :query was read as SELECT i0."id"

and the rest of the SQL is interpreted as "tags" incorrectly:

It appears that the values need to be escaped in a way for the entire SQL value to be taken all together.

[arkgil]

@dbern-stripe thanks for creating the issue! Sadly, I don't think the StatsD protocol specifies how to perform escaping of special characters (i.e. ., |, :, and @). Whatever approach we take here, needs to be understandable by agents that support the protocol.

Do you have an idea on how we could approach this?

[dbern-stripe]

@arkgil unfortunately the spec isn't clear: https://docs.datadoghq.com/developers/dogstatsd/datagram_shell?tab=metrics

Judging from datadog's parser: https://github.com/DataDog/datadog-agent/blob/0454961e636342c9fbab9e561e6346ae804679a9/pkg/dogstatsd/parse.go#L67-L86 it splits on commas and that's it.

https://docs.datadoghq.com/getting_started/tagging/

The best answer I can find on this is that this kind of detail belongs in a trace instead of tagged metric, so perhaps OpenTelemetry.Tracer would be better suited for this information, and this issue is resolved with some documentation that addresses this scenario. (by pointing to opentelemetry instead)

[Saba307]

given this configuration:

      {TelemetryMetricsStatsd,
       [
         formatter: :datadog,
         metrics: metrics(),
         # 512 bytes is the default, but including the SQL query is larger than that.
         mtu: 2048
       ]}

# ...
      summary("my_app.repo.query.query_time", tags: [:query], unit: {:native, :millisecond}),

The repo is an Ecto.Repo, using Ecto version 3.7.1 and EctoSQL 3.7.0. Here is where it emits the telemetry

emits this to statsd:

gostatsd1 | time="2021-10-12T19:07:24Z" level=info msg="stats.timers.my_app.repo.query.query_time. i0.\"column_one\". i0.\"data\". i0.\"inserted_at\". i0.\"updated_at\" FROM \"my_table\" AS i0. i0.\"user_id\".environment.development.host.st-dbern1.query.SELECT i0.\"id\".s.172.21.0.1.std 0.000000 [1634065644](tel:1634065644)"

From here, it's interpreted incorrectly at a sink like NewRelic, where it looks like the tag :query was read as SELECT i0."id"

and the rest of the SQL is interpreted as "tags" incorrectly:

It appears that the values need to be escaped in a way for the entire SQL value to be taken all together.

Legal all websites networks settings services servers hosting search console mail boxes online platforms marketing brends blog companies members error comments reply button notices ads api faq languages podcast audio video books other cookies choose product platforms websites networks Legal Public Original install.

[Saba307]

Legal all websites networks settings services servers hosting search console mail boxes online platforms languages about cookies choose product platforms websites networks Legal Public Original install.