RFC: Use ResourceAttributes and ScopeAttributes for log labels as well

[jwhitaker-gridcog]

Heya, have just been trying to get something working on #1103 - currently when Exploring logs, you can see labels from LogAttributes, but not from ResourceAttributes or ScopeAttributes. This is pretty important functionality - "I have a random log line from somewhere in my k8s cluster, which pod did it come from?"

To query, I'm merging all the Attributes maps together and prefixing their keys so I can track where they came from.

I wasn't 100% sure on the best way to model this - I flipped and flopped between having a special .labelColumns field added on QueryBuilderOptions , or in a special .magicMapForLabels bool on SelectedColumn - I went with the latter.

This seems to work OK!

SQL Preview

SELECT
  Timestamp as "timestamp", Body as "body", SeverityText as "level",
  mapConcat(
    mapApply((k, v) -> ('attr.' || k,  v), "LogAttributes"),
    mapApply((k, v) -> ('res.' || k,  v), "ResourceAttributes"),
    mapApply((k, v) -> ('span.' || k,  v), "ScopeAttributes")
  ) as "labels",
  TraceId as "traceID"
FROM "otel"."otel_logs"
WHERE
  ( timestamp >= $__fromTime AND timestamp <= $__toTime )
  AND ( ResourceAttributes['k8s.namespace.name'] = 'sys-nginx-gateway' )
  AND ( ResourceAttributes['k8s.node.name'] = 'i-04fb08d3a1fcd7eed.ap-southeast-2.compute.internal' )
ORDER BY timestamp DESC LIMIT 1000

Could I ask for a pointer if this is an ok approach to take to achieve this? I can't promise I'd be able to finish it off but I can probably get pretty far at least.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[jwhitaker-gridcog]

specific design things I was wanting some guidance on:

• Is ramming all of these inside one labels column really needed? My limited understanding of Grafana's requirements is that it is; Grafana needs the resulting dataframe to have a single column of labels. However, if Grafana allowed some official way of providing multiple labels cols that I'm unaware of, this would be both less hacky, and may appear separated in the UI as well, both of which would be desirable.

• Assuming we do need to put everything inside one labels, is prefixes the way to do it? I've somewhat arbitrarily gone with attr.${key} for LogAttributes, res.${key} for ResourceAttributes, and scope.${key} for ScopeAttributes. This is user-exposed so I wanted it to be reasonably succinct - something else I wondered about was using a valid sql identifier, e.g. ResourceAttributes['${key}'] etc. This is probably too verbose to be exposed in the labels block though. N.B. I should probably add some comment to this effect, but this namespacing system is not just for UX, it's also wired back through the Filters system so the quick +/- buttons work, so needs to be reversible.