RFC: Use ResourceAttributes and ScopeAttributes for log labels as well

[jwhitaker-gridcog]

Heya, have just been trying to get something working on #1103 - currently when Exploring logs, you can see labels from LogAttributes, but not from ResourceAttributes or ScopeAttributes. This is pretty important functionality - "I have a random log line from somewhere in my k8s cluster, which pod did it come from?"

To query, I'm merging all the Attributes maps together and prefixing their keys so I can track where they came from.

I wasn't 100% sure on the best way to model this - I flipped and flopped between having a special .labelColumns field added on QueryBuilderOptions , or in a special .magicMapForLabels bool on SelectedColumn - I went with the latter.

This seems to work OK!

SQL Preview

SELECT
  Timestamp as "timestamp", Body as "body", SeverityText as "level",
  mapConcat(
    mapApply((k, v) -> ('attr.' || k,  v), "LogAttributes"),
    mapApply((k, v) -> ('res.' || k,  v), "ResourceAttributes"),
    mapApply((k, v) -> ('span.' || k,  v), "ScopeAttributes")
  ) as "labels",
  TraceId as "traceID"
FROM "otel"."otel_logs"
WHERE
  ( timestamp >= $__fromTime AND timestamp <= $__toTime )
  AND ( ResourceAttributes['k8s.namespace.name'] = 'sys-nginx-gateway' )
  AND ( ResourceAttributes['k8s.node.name'] = 'i-04fb08d3a1fcd7eed.ap-southeast-2.compute.internal' )
ORDER BY timestamp DESC LIMIT 1000

Could I ask for a pointer if this is an ok approach to take to achieve this? I can't promise I'd be able to finish it off but I can probably get pretty far at least.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[jwhitaker-gridcog]

specific design things I was wanting some guidance on:

• Is ramming all of these inside one labels column really needed? My limited understanding of Grafana's requirements is that it is; Grafana needs the resulting dataframe to have a single column of labels. However, if Grafana allowed some official way of providing multiple labels cols that I'm unaware of, this would be both less hacky, and may appear separated in the UI as well, both of which would be desirable.

• Assuming we do need to put everything inside one labels, is prefixes the way to do it? I've somewhat arbitrarily gone with attr.${key} for LogAttributes, res.${key} for ResourceAttributes, and scope.${key} for ScopeAttributes. This is user-exposed so I wanted it to be reasonably succinct - something else I wondered about was using a valid sql identifier, e.g. ResourceAttributes['${key}'] etc. This is probably too verbose to be exposed in the labels block though. N.B. I should probably add some comment to this effect, but this namespacing system is not just for UX, it's also wired back through the Filters system so the quick +/- buttons work, so needs to be reversible.

[adamyeats]

@SpencerTorres Would you be able to comment on any of the points raised above?

[lasermoth]

I just came across this, as I was also going down the rabbit hole of wanting ResourceAttributes as log labels.

My current work around was to disable the "use Otel" flag and run mapConcat(LogAttributes, ResourceAttributes) on the labels field which is rather ugly.

I believe the labels field is special. I can at least see is used in the loki datasource this way. It does also appear exclusively referenced.

Looking forward to this contribution.

[lasermoth]

Is this still being worked on, it should we look to open a new PR?

[lasermoth]

@SpencerTorres - Do you have capacity to help with the questions here

[SpencerTorres]

Hello! Thanks for submitting this. This is a known issue, but there isn't really a clear way to solve it. I'll elaborate by answering your questions:

Is ramming all of these inside one labels column really needed? My limited understanding of Grafana's requirements is that it is; Grafana needs the resulting dataframe to have a single column of labels. However, if Grafana allowed some official way of providing multiple labels cols that I'm unaware of, this would be both less hacky, and may appear separated in the UI as well, both of which would be desirable.

Yes, Grafana looks for a labels field on the data frame. There's no official way to have multiple label fields, but we did discuss the possibility of merging them on the backend. Something like labels1 labels2 labels3 would be merged by either the backend or frontend processing logic. I was hesitant to add this since it's very opinionated and specific to our datasource, I didn't want to break any workflows that coincidentally use the same field names as these.

Assuming we do need to put everything inside one labels, is prefixes the way to do it? I've somewhat arbitrarily gone with attr.${key} for LogAttributes, res.${key} for ResourceAttributes, and scope.${key} for ScopeAttributes. This is user-exposed so I wanted it to be reasonably succinct - something else I wondered about was using a valid sql identifier, e.g. ResourceAttributes['${key}'] etc. This is probably too verbose to be exposed in the labels block though. N.B. I should probably add some comment to this effect, but this namespacing system is not just for UX, it's also wired back through the Filters system so the quick +/- buttons work, so needs to be reversible.

This is where the label merging idea gets a bit murky. Yes I would go with prefixing, but once we start aliasing these as labels1 labels2, etc. it gets confusing on how we enable filtering on these values.

In the logs panel there are hooks whenever you "filter in" and "filter out" a value. All we know is the actual field name. So if your field name is pod.name, we don't really know whether that actually came from ResourceAttributes or labels1 or some other column, it's just whatever's in labels. If you were to click filter on this I believe it would add a filter for something like labels['pod.name'] = 'some value'. All we know is that the value came from the labels field.

We could add prefixes for ResourceAttributes so that it appears as ResourceAttributes.pod.name and then we can check for whether this prefix matches a selected column and then auto convert it, making ResourceAttributes.pod.name turn into ResourceAttributes['pod.name'], but this requires that the labels were merged, and that the ResourceAttributes was selected. There are a lot of edge cases here. This same problem applies to our Ad-Hoc filters feature. I believe there's already some logic in there that does something like this, but it clearly doesn't cover the full use case.

If you have any more ideas on how to handle this, I'm glad to explore them. I'll bring this up again internally to see if we can come up with a clean solution for this. We might just have to make the Labels dropdown accept multiple map/json columns, or maybe have them pre-configured per datasource.

[jwhitaker-gridcog]

Thanks for looking at this @SpencerTorres . Yeah, this PR does nearly what you're suggesting with ResourceAttributes.pod.name, but I introduced the convention res.pod.name as this is user-exposed and brevity might be appreciated. It does seem to work, however it probably doesn't cope with some of the edge cases you're talking about.

I wonder if an approach to make it less flaky on if merged labels appear in the query or not, might be to add an alias column to the otel logs table itself in the schema? That would be more of a guarantee that merged labels are around, and additionally could help nail down the specific prefixes you want to use.

[SpencerTorres]

This should be solved by #1369, available in plugin v4.11.0

[jwhitaker-gridcog]

that looks awesome Spencer, thankyou!!