[Discover] Update highlight fields section with defined sets

[tonyghiani]

📓 Summary

We want to provide the most insightful highlights on the Log detail flyout.

This work should research and define the sections and fields which we consider important and that we want to show to the users.

🎨 Design

🖌️ Figma file

✔️ Acceptance Criteria

Each section should show 3 entries (ordered by priority) and the +more button if more are available, but each row should only render the 3 most important fields.

Top-level fields

Basically, the same as we have now, but having the dataset on top as another badge would give immediate value to knowing the source of the log entry.

• @timestamp
• log.level
• message or result of virtual column content
• data_stream.dataset

Highlights

Service & Infrastructure

• service.name
• host.name
• trace.id
• • orchestrator.cluster.name, orchestrator.resource.id

Cloud

• cloud.provider
• cloud.region
• cloud.project.id
• • cloud.availability_zone, cloud.instance.id

Other

• log.file.path
• data_stream.namespace
• agent.name
• labels
• tags

I also thought, in case the fields are present, we could render conditionally two other sections to start for Kubernetes and Web details:

Kubernetes

• kubernetes.pod.name
• kubernetes.container.name
• kubernetes.node.hostname
• • kubernetes.pod.ip, kubernetes.pod.uid, kubernetes.namespace

Web

• url.path + url.query
• http.request.id
• http.response.status_code

This should guarantee 3 core sections (Service & Infrastructure, Cloud, Other) and additional ad-hoc fields based on the log purpose and source.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[achyutjhunjhunwala]

This looks really good @tonyghiani Thank you for adding this
One point around the Kubernetes section -

If i am not wrong, kubernetes pods names and container name will only be different in case of multi container pods which i feel is rare. I would suggest to move the namespace to be on the 1st 3 priority list and move container name to non priority.

Kubernetes

• kubernetes.pod.name
• kubernetes.namespace
• kubernetes.node.hostname
• • kubernetes.container.name, kubernetes.pod.ip, kubernetes.pod.uid,

Web field would be a really good addition. But composite fields like url.path + query might add complexities around filtering.

What about

Web

• url.path
• url.query
• http.response.status_code
• • http.request.id

[tonyghiani]

Kubernetes pods names and container name will only be different in case of multi container pods which i feel is rare. I would suggest to move the namespace to be on the 1st 3 priority list and move container name to non priority.

Sounds correct, good catch!

But composite fields like url.path + query might add complexities around filtering.

Is a good point, when I wrote this I though of treating this as a we would do with a virtual column field, allowing to filter for both specific fields, but also your suggestion looks good!

[ruflin]

Let me suggest an alternative to the boxes we have at the moment. There is hierarchy to all the info we show at the moment.

• service runs on container / host runs on a cluster runs on a cloud
• log file is collected from a container / host that might run on a cluster which could run on a cloud

The goal for the user is to very quickly understand "what" created the log message and the content of it. This could our first "box".

Then about each item in this list, there can be additional info: more k8s fields, cloud details etc.

For the log message itself, one thing we have missed so far is tags and labels, how do these fit in?

Last, there is context aware details where we see a trace id or we find some web specific fields. The way I think of this is an "extension" box that others like APM/Host or Integrations can hook into to show more specific data. Some we can hardcore.

[ruflin]

@tommyers-elastic Would be great to get your input on the above in the context of the asset work you are currently doing.

[tonyghiani]

The goal for the user is to very quickly understand "what" created the log message and the content of it. This could our first "box".

I see your point @ruflin, this might imply we should rethink the structure of the top-level details we show and add more info about "where" the message comes from. It could make the design slightly different and we still need to think how to keep it compact.

For the log message itself, one thing we have missed so far is tags and labels, how do these fit in?

You are right, I added them to the issue and we'll figure out how to fit them into the main details.

I'll add some examples based on real documents of how these details could fit and their organization in the flyout.

[tonyghiani]

After brainstorming more about this with @ruflin, answering the user question about "where" a log comes from and making the detection of an issue smoother, it would make sense to display on the flyout something that, using some of the fields mentioned for the highlights sections, answer a mental path the user has about the origin of this log.

It could look something like:

• What service is this log from? service.name
• Where is the service spawned? host.name, orchestrator.cluster.name
• Where is it located? cloud.provider, cloud.region, cloud.instance.id
• What shipped the log? agent.name, agent.type

This would better answer the question "Where can I find where this log comes from", and we should show that after the "What is this log" part (message, log.level, timestamp...).

All the remaining fields will stay inside the highlight boxes as they are now for users who want additional info about specific fields (k8s, web, others, etc..).

@isaclfreire we were wondering if there is a way to give importance to this section compared to the others and we can tell the user the question we are trying to solve for them. Would be great if we work on this and find a way to design this new section. Let me know if you want to go further into details about this idea and if you have any question!

[achyutjhunjhunwala]

I feel cloud.provider does not add a lot of value at all. With my experience on being the other side as a user of the solution, as a customer, we always knew where the infrastructure is running. I don't think a lot of customers use a mix of AWS/GCP/Azure. Rather than it is very safe to assume that most of them will have 1 Cloud provider. Does it still makes sense to give this a dedicated FIELD space on the Flyout. It could be simply an icon next to cloud.region ?

[achyutjhunjhunwala]

Also, i really liked the idea of displaying certain information the way we do it in APM

Service Name and then some possible icons.

[tonyghiani]

It could be simply an icon next to cloud.region?

That sounds reasonable to me, let's hear @isaclfreire opinion about it.

[tonyghiani]

Update: @isaclfreire and I synced on how we can give more importance to the fields defined in this comment, we'll look into a possible design for this section in the upcoming weeks.

[isaclfreire]

After some discussion with @tonyghiani today I'll be working with @mdbirnstiehl to rethink that section for the first version and in the meantime also setup a card sorting exercise (issue: https://github.com/elastic/observability-dev/issues/3028) with users to gather external feedback.