Skip to content

[azure logs] Add support for body.structured logs events from azurelogs encoding extension (OTel) #15891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
+36 −1
Draft

[azure logs] Add support for body.structured logs events from azurelogs encoding extension (OTel) #15891

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,17 @@ processors:
field: azure
target_field: azure-eventhub
ignore_missing: true

# --------------------------------------
# `message` field cleanup steps.
# --------------------------------------
- script:
source: ctx.message = ctx.message.replace(params.empty_field_name, '')
params:
empty_field_name: '"":"",'
ignore_failure: true
if: 'ctx.message != null'
description: 'Replaces empty fields with an empty string in the `message` field.'
- rename:
field: message
target_field: event.original
Expand All @@ -24,9 +30,23 @@ processors:
ignore_missing: true
if: 'ctx.event?.original != null'
description: 'The `message` field is no longer required if the document has an `event.original` field.'

# Encoding extension
- rename:
field: body.structured
target_field: azure.activitylogs
ignore_missing: true
description: 'Renames the `body.structured` field to `azure.activitylogs` to store the structured data.'
- remove:
field: body
ignore_missing: true
description: 'Cleans up the `body` field after the structured data has been moved to the `azure.activitylogs` field.'
- json:
field: event.original
target_field: azure.activitylogs
if: 'ctx.azure?.activitylogs == null'
description: 'Parses the `event.original` field as JSON and stores it in the `azure.activitylogs` field.'

- date:
field: azure.activitylogs.time
target_field: '@timestamp'
Expand Down
17 changes: 16 additions & 1 deletion packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,23 @@ processors:
target_field: azure-eventhub
if: 'ctx.azure?.eventhub != null'
ignore_missing: true

- set:
field: event.kind
value: event


#
# Incoming pipeline input can be one of the following:
#
# - `message`
# - `body.structured`
#

- set:
field: tmp_json
copy_from: body.structured
if: 'ctx.body?.structured != null'

#
# Set `event.dataset` value based on the event category.
# ------------------------------------------------------
Expand All @@ -28,6 +41,7 @@ processors:
field: message
target_field: tmp_json
description: 'Parses the message field as JSON and stores it in a temporary field to identify the event dataset.'
if: 'ctx.tmp_json == null'

# Unfortunately, some Azure services produce logs with field names capitalized,
# violating the spec [1].
Expand All @@ -44,6 +58,7 @@ processors:
if: 'ctx.tmp_json?.category == null'
ignore_missing: true
description: 'Rename the invalid `Category` field to `category` to apply the correct routing rules.'

# Unfortunately, some Azure services generate logs with
# `CategoryValue` field instead of `Category` field.
#
Expand Down