[elastic_agent] Support new pipeline fields from beats logs

[jsoriano]

Proposed commit message

Support fields added and fixed for pipeline metrics in Beats 8.15.0.

• Definitions added for new fields.
• Ingest pipeline to adapt renamed field.
• Revert kibana version constraint to ^8.11.2 || ^9.0.0.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Beats change in [libbeat] Add a metrics observer to the queue beats#39774

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[rdner]

Suggested change

	field: monitoring.metrics.libbeat.pipeline.queue.filled.pct.events
	target_field: monitoring.metrics.libbeat.pipeline.queue.filled.events
	field: monitoring.metrics.libbeat.pipeline.queue.filled.pct.events
	target_field: monitoring.metrics.libbeat.pipeline.queue.filled.pct

[rdner]

Why are there so many changes in field files? Were we missing so many fields before?

[jsoriano]

According to the PR in Beats all these fields have been added: elastic/beats#39774

Haven't been able to confirm.

[faec]

This is a meta-comment to clarify what the situation was as of ~3 days ago and how it got that way.

Timeline:

• April 30, 2024: pipeline.queue.filled.pct.events is first defined in Add queue percentage to libbeat metrics beats#39205, released in 8.15.0. The name is a mistake (my fault).
• May 2, 2024: The elastic_agent package is updated to list queue.filled.pct.events as one of its fields. Add panel to dashboard, add percent field #9765
• June 11, 2024: Existing queue metrics are fixed and several new ones are added. pipeline.queue.filled.pct.events is renamed to pipeline.queue.filled.pct. [libbeat] Add a metrics observer to the queue beats#39774, released in 8.15.0.
• August 9, 2024: A bug that prevented the queue metrics from being registered properly is fixed. [libbeat] Fix and test initialization of queue monitors beats#40480, released in 8.15.1.
• January 29, 2025: The new queue metrics are added to Metricbeat's schema for the beat/stats metricset, so they will be ingested by Metricbeat monitoring. Include full queue metrics in the monitoring index beats#42439, released in 8.17.2 and 8.18.0.
• February 4, 2025: The tracking issue to properly display the new metrics in the Agent dashboards is closed, seemingly due to a misunderstanding about the scope of what had been completed so far Queue filled metrics are not indexed nor displayed in Elastic Agent dashboard beats#42093 (I think @strawgate was working on some of that? But whether because of this issue closure or something else, it looks like that followup was incomplete.)

Commentary:

• The queue filled metric never existed under any name until 8.15.0, which included both the initial definition (Add queue percentage to libbeat metrics beats#39205) and the naming fix ([libbeat] Add a metrics observer to the queue beats#39774); 8.14.3 and earlier included neither.
• Neither the filled metric nor any of the other new ones were ingested by Metricbeat monitoring until v8.17.2 and later.
• Until today, I believed no released version ever reported metrics under the mistaken name queue.filled.pct.events, because the name was corrected before it hit a release. This was almost true, however:
  • There was an "inert" occurrence of the old name, with no references, that was missed in the initial fix, and stayed there until the Metricbeat schema update (https://github.com/elastic/beats/pull/42439/files#diff-558176dd2327daca24e089ce7ec773653e787ef7438af3d3df257e655b383e65L133).
  • The schema update PR cleaned up a few "redundant" metrics variable definitions that had no code references, since the same variables were created and used elsewhere, not noticing that the "redundant" filled metric had used the old / incorrect name. [But of course, it was unreferenced and absent from the Metricbeat schema, so there was no way for it to actually show up anywhere except the logs, right?]
  • The real queue metrics also cleared their registry on initialization, so the old name only existed for a short window during pipeline initialization when the queue had not yet been created.
  • Except, the bug fixed in [libbeat] Fix and test initialization of queue monitors beats#40480 prevented the old name from being removed in 8.15.0.
  • If 8.15.0 was run with self-monitoring (self-monitoring doesn't enforce the schema restriction of Metricbeat monitoring, so it includes all variables), queue.filled.pct.events would be ingested, always with a value of zero.
  • Also, for completeness: Between versions 8.15.1 and 8.17.1 (inclusive), if self-monitoring is enabled, and a metrics scan interval triggers in the brief window before the queue is created (I'm not sure this is actually possible, but if so it's probably extremely rare), a (zero) queue.filled.pct.events field would be ingested.
• The upshot is: running with self-monitoring on precisely version 8.15.0, then upgrading to a later version with self-monitoring, or to 8.17.2 or later with any monitoring, can cause a mapping conflict where the earlier queue.filled.pct.events field interferes with the queue.filled.pct field used by all other versions.
• One important consequence of all this: in trying to untangle this mess, it is always safe to delete any field with the name queue.filled.pct.events. To the extent it ever appeared at all, it was always zero.
• This still leaves the field definitions in the Agent package, which still use the old field name, and would seem to conflict with the real monitoring values since 8.17.2, although I'm not sure exactly how a nested example like this would be handled -- if that did cause a mapping conflict, I would expect it to be an extremely common error, which doesn't seem to have been the case? So that should be addressed, but our recent issues might be an example of the former corner case instead.

[jsoriano]

Thanks for the explanation!

• The upshot is: running with self-monitoring on precisely version 8.15.0, then upgrading to a later version with self-monitoring, or to 8.17.2 or later with any monitoring, can cause a mapping conflict where the earlier queue.filled.pct.events field interferes with the queue.filled.pct field used by all other versions.

Then this PR and #15244 should fix the issue in the integration, right?

• Update queue.filled.pct.events to queue.filled.pct #15244 fixes the mapping.
• [elastic_agent] Support new pipeline fields from beats logs #15280 (this one) ensures that only the correct mapping is used, and adds additional mappings for the other fields added in [libbeat] Add a metrics observer to the queue beats#39774.

Between versions 8.15.1 and 8.17.1 (inclusive), if self-monitoring is enabled, and a metrics scan interval triggers in the brief window before the queue is created (I'm not sure this is actually possible, but if so it's probably extremely rare), a (zero) queue.filled.pct.events field would be ingested.

The pipelines I am adding in this PR would be intended to handle these cases. If this is so rare, is it worth it to add them?

• The queue filled metric never existed under any name until 8.15.0

This explains why I didn't manage to reproduce the issue with the oldest version supported by the package (8.11.2).

[cmacknz]

If this is so rare, is it worth it to add them?

IMO yes we already have a support case about this, let's just put in the fixes and then not have to think about or explain this again.

[faec]

If this is so rare, is it worth it to add them?

In addition to Craig's comment: I'd expect that the fix for that case is the same as the (presumably more common) one of running 8.15.0 with self-monitoring -- the correct behavior with any version of Beats is always to delete the queue.filled.pct.events field if it exists, since in any case where it could have existed it didn't contain real data.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[jsoriano]

If this is so rare, is it worth it to add them?

In addition to Craig's comment: I'd expect that the fix for that case is the same as the (presumably more common) one of running 8.15.0 with self-monitoring -- the correct behavior with any version of Beats is always to delete the queue.filled.pct.events field if it exists, since in any case where it could have existed it didn't contain real data.

Ok, so instead of renaming it I will remove it.

[jsoriano]

Btw, another follow up issue created, to have system tests on this package: elastic/elastic-package#2903

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 326c7b1

History

• 💚 Build #31196 succeeded a980d8d
• 💚 Build #31134 succeeded bc4bdcf

cc @jsoriano

[elastic-sonarqube]

Quality Gate failed

Failed conditions
11.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!