Add helper method to wait for log message to be observed

[donoghuc]

Release notes

[rn:skip]

What does this PR do?

There appears to be a race condition in tests whereby log messages are not observed with a single interogation of the container logs. This commit adds a helper method to wait for log messages. This should make the tests more resilient when there is a delay in log messages being captured.

Why is it important/What is the impact to the user?

NA

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

How to test this PR locally

➜  logstash git:(more-resilient-log-assertion) ✗ ci/docker_acceptance_tests.sh oss

Logs

Example failure:
https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/1776#_

2025-04-23 23:45:53 PDT | Failures:
-- | --
  | 2025-04-23 23:45:53 PDT |  
  | 2025-04-23 23:45:53 PDT | 1) A container running the oss image behaves like it applies settings correctly when setting config.string persists ${CONFIG_STRING} key in logstash.yml, resolves when running and spins up without issue
  | 2025-04-23 23:45:53 PDT | Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { \|failure, _opts\| raise failure }
  | 2025-04-23 23:45:53 PDT |  
  | 2025-04-23 23:45:53 PDT | expected true
  | 2025-04-23 23:45:53 PDT | got false
  | 2025-04-23 23:45:53 PDT | Shared Example Group: "it applies settings correctly" called from ./docker/spec/oss/container_spec.rb:8
  | 2025-04-23 23:45:53 PDT | # ./docker/shared_examples/container_options.rb:92:in `block in <main>'
  | 2025-04-23 23:45:53 PDT |  
  | 2025-04-23 23:45:53 PDT | Finished in 8 minutes 24 seconds (files took 2.74 seconds to load)

[mergify]

This pull request does not have a backport label. Could you fix it @donoghuc? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[donoghuc]

While debugging this I was printing the logs:

�[2025-04-24T17:48:01,563][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://es:9200"]}
�[2025-04-24T17:48:01,592][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://kimchy:xxxxxx@es:9200/]}}
[2025-04-24T17:48:01,641][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"es: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: es: Name or service not known>}
�[2025-04-24T17:48:01,642][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://kimchy:xxxxxx@es:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://es:9200/][Manticore::ResolutionFailure] es: Name or service not known"}

As you can see there are some rogue byte sequences in there, but they appear to only be on line endings so they are probably not breaking up the string we are looking for. I looked a bit in to upserve/docker-api#290 and it seems like maybe we could switch methods if we are still having issues.

[donoghuc]

Working locally:

Finished in 5 minutes 33 seconds (files took 0.7862 seconds to load)
44 examples, 0 failures

[donoghuc]

Kicked off exhaustive test pipeline on this branch: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/1780

[donoghuc]

https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/1780#019668f9-ff2c-4a32-86fc-42cbc660fc9c revealed a bug: c679c49

Re running with patch https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/1781

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c679c49

History

• 💚 Build #2880 succeeded 5fa7c57

[kaisecheng]

LGTM. Thanks for your time to keep CI healthy

[github-actions]

@Mergifyio backport 8.17 8.18 8.19 9.0

[mergify]

backport 8.17 8.18 8.19 9.0

✅ Backports have been created

• #17605 [8.17] (backport #17589) Add helper method to wait for log message to be observed has been created for branch 8.17
• #17606 [8.18] (backport #17589) Add helper method to wait for log message to be observed has been created for branch 8.18
• #17607 [8.19] (backport #17589) Add helper method to wait for log message to be observed has been created for branch 8.19
• #17608 [9.0] (backport #17589) Add helper method to wait for log message to be observed has been created for branch 9.0