Skip to content

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Aug 18, 2025

Release notes

[rn:skip]

What does this PR do?

Harmonize the testing of observability SRE container artifacts:

  1. Make sure container orchestration is happening at rspec level (previously both gradle and rspec)
  2. Share orchestration and ES interaction code in rspec
  3. Make dynamically interpolating values in docker compose manifests standardize and ensure the stack is the same as logstash version
  4. Simplify and document running the tests for local dev work.

Why is it important/What is the impact to the user?

NA

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files (and/or docker env variables)
  • I have added tests that prove my fix is effective or that my feature works

Related issues


This is an automatic backport of pull request #17781 done by [Mergify](https://mergify.com).

* Move ES config from ENV to file

* Use fips filebeat impl for smoke tests

* Standardize container orchestration

Make rspec responsible for container orchestration. DRY shared functionality.

* Ensure we test against the right versions

Dynamically look up the version of logstash and test against the corresponding ES image.
Also, extract this to make local dev easier.

(cherry picked from commit 1195743)
@mergify mergify bot added the backport label Aug 18, 2025
@mergify mergify bot mentioned this pull request Aug 18, 2025
5 tasks
@github-actions
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@donoghuc
Copy link
Member

This is the failure

fips_test_elasticsearch  | {"@timestamp":"2025-08-18T21:35:54.633Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"fae4ed9cf1a0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.ElasticsearchSecurityException","error.message":"failed to load SSL configuration [xpack.inference.elastic.http.ssl] - failed to initialize a TrustManager for the system keystore","error.stack_trace":"org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.inference.elastic.http.ssl] - failed to initialize a TrustManager for the system keystore\n\tat [email protected]/org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:622)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1430)\n\tat java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1708)\n\tat [email protected]/org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:618)\n\tat [email protected]/org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:160)\n\tat [email protected]/org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:500)\n\tat [email protected]/org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:329)\n\tat [email protected]/org.elasticsearch.node.NodeConstruction.lambda$construct$17(NodeConstruction.java:999)\n\tat [email protected]/org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:185)\n\tat java.base/java.util.stream.ReferencePipeline$7$1FlatMap.accept(ReferencePipeline.java:289)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)\n\tat java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:636)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:656)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:662)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:667)\n\tat [email protected]/org.elasticsearch.node.NodeConstruction.construct(NodeConstruction.java:1021)\n\tat [email protected]/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:310)\n\tat [email protected]/org.elasticsearch.node.Node.<init>(Node.java:185)\n\tat [email protected]/org.elasticsearch.bootstrap.Elasticsearch$1.<init>(Elasticsearch.java:405)\n\tat [email protected]/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:405)\n\tat [email protected]/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:100)\nCaused by: org.elasticsearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore\n\tat [email protected]/org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:68)\n\tat [email protected]/org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:479)\n\tat java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1229)\n\tat [email protected]/org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:620)\n\t... 24 more\nCaused by: java.security.KeyStoreException: problem accessing trust store\n\tat java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:72)\n\tat java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:273)\n\tat [email protected]/org.elasticsearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:172)\n\tat [email protected]/org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:66)\n\t... 27 more\nCaused by: java.security.KeyStoreException: BCFKS not found\n\tat java.base/java.security.KeyStore.getInstance(KeyStore.java:873)\n\tat java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:367)\n\tat java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:327)\n\tat java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:56)\n\tat java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:48)\n\t... 30 more\nCaused by: java.security.NoSuchAlgorithmException: BCFKS KeyStore not available\n\tat java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:147)\n\tat java.base/java.security.Security.getImpl(Security.java:759)\n\tat java.base/java.security.KeyStore.getInstance(KeyStore.java:870)\n\t... 34 more\n"}

There is a bug in the 9.2.0 series of ES fips preventing us from using it in CI
Pin to 8.19.3 for now.
@donoghuc donoghuc force-pushed the mergify/bp/main/pr-17781 branch from 2cca606 to 49297fa Compare August 19, 2025 20:16
Explicitly require fips provider and use fipsmode
This commit updates filebeat to use the 9 series format and increases the
log size with new minimum 1024 char size to start streaming. This also updates
the default stack version to 9.2 (currently that is version that tracks main)
and adds some more logging to help see what is going on during docker-compose and
wait.
@donoghuc
Copy link
Member

donoghuc commented Aug 20, 2025

@elastic-sonarqube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @donoghuc

donoghuc added a commit to donoghuc/logstash that referenced this pull request Aug 20, 2025
When working on porting tests to elastic#18001
the main LS branch we started testing against the 9.2 series of the ES stack.
There were some required changes there (listed below) that are 8.19 compatable.
This commit ports those changes to the 8.19 series to keep them as close as
possible.

1. Elasticsearch container needs explicit fips config to boot
2. Filebeat needs > 64k file size to start filestream processing
3. Generally the logging in the shared helpers was updated to make reasoning
about the state of failures easier in log output.
Copy link
Member

@donoghuc donoghuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are green! I opened #18014 to backport all the changes necessary to use the 9.2 stack as they are all 8.19 compatable. This makes 8.19 more future proof and keeps things consistent.

Copy link
Member

@yaauie yaauie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 👍🏼

@donoghuc donoghuc merged commit 75abd8a into main Aug 21, 2025
13 checks passed
@donoghuc donoghuc deleted the mergify/bp/main/pr-17781 branch August 21, 2025 18:55
donoghuc added a commit that referenced this pull request Aug 21, 2025
When working on porting tests to #18001
the main LS branch we started testing against the 9.2 series of the ES stack.
There were some required changes there (listed below) that are 8.19 compatable.
This commit ports those changes to the 8.19 series to keep them as close as
possible.

1. Elasticsearch container needs explicit fips config to boot
2. Filebeat needs > 64k file size to start filestream processing
3. Generally the logging in the shared helpers was updated to make reasoning
about the state of failures easier in log output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants