Forwardport observability-sre internal distro support from 8.19 to main #17785
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
|
This pull request does not have a backport label. Could you fix it @yaauie? 🙏
|
yaauie
commented
Jul 9, 2025
There was a problem hiding this comment.
@kaisecheng for your review, can you focus on docker/Makefile and docker/Dockerfile.erb?
donoghuc
reviewed
Jul 9, 2025
|
I would be curious to see a CI run with this #17787 I think that would solve the rest of the test failures. I think the separate container env (which is actually using bundler env to run jruby 9.4.13.0) is exposing an issue we want to solve generally with the upgrade. |
donoghuc
reviewed
Jul 10, 2025
|
Makefile and Dockerfile.erb look good to me. |
Unfortunately these tests would not actually trigger because they are gated on unit/integration tests. IMO the path forward for this is:
|
donoghuc
approved these changes
Jul 11, 2025
There was a problem hiding this comment.
I wrote up next steps in #17785 (comment)
I think this is a complete forward port for what has been merged in 8.19 so far. There are still some moving parts. I would like to see everything listed in my comment completed then once we have artifacts publishing I would like to do a final (manual) comparison to make sure that those artifacts are running as expected.
yaauie
force-pushed
the
forwardport-fedramp-high-to-main
branch
from
76dabd9 to
08a65ae
Compare
donoghuc
approved these changes
Jul 14, 2025
.ruby-version
Outdated
| @@ -1 +1 @@ | |||
| jruby-9.4.9.0 | |||
| jruby-9.4.13.0 | |||
There was a problem hiding this comment.
Suggested change
| jruby-9.4.13.0 | |
| jruby-9.4.9.0 |
Can we handle this separately in #17798? That way i can do an easy backport.
There was a problem hiding this comment.
🤦🏼 yep, I cut that out, waiting once more for green.
This is the CLEAN subset of a cherry-pick of the merge-commit from the observabilitySRE feature branch into 8.x in PR elastic#17541 (0b1d299), OMITTING changes to `docker/*` and `rakelib/artifacts.rake` that would conflict due to substantial refactorings on `main`.
This is a forward-port of _functionality_ from the observabilitySRE feature branch into 8.x in PR elastic#17541 (0b1d299), wholly re-implementing the changes in `docker/*` and `rakelib/artifacts.rake` from the 8.x-style docker structure to the refactored structure present on `main`.
When the fedramp high feature branch was merged into 8.x the PR pipeline accidentally duplicated the top level `steps` key. This was a mistake and is causing issues generating exhaustive test pipeline definition. This commit fixes the bug by ensuring there is a single `steps` key that defines all the steps in the pipeline.
The `artifactDockerObservabilitySRE` gradle task *always* produces a tag with a `SNAPSHOT` postfix. In the staging pipeline we use the shared `qualified-version` script for determining the LS version. That script correctly handles conditionally adding a `SNAPSHOT` postfix which is important for the tagging scheme for pushing to our container registry. Given the intermediate tag produced by the gradle task is never pushed anywhere we can update the build script to ensure the "local" artifact is always referenced with the `SNAPSHOT` postfix.
…lastic#17627) * Use dedicated elasticsearch image for observabilitySRE smoke testing The ES team has started publishing a purpose built image for the fedramp high project. Update our smoke test stack to use this container. * Override default entrypoint into elasticsearch container The new image does not provide the stub `/app/elasticsearch.sh` file https://github.com/elastic/elasticsearch/blob/1a1763c591c4c32bf66f0df3bce2040e8f19a1a2/distribution/docker/README.md?plain=1#L16-L19 previously available. This commit overrides the entrypoint to avoid needing that file. See: https://github.com/elastic/elasticsearch/blob/1a1763c591c4c32bf66f0df3bce2040e8f19a1a2/distribution/docker/src/docker/Dockerfile.ess#L38C5-L40C37 * Remove entrypoint workaround due to fix landing upstream
* Comment to clarify why FIPS flag is not needed for smoke tests * Use full versions of docker commands for readability * Simplify grock pattern match The grok pattern is unanchored-by-default, we don't need the leading and trailing wildcards.
elastic#17623) * Add a step to exhaustive tests for observabilitySRE accetpance testing This commit shows the proposed pattern for adding acceptance testing for the observability SRE image. This will run when exhaustive tests run. A new gradle task will hook in to rspec similar to how it is done for the smoke tests. The main difference is that instead of building a container, the latest is pulled from the container registry and run on a fips configured host VM. * WIP: Idea for how to handle multipe container configs for acceptance tests This commit shows the rough structure for how I am planning on handling docker compose networks for acceptance tests. The main idea is to use interpolation in the docker compose file to point to different configuration files for filebeat/logstash/elasticsearch. This is mainly due to the nature of these tests showing behavior when the system is and is not configured properly for FIPS. The breakdown in responsibility is: 1. Gradle handles cert generation (similar to smoke test, this avoids checking in PKI) 2. Rspec handles stopping/starting docker compose and managing environment vars for intperolation in docker compose manifests (different from smoke tests where a single static docker compose is started in gradle) 3. Rspec handles deciding when containers are ready and querying state about data flowing through the system 4. Gradle cleans up certs THis is just a rough sketch, there are still bugs to be worked out but before i get too far in to it I want to get the idea out there. * Add tests describing behavior of LS -> ES with non-fips config This commit adds a test to show that data will not flow from LS to ES when weak non fips config is used. * Use latest ES image This will be handled separately in a separate PR, but taking this commit for now on this branch. * Remove custom entrypoint from new container The latest ES images do not require this workaround. * Take up code review suggestions 1. Remove rogue character from test file causing interpreter failure 2. Split out helpers for docker compose orchestration 3. Only send a single message instead of infinite through to ES * Add full prefix name for new image * Test filebeat -> LS -> ES using fips config As described in elastic/ingest-dev#5471 this commit adds a test for filebeat sending data through logstash to elasticsearch using fips config. * Test LS wont accept input from non fips configured filebeat This test ensures logstash will not accept data from filebeat when using weak tls configuration. See elastic/ingest-dev#5472 * Fix a funny typo. Crytpo is actually kind of a funny. * Ensure we are using the purpose build ES image in testing Similar to elastic#17627 * Ensure JAVA_HOME is set etc Use the same buildkite agent script for setting up a vm based runner as other pipes
yaauie
force-pushed
the
forwardport-fedramp-high-to-main
branch
from
08a65ae to
a808309
Compare
|
💚 Build Succeeded
History
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release notes
[rn:skip]
What does this PR do?
Forward-ports support for the observabilitySRE internal distribution from
8.19tomain, enabling us to create 9.x artifacts.This is a sequence of cherry-picks from the 8.19 branch including all commits that deal with the creation or validation of the internal observabilitySRE docker artifacts.
0b1d299: DIRTY forward-port of Merge feature branch for observability SRE image creation into 8.x #17541.docker/Makefile,docker/Dockerfile.erb, andartifacts.rakedue to a substantial refactoring of those files onmain. I reset these files to their state onmainand re-implemented their functionality entirely in the context ofmain(including propagating a small change tobuild.gradleto wire up to the new structure)x-pack/lib/x-pack/logstash_registry.rbcaused by the removal of certain modules in 9.0; these were resolved by keeping the contents onmain.2b44f6b- clean forward-port of Fix pull request pipeline definition for buildkite #175520df0e99- clean forward-port of Ensure observabilitySRE image is pushed on DRA staging #17569e245e6b- clean forward-port of Use dedicated elasticsearch image for observabilitySRE smoke testing #1762774bba3f- clean forward-port of Restore code review changes #1753910e41a8- clean forward-port of Add a step to exhaustive tests for observabilitySRE accetpance testing #17623Why is it important/What is the impact to the user?
Enables our internal customer to deploy docker artifacts based on Logstash 9.x
Checklist
[ ] My code follows the style guidelines of this project[ ] I have commented my code, particularly in hard-to-understand areas[ ] I have made corresponding changes to the documentation[ ] I have made corresponding change to the default configuration files (and/or docker env variables)[ ] I have added tests that prove my fix is effective or that my feature worksAuthor's Checklist
How to test this PR locally