[Helm] Add Logstash Output to Elastic Agent Standalone #10644
Conversation
|
This pull request does not have a backport label. Could you fix it @btrieger? 🙏
|
cmacknz
added
the
Team:Elastic-Agent-Control-Plane
|
Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane) |
|
Hey @btrieger |
|
@pierrehilbert added the changelog fragment |
…l to avoid removal of quotes around strings
|
I added the remaining ssl flags and they should work with any of the outputs. I didn't provide an example of using them as I don't have a functioning logstash cluster that needs them but I did validate that they are being set as expected with a template command. Was thinking maybe I could document all the possible input variables outside the values schema json for every day users. Would it be best to put this in the examples readme for logstash? On a side note the rest of the outputs should remove the fromYaml and toYaml around the the call to the SSLConfig rendering as it removes the quotes and in the case of ca fingerprints this could prove problematic. |
pkoutsovasilis
left a comment
pkoutsovasilis
left a comment
There was a problem hiding this comment.
thanks for the contribution @btrieger . This is a big one so I will have to go through that one more time but in the meantime I left some comments. Also please any change that you do for the k8s templates try to introduce it also for the eck ones, from what I can tell everything here is compatible with both
|
@pkoutsovasilis will do I was tyring to keep it small but it seemed kind of unavoidable if I wanted to add all the variables available for logstash. |
…t supported for es
|
@btrieger is it possible to do a basic integration test in a Kubernetes cluster? We have tooling for setting up a cluster and a ESS stack, so we'd need to start logstash with the right configuration, have agent ship to logstash, and verify if we see the data in ES. Would that be a lot of work, in your view? |
|
@swiatekm it shouldn't be to bad. Where is the tooling located? I have run it all locally and confirmed it is working so I have the k8s manifests |
|
@btrieger have a look at . You can even skip sending the data and just demonstrate that agent is healthy with a logstash output. |
|
@swiatekm I added a logstash test. Let me know your thoughts. |
swiatekm
left a comment
swiatekm
left a comment
There was a problem hiding this comment.
The test looks good to me, minus the rather involved health check. Does it really need to be so complicated?
| // Check Logstash API health | ||
| err := kCtx.client.Resources().ExecInPod(ctx, namespace, pod.Name, "logstash", | ||
| []string{"curl", "-s", "localhost:9600/_node/stats"}, stdout, stderr) | ||
| if err != nil { | ||
| return false | ||
| } | ||
|
|
||
| var result map[string]interface{} | ||
| if err := json.Unmarshal(stdout.Bytes(), &result); err != nil { | ||
| return false | ||
| } | ||
|
|
||
| // Check if the pipeline is running - try multiple paths as structure may vary | ||
| if pipelines, ok := result["pipelines"].(map[string]interface{}); ok { | ||
| // Check for "main" pipeline (default name) | ||
| if main, ok := pipelines["main"].(map[string]interface{}); ok { | ||
| if workers, ok := main["workers"].(float64); ok && workers > 0 { | ||
| return true | ||
| } | ||
| } | ||
| // If there are any pipelines with workers, that's good enough | ||
| for _, pipelineData := range pipelines { | ||
| if pipelineMap, ok := pipelineData.(map[string]interface{}); ok { | ||
| if workers, ok := pipelineMap["workers"].(float64); ok && workers > 0 { | ||
| return true | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Alternative check: just verify the API is responding and has valid structure | ||
| if _, hasJVM := result["jvm"]; hasJVM { | ||
| if _, hasPipelines := result["pipelines"]; hasPipelines { | ||
| t.Logf("Logstash API is responding with pipeline data") | ||
| return true | ||
| } | ||
| } |
There was a problem hiding this comment.
Do we really need to do all this just to check Logstash health?
6 participants
What does this PR do?
This PR adds in the Logstash output for elastic agents to the standalone helm chart.
Why is it important?
Without this, standalone agents cannot connect to Logstash clusters. This makes the chart unusable for end users who have logstash in between agent and elasticsearch.
Checklist
./changelog/fragmentsusing the changelog toolDisruptive User Impact
This should have no user impact and be backward compatable.
How to test this PR locally
Follow steps in examples/kubernetes-logstash-output
Related issues