Skip to content

Commit

Permalink
Merge branch 'main' into feat/experimental-api-userdata
Browse files Browse the repository at this point in the history
  • Loading branch information
marc-gr authored Nov 8, 2024
2 parents 0213def + 6be1c24 commit f1de1c0
Show file tree
Hide file tree
Showing 25 changed files with 1,257 additions and 81 deletions.
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ CHANGELOG*
/libbeat/processors/dns/ @elastic/sec-deployment-and-devices
/libbeat/processors/registered_domain/ @elastic/sec-deployment-and-devices
/libbeat/processors/syslog/ @elastic/sec-deployment-and-devices
/libbeat/processors/translate_ldap_attribute/ @elastic/sec-windows-platform
/libbeat/processors/translate_sid/ @elastic/sec-windows-platform
/libbeat/reader/syslog/ @elastic/sec-deployment-and-devices
/libbeat/scripts @elastic/ingest-eng-prod
Expand Down
5 changes: 4 additions & 1 deletion CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Set timeout of 1 minute for FQDN requests {pull}37756[37756]
- Fix issue where old data could be saved in the memory queue after acknowledgment, increasing memory use {pull}41356[41356]
- Ensure Elasticsearch output can always recover from network errors {pull}40794[40794]
- Add `translate_ldap_attribute` processor. {pull}41472[41472]

*Auditbeat*

Expand Down Expand Up @@ -171,6 +172,8 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Improve modification time handling for entities and entity deletion logic in the Active Directory entityanalytics input. {pull}41179[41179]
- Journald input now can read events from all boots {issue}41083[41083] {pull}41244[41244]
- Fix double encoding of client_secret in the Entity Analytics input's Azure Active Directory provider {pull}41393[41393]
- Fix errors in SQS host resolution in the `aws-s3` input when using custom (non-AWS) endpoints. {pull}41504[41504]
- The azure-eventhub input now correctly reports its status to the Elastic Agent on fatal errors {pull}41469[41469]

*Heartbeat*

Expand Down Expand Up @@ -237,7 +240,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Replace Ubuntu 20.04 with 24.04 for Docker base images {issue}40743[40743] {pull}40942[40942]
- Reduce memory consumption of k8s autodiscovery and the add_kubernetes_metadata processor when Deployment metadata is enabled
- Add `lowercase` processor. {issue}22254[22254] {pull}41424[41424]

- Add `uppercase` processor. {issue}22254[22254] {pull}41535[41535]
*Auditbeat*

- Added `add_session_metadata` processor, which enables session viewer on Auditbeat data. {pull}37640[37640]
Expand Down
8 changes: 4 additions & 4 deletions NOTICE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1865,11 +1865,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4
Version: v4.6.0
Version: v4.8.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4@v4.6.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4@v4.8.0/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -12800,12 +12800,12 @@ Contents of probable licence file $GOMODCACHE/github.com/dolmen-go/contextio@v0.


--------------------------------------------------------------------------------
Dependency : github.com/andrewkroh/goja
Dependency : github.com/elastic/goja
Version: v0.0.0-20190128172624-dd2ac4456e20
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/andrewkroh/[email protected]/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/elastic/[email protected]/LICENSE:

Copyright (c) 2016 Dmitry Panov

Expand Down
216 changes: 216 additions & 0 deletions filebeat/tests/integration/translate_ldap_attribute_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
// Licensed to Elasticsearch B.V. under one or more contributor
// license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright
// ownership. Elasticsearch B.V. licenses this file to you under
// the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

//go:build integration

package integration

import (
"context"
"errors"
"fmt"
"io"
"os"
"path"
"path/filepath"
"testing"
"time"

"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/image"
"github.com/docker/docker/client"
"github.com/docker/go-connections/nat"
"github.com/go-ldap/ldap/v3"
"github.com/stretchr/testify/require"

"github.com/elastic/beats/v7/libbeat/tests/integration"
"github.com/elastic/elastic-agent-autodiscover/docker"
)

const translateguidCfg = `
filebeat.inputs:
- type: filestream
id: "test-translateguidCfg"
paths:
- %s
queue.mem:
flush.min_events: 1
flush.timeout: 0.1s
path.home: %s
output.file:
path: ${path.home}
filename: "output-file"
logging:
metrics:
enabled: false
processors:
- add_fields:
fields:
guid: '%s'
- translate_ldap_attribute:
field: fields.guid
target_field: fields.common_name
ldap_address: 'ldap://localhost:1389'
ldap_base_dn: 'dc=example,dc=org'
ldap_bind_user: 'cn=admin,dc=example,dc=org'
ldap_bind_password: 'adminpassword'
ldap_search_attribute: 'entryUUID'
`

func TestTranslateGUIDWithLDAP(t *testing.T) {
startOpenldapContainer(t)

var entryUUID string
require.Eventually(t, func() bool {
var err error
entryUUID, err = getLDAPUserEntryUUID()
return err == nil
}, 10*time.Second, time.Second)

filebeat := integration.NewBeat(
t,
"filebeat",
"../../filebeat.test",
)
tempDir := filebeat.TempDir()

// 1. Generate the log file path
logFilePath := path.Join(tempDir, "log.log")
integration.GenerateLogFile(t, logFilePath, 1, false)

// 2. Write configuration file and start Filebeat
filebeat.WriteConfigFile(
fmt.Sprintf(translateguidCfg, logFilePath, tempDir, entryUUID),
)
filebeat.Start()

var outputFile string
require.Eventually(t, func() bool {
outputFiles, err := filepath.Glob(path.Join(tempDir, "output-file-*.ndjson"))
if err != nil {
return false
}
if len(outputFiles) != 1 {
return false
}
outputFile = outputFiles[0]
return true
}, 10*time.Second, time.Second)

// 3. Wait for the event with the expected translated guid
filebeat.WaitFileContains(
outputFile,
fmt.Sprintf(`"fields":{"guid":"%s","common_name":["User1","user01"]}`, entryUUID),
10*time.Second,
)
}

func startOpenldapContainer(t *testing.T) {
ctx := context.Background()
c, err := docker.NewClient(client.DefaultDockerHost, nil, nil)
if err != nil {
t.Fatal(err)
}

reader, err := c.ImagePull(ctx, "bitnami/openldap:2", image.PullOptions{})
if err != nil {
t.Fatal(err)
}
if _, err = io.Copy(os.Stdout, reader); err != nil {
t.Fatal(err)
}
reader.Close()

resp, err := c.ContainerCreate(ctx,
&container.Config{
Image: "bitnami/openldap:2",
ExposedPorts: nat.PortSet{
"1389/tcp": struct{}{},
},
Env: []string{
"LDAP_URI=ldap://openldap:1389",
"LDAP_BASE=dc=example,dc=org",
"LDAP_BIND_DN=cn=admin,dc=example,dc=org",
"LDAP_BIND_PASSWORD=adminpassword",
},
},
&container.HostConfig{
PortBindings: nat.PortMap{
"1389/tcp": []nat.PortBinding{
{
HostIP: "0.0.0.0",
HostPort: "1389",
},
},
},
}, nil, nil, "")
if err != nil {
t.Fatal(err)
}

if err := c.ContainerStart(ctx, resp.ID, container.StartOptions{}); err != nil {
t.Fatal(err)
}

t.Cleanup(func() {
defer c.Close()
if err := c.ContainerRemove(ctx, resp.ID, container.RemoveOptions{RemoveVolumes: true, Force: true}); err != nil {
t.Error(err)
}
})
}

func getLDAPUserEntryUUID() (string, error) {
// Connect to the LDAP server
l, err := ldap.DialURL("ldap://localhost:1389")
if err != nil {
return "", fmt.Errorf("failed to connect to LDAP server: %w", err)
}
defer l.Close()

err = l.Bind("cn=admin,dc=example,dc=org", "adminpassword")
if err != nil {
return "", fmt.Errorf("failed to bind to LDAP server: %w", err)
}

searchRequest := ldap.NewSearchRequest(
"dc=example,dc=org",
ldap.ScopeWholeSubtree, ldap.NeverDerefAliases, 1, 0, false,
"(cn=User1)", []string{"entryUUID"}, nil,
)

sr, err := l.Search(searchRequest)
if err != nil {
return "", fmt.Errorf("failed to execute search: %w", err)
}

// Process search results
if len(sr.Entries) == 0 {
return "", errors.New("no entries found for the specified username.")
}
entry := sr.Entries[0]
entryUUID := entry.GetAttributeValue("entryUUID")
if entryUUID == "" {
return "", errors.New("entryUUID is empty")
}
return entryUUID, nil
}
5 changes: 2 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0
github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs v1.2.1
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/consumption/armconsumption v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.6.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.8.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/costmanagement/armcostmanagement v1.1.1
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor v0.8.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0
Expand Down Expand Up @@ -415,8 +415,7 @@ replace (

github.com/Shopify/sarama => github.com/elastic/sarama v1.19.1-0.20220310193331-ebc2b0d8eef3
github.com/apoydence/eachers => github.com/poy/eachers v0.0.0-20181020210610-23942921fe77 //indirect, see https://github.com/elastic/beats/pull/29780 for details.
github.com/dop251/goja => github.com/andrewkroh/goja v0.0.0-20190128172624-dd2ac4456e20
github.com/dop251/goja_nodejs => github.com/dop251/goja_nodejs v0.0.0-20171011081505-adff31b136e6
github.com/dop251/goja => github.com/elastic/goja v0.0.0-20190128172624-dd2ac4456e20
github.com/fsnotify/fsevents => github.com/elastic/fsevents v0.0.0-20181029231046-e1d381a4d270
github.com/fsnotify/fsnotify => github.com/elastic/fsnotify v1.6.1-0.20240920222514-49f82bdbc9e3
github.com/google/gopacket => github.com/elastic/gopacket v1.1.20-0.20241002174017-e8c5fda595e6
Expand Down
8 changes: 4 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,8 @@ github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 h1:ywEEhmNahHBihViHepv3xP
github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0/go.mod h1:iZDifYGJTIgIIkYRNWPENUnqx6bJ2xnSDFI2tjwZNuY=
github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs v1.2.1 h1:0f6XnzroY1yCQQwxGf/n/2xlaBF02Qhof2as99dGNsY=
github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs v1.2.1/go.mod h1:vMGz6NOUGJ9h5ONl2kkyaqq5E0g7s4CHNSrXN5fl8UY=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.6.0 h1:AAIdAyPkFff6XTct2lQCxOWN/+LnA41S7kIkzKaMbyE=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.6.0/go.mod h1:noQIdW75SiQFB3mSFJBr4iRRH83S9skaFiBv4C0uEs0=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.8.0 h1:0nGmzwBv5ougvzfGPCO2ljFRHvun57KpNrVCMrlk0ns=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.8.0/go.mod h1:gYq8wyDgv6JLhGbAU6gg8amCPgQWRE+aCvrV2gyzdfs=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/costmanagement/armcostmanagement v1.1.1 h1:ehSLdbLah6kk6HTVc6e/lrbmbz7MMbpNxkOd3OYlhB0=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/costmanagement/armcostmanagement v1.1.1/go.mod h1:Am1cUioOk0HdZIsjpXJkQ4RIeQbwYsW6LkNIc5z/5XY=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/eventhub/armeventhub v1.2.0 h1:+dggnR89/BIIlRlQ6d19dkhhdd/mQUiQbXhyHUFiB4w=
Expand Down Expand Up @@ -140,8 +140,6 @@ github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuy
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alexbrainman/sspi v0.0.0-20210105120005-909beea2cc74 h1:Kk6a4nehpJ3UuJRqlA3JxYxBZEqCeOmATOvrbT4p9RA=
github.com/alexbrainman/sspi v0.0.0-20210105120005-909beea2cc74/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
github.com/andrewkroh/goja v0.0.0-20190128172624-dd2ac4456e20 h1:7rj9qZ63knnVo2ZeepYHvHuRdG76f3tRUTdIQDzRBeI=
github.com/andrewkroh/goja v0.0.0-20190128172624-dd2ac4456e20/go.mod h1:cI59GRkC2FRaFYtgbYEqMlgnnfvAwXzjojyZKXwklNg=
github.com/andybalholm/brotli v1.0.5 h1:8uQZIdzKmjc/iuPu7O2ioW48L81FgatrcpfFmiq/cCs=
github.com/andybalholm/brotli v1.0.5/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/antlr4-go/antlr/v4 v4.13.0 h1:lxCg3LAv+EUK6t1i0y1V6/SLeUi0eKEKdhQAlS8TVTI=
Expand Down Expand Up @@ -381,6 +379,8 @@ github.com/elastic/go-ucfg v0.8.8 h1:54KIF/2zFKfl0MzsSOCGOsZ3O2bnjFQJ0nDJcLhviyk
github.com/elastic/go-ucfg v0.8.8/go.mod h1:4E8mPOLSUV9hQ7sgLEJ4bvt0KhMuDJa8joDT2QGAEKA=
github.com/elastic/go-windows v1.0.2 h1:yoLLsAsV5cfg9FLhZ9EXZ2n2sQFKeDYrHenkcivY4vI=
github.com/elastic/go-windows v1.0.2/go.mod h1:bGcDpBzXgYSqM0Gx3DM4+UxFj300SZLixie9u9ixLM8=
github.com/elastic/goja v0.0.0-20190128172624-dd2ac4456e20 h1:bVZ3kDKa8Tqw9qvNrD91MwJMW6alg4Wn31l1TQ6RlTY=
github.com/elastic/goja v0.0.0-20190128172624-dd2ac4456e20/go.mod h1:A1DWjF89MFVnxzmzTaMF7CwVy9PDem7DalMkm8RIMoY=
github.com/elastic/gopacket v1.1.20-0.20241002174017-e8c5fda595e6 h1:VgOx6omXIMKozR+R4HhQRT9q1Irm/h13DLtSkejoAJY=
github.com/elastic/gopacket v1.1.20-0.20241002174017-e8c5fda595e6/go.mod h1:riddUzxTSBpJXk3qBHtYr4qOhFhT6k/1c0E3qkQjQpA=
github.com/elastic/gosigar v0.14.3 h1:xwkKwPia+hSfg9GqrCUKYdId102m9qTJIIr7egmK/uo=
Expand Down
1 change: 1 addition & 0 deletions libbeat/cmd/instance/imports_common.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ import (
_ "github.com/elastic/beats/v7/libbeat/processors/registered_domain"
_ "github.com/elastic/beats/v7/libbeat/processors/script"
_ "github.com/elastic/beats/v7/libbeat/processors/syslog"
_ "github.com/elastic/beats/v7/libbeat/processors/translate_ldap_attribute"
_ "github.com/elastic/beats/v7/libbeat/processors/translate_sid"
_ "github.com/elastic/beats/v7/libbeat/processors/urldecode"
_ "github.com/elastic/beats/v7/libbeat/publisher/includes" // Register publisher pipeline modules
Expand Down
6 changes: 6 additions & 0 deletions libbeat/docs/processors-list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,9 @@ endif::[]
ifndef::no_timestamp_processor[]
* <<processor-timestamp,`timestamp`>>
endif::[]
ifndef::no_translate_ldap_attribute_processor[]
* <<processor-translate-guid, `translate_ldap_attribute`>>
endif::[]
ifndef::no_translate_sid_processor[]
* <<processor-translate-sid, `translate_sid`>>
endif::[]
Expand Down Expand Up @@ -279,6 +282,9 @@ endif::[]
ifndef::no_timestamp_processor[]
include::{libbeat-processors-dir}/timestamp/docs/timestamp.asciidoc[]
endif::[]
ifndef::no_translate_ldap_attribute_processor[]
include::{libbeat-processors-dir}/translate_ldap_attribute/docs/translate_ldap_attribute.asciidoc[]
endif::[]
ifndef::no_translate_sid_processor[]
include::{libbeat-processors-dir}/translate_sid/docs/translate_sid.asciidoc[]
endif::[]
Expand Down
Loading

0 comments on commit f1de1c0

Please sign in to comment.