[libbeat] 'append' processor panics when field value is a slice

[nkormakov]

Winlogbeat 8.12.2, 8.19.6 (tested these two, probably all versions affected)

this processor adds various field including host.ip

processors:
  - add_host_metadata:
      cache.ttl: 5m
      netinfo.enabled: true

then i've tried to aggregate multiple ip into one field

  - append:
      target_field: "related.ip"
      fields:
        - host.ip
        - winlog.event_data.ClientIP
        - winlog.event_data.IpAddress
        - winlog.event_data.Ipaddress
        - winlog.event_data.DestinationIp
        - winlog.event_data.SourceIp
        - winlog.event_data.TargetIP
        - winlog.user_data.IpAddress
      allow_duplicate: false
      ignore_missing: true
      ignore_empty_values: true
      fail_on_error: false

the aforementioned config causes this trace

panic: runtime error: hash of unhashable type []string
goroutine 449 [running]:
github.com/elastic/beats/v7/libbeat/processors/actions.removeDuplicates(...)
        github.com/elastic/beats/v7/libbeat/processors/actions/append.go:176
github.com/elastic/beats/v7/libbeat/processors/actions.(*appendProcessor).appendValues(0xc000146960, {0xc000057510, 0xa}, {0xc00010be80, 0x8, 0x0?}, {0x0, 0x0, 0x0?}, 0xc000c3da40)
        github.com/elastic/beats/v7/libbeat/processors/actions/append.go:143 +0x9e8
github.com/elastic/beats/v7/libbeat/processors/actions.(*appendProcessor).Run(0xc000146960, 0xc000c3da40?)
        github.com/elastic/beats/v7/libbeat/processors/actions/append.go:82 +0x90
github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run(0xc000a51200, 0x182d285?)
        github.com/elastic/beats/v7/libbeat/publisher/processing/processors.go:129 +0xa2
github.com/elastic/beats/v7/libbeat/publisher/processing.(*processorFn).Run(0xc0010080d0?, 0xc000c3da40?)
        github.com/elastic/beats/v7/libbeat/publisher/processing/processors.go:159 +0x22
github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run(0xc00101a060, 0x5?)
        github.com/elastic/beats/v7/libbeat/publisher/processing/processors.go:129 +0xa2
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish(0xc00102a000, {{0x2934b2cc, 0xee09d0f79, 0x0}, 0x0, 0xc001327e90, {0x39fcbe0, 0xc000c3da00}, 0x0})
        github.com/elastic/beats/v7/libbeat/publisher/pipeline/client.go:97 +0x125
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish(0xc0006ac048?, {{0x2934b2cc, 0xee09d0f79, 0x0}, 0x0, 0xc001327e90, {0x39fcbe0, 0xc000c3da00}, 0x0})
        github.com/elastic/beats/v7/libbeat/publisher/pipeline/client.go:77 +0x10f
github.com/elastic/beats/v7/winlogbeat/beater.(*eventLogger).run(0xc0000a9380, 0xc0000142a0, {0x422f578?, 0xc0000b6a00}, {{0xc000b89440, 0x22}, 0xdd337, {0x1b88e3a0, 0xee09d0e56, 0x0}, ...}, ...)
        github.com/elastic/beats/v7/winlogbeat/beater/eventlogger.go:214 +0x16af
github.com/elastic/beats/v7/winlogbeat/beater.(*Winlogbeat).processEventLog(0x0?, 0x0?, 0x0?, {{0xc000b89440, 0x22}, 0xdd337, {0x1b88e3a0, 0xee09d0e56, 0x0}, {0xc00011b000, ...}}, ...)
        github.com/elastic/beats/v7/winlogbeat/beater/winlogbeat.go:217 +0xb3
created by github.com/elastic/beats/v7/winlogbeat/beater.(*Winlogbeat).Run
        github.com/elastic/beats/v7/winlogbeat/beater/winlogbeat.go:185 +0x3da

I've commented out every field and the error went away only when I remove host.ip, all other fields are ok and no panic

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

[andrewkroh]

This is a bug in the append processor at L175

beats/libbeat/processors/actions/append.go

Lines 171 to 178 in ac86553

	func removeDuplicates(dirtyArr []interface{}) (cleanArr []interface{}) {
	set := make(map[interface{}]bool, 0)
	for _, val := range dirtyArr {
	if _, ok := set[val]; !ok {
	set[val] = true
	cleanArr = append(cleanArr, val)
	}
	}

The Go spec states

The comparison operators == and != must be fully defined for operands of the key type; thus the key type must not be a function, map, or slice. If the key type is an interface type, these comparison operators must be defined for the dynamic key values; failure will cause a run-time panic.

Demonstration of the problem: https://play.golang.com/p/LGbW0ic3n5Y

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)