[Heartbeat] Synthetics journey/end event has incorrect mapping for start/end time

[vigneshshanmugam]

Summary

With the introduction of the recent changes in the libbeat #38767, We kind of missed out some of the parsing errors on the synthetics documents. When synthetics emits journey/end event type, we write payload.start and payload.end which gets sent to the ES even though we don't use that event in the UI which was done as part of the Summarizer changes #36519. As a result, we are seeing the parsing exception as the type doesn't match the expected format in the ES.

{"log.level":"warn","@timestamp":"2024-07-11T18:04:49.262-0700","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus","file.name":"elasticsearch/client.go","file.line":488},"message":"Cannot index event (status=400): dropping event! Look at the event log to view the event and cause.","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-07-11T18:04:49.262-0700","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus","file.name":"elasticsearch/client.go","file.line":489},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Meta:null, Fields:null, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}, EncodedEvent:(*elasticsearch.encodedEvent)(0x1400059a480)} (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:1176] failed to parse field [synthetics.payload.start] of type [date] in document with id 'YBt4pJABaBhXKe_gqLtt'. Preview of field's value: '6174580.705686833'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [6174580.705686833] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"heartbeat","log.type":"event","ecs.version":"1.6.0"}

Not sure how we missed this, Probably there are some changes on the ES side from 8.14.x, I haven't verified this yet.

Run with the new logging flag to output the full event, env ELASTIC_SYNTHETICS_CAPABLE=true ./heartbeat -E logging.event_data.to_stderr=true -e

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[vigneshshanmugam]

Potential fix for this - elastic/synthetics#709. Will add the mappings for the new fields to keep in sync with step duration.