[o365]: additionalDetails_value causing index failures #15867
Description
Integration Name
Microsoft Office 365 [o365]
Dataset Name
o365.audit
Integration Version
2.33.1
Agent Version
9.2.0
Agent Output Type
elasticsearch
Elasticsearch Version
9.2.0
OS Version and Architecture
Ubuntu
Software/API Version
No response
Error Message
[1:664] failed to parse field [o365.audit.ExtendedProperties.additionalDetails_value] of type [keyword] in document with id 'nC4ODGiXP2iGesL5rC/SAOug70s='. Preview of field's value: '{UserType=Member}'
Event Original
No response
What did you do?
I attempted to deploy o365 audit
What did you see?
I saw documents going into the failure store with the above error message or similar. This is because the component template maps all fields under ExtendedProperties to keyword and in 2.32.0 a json extraction was added to additionalDetails_value and fields were copied from addtionalDetails_value but the new additionalDetails_value object is not removed after the needed fields are extracted.
What did you expect to see?
I expect no documents to go into the failure store.
Anything else?
No response