v2.3.3 collects duplicates even with skipKnownLogs: True

[wrightca]

Used v2.1 for years but I'm trying to upgrade to v2.3.3 to take advantage of the rewrite. However the new version is collecting/sending duplicate log records to Graylog even with the skipKnownLogs: True setting.

Config File:

log: # Define the executables own log settings
path: '/folder/collector.log'
debug: False
collect: # Define how to collect audit logs
workingDir: '/folder/'
autoSubscribe: True
skipKnownLogs: True
hoursToCollect: 24
contentTypes:
Audit.General: True
Audit.AzureActiveDirectory: True
Audit.Exchange: True
Audit.SharePoint: True
DLP.All: True
output: # Define outputs to send audit logs to
graylog:
enabled: True
address: IP
port: PORT

[wrightca]

I see it writing known_blobs now and updating that with each run, but that doesn't seem to prevent the duplicates.

[Nautilus980]

I confirm the behavior described. By my investigation its because the log entries are identified (as known_blobs) by its hashes, but Office 365 does not keep the JSON message fields order constant. I mean, sometimes it is

{
"first_name": "John",
"last_name": "Smith",
}

next time it is

{
"last_name": "Smith",
"first_name": "John",
}

despite the "Id" field is the same – so it's the single one log entry.