Added support for JSON containing multiple events

[tim427]

Currently the intelmq.bots.parsers.json.parser is only able to parse or single events in JSON, or multiple events in JSON, each on their own line.

This PR contains an option to parse multiple events within a JSON, by adding the multiple_events (boolean) to the config.

[sebix]

Do you have an example feed at hand (so we can extract an example for the tests, add it to the docs)? To my knowledge no documented feed is using such a format.

[sebix]

Converting the data forth and back appears to be inefficient.

[tim427]

Yeah, could imagine this. Any tips to do this a proper way?

Currently this PR is running in our production and works just fine, but I agree on the double JSON conversion isn't the most efficient way to do this.

[kamil-certat]

I'd suggest just loading here, and then in L33 checking config (or maybe just the object type?), and using MessafeFactory.from_dict if we already loaded data :)

[tim427]

Do you have an example feed at hand (so we can extract an example for the tests, add it to the docs)? To my knowledge no documented feed is using such a format.

Our National Cyber Security Centre (NCSC) is sending us "IntelMQ JSON's" in a ZIP-file by mail.
The ZIP-file contains a single JSON-file.

Here's an example (I tried to anonymise most values):

[
    {
        "extra.dataset_collections": "0",
        "extra.dataset_files": "1",
        "extra.dataset_infected": "false",
        "extra.dataset_ransom": "null",
        "extra.dataset_rows": "0",
        "extra.dataset_size": "301",
        "protocol.application": "https",
        "protocol.transport": "tcp",
        "source.asn": 12345689,
        "source.fqdn": "fqdn-example-1.tld",
        "source.geolocation.cc": "NL",
        "source.geolocation.city": "Enschede",
        "source.geolocation.latitude": 52.0000000000000,
        "source.geolocation.longitude": 6.0000000000000,
        "source.geolocation.region": "Overijssel",
        "source.ip": "127.1.2.1",
        "source.network": "127.1.0.0/16",
        "source.port": 80,
        "time.source": "2024-12-16T02:08:06+00:00"
    },
    {
        "extra.dataset_collections": "0",
        "extra.dataset_files": "1",
        "extra.dataset_infected": "false",
        "extra.dataset_ransom": "null",
        "extra.dataset_rows": "0",
        "extra.dataset_size": "615",
        "extra.os_name": "Ubuntu",
        "extra.software": "Apache",
        "extra.tag": "rescan",
        "extra.version": "2.4.58",
        "protocol.application": "https",
        "protocol.transport": "tcp",
        "source.asn": 12345689,
        "source.fqdn": "fqdn-example-2.tld",
        "source.geolocation.cc": "NL",
        "source.geolocation.city": "Eindhoven",
        "source.geolocation.latitude": 51.0000000000000,
        "source.geolocation.longitude": 5.0000000000000,
        "source.geolocation.region": "North Brabant",
        "source.ip": "127.1.2.2",
        "source.network": "127.1.0.0/16",
        "source.port": 443,
        "time.source": "2024-12-16T02:08:12+00:00"
    },
    {
        "extra.dataset_collections": "0",
        "extra.dataset_files": "1",
        "extra.dataset_infected": "false",
        "extra.dataset_ransom": "null",
        "extra.dataset_rows": "0",
        "extra.dataset_size": "421",
        "protocol.application": "http",
        "protocol.transport": "tcp",
        "source.asn": 12345689,
        "source.geolocation.cc": "NL",
        "source.geolocation.city": "Enschede",
        "source.geolocation.latitude": 52.0000000000000,
        "source.geolocation.longitude": 6.0000000000000,
        "source.geolocation.region": "Overijssel",
        "source.ip": "127.1.2.3",
        "source.network": "127.1.0/16",
        "source.port": 9000,
        "time.source": "2024-12-15T21:09:49+00:00"
    }
]

[sebix]

I added a few changes here:

• A test cases
  • that's where I noticed that the json parser didn't add the required classification.type field if it doesn't exist in input data, so added that as well
• The optimizations as discussed above
  • Which also revealed another bug in Message.from_dict which modified the parameter
• add documentation

[sebix]

...and found & fixed another bug in intelmq.lib.message.Message.from_dict:
Raise a ValueError if message type is not determinable

[sebix]

As I wrote a major part of this PR, I won't merge it myself

@aaronkaplan could you do the review instead?

[sebix]

Rebased on develop to fix conflicts

[sebix]

@kamil-certat maybe you can have a look?

[kamil-certat]

It looks good except of non-handled case of both flags being set. Please either handle it or explicitly forbid

[kamil-certat]

By having two flags and no enforcement if they are exclusive, we allow setting all combinations, including

splitlines = True
multiple_events = True

which is not supported by the code, but still makes a theoretically possible case of multiple lines, each of them containing a list of multiple event dictionaries.

I'd suggest on of following solutions:

1. support that case,
2. switch from two flags to a one format (or similar) configuration, that supports options like e.g. single, splitlines, multiple
3. add a config validation forbidding starting a bot with both flags set.

[sebix]

Thanks for catching this.
I chose the easiest option: Forbidding it.
There's no reason to implement it; switching to a different mode setting would also increase the required effort, and require upgrade functions. We can always make the switch to a format parameter later with the same effort as now.