Feature Request: Add community_id to all network log types

[dcode]

If any log has 5-tuple information, it should contain the community_id field for correlation across data types. As it stands today, one lookup has to find the conn entry, and another lookup to find related logs.

[ckreibich]

Sorry for the late response here @dcode — yes, that is true. This was actually our original idea, until we noticed that Zeek currently lacks a good way to generalize log modifications to a set of applicable logs. You'd currently need to redef each applicable log's Info record. That's certainly doable but not very elegant.

Out of curiosity, in which log would you most like to have this?

[sethhall]

That's not entirely true. Zeek's logging framework does have a generalized mechanism. The current limitation is that when the mechanism is used to extend all logs, you can't inspect the content of the log in question in order to decide what you put in the extension field.

If we solved that problem, this would be easily doable.

[ckreibich]

What mechanism do you mean? In filters or the writer?

[sethhall]

There is a feature in the logging framework to create log extension fields which are globally applied across all logs. Some common things people use it for are to add the worker that wrote the log line and the timestamp when the log line was written. The unfortunate part and what mostly limits it's utility is that you can't access the record being logged in the callback function that this feature is implemented through. If you could access the logged record in some way, you could inspect it to see if it has an "id" field and check it's type and then do some extended informational logging (like writing the community_id).

Here is the function prototype for the log extension mechanism:
https://github.com/bro/bro/blob/master/scripts/base/frameworks/logging/main.bro#L150

Here is a short example using it globally...

type LogExtension: record {
        path:   string &log;
        system_name:   string &log;
        write_ts: time   &log;
};

function add_log_extension(path: string): LogExtension
        {
        return LogExtension($path        = path,
                                     $system_name = peer_description,
                                     $write_ts    = network_time());
        }
redef Log::default_ext_prefix = "_";
redef Log::default_ext_func = add_log_extension;

Ideally, the Log::default_ext_func function would have a second argument that is an anonymous record and Zeek would give you the ability to inspect anonymous records.

[dcode]

@sethhall, I've used that log extension before and that's the first thing I thought of, but without the log record, it doesn't fit this usecase.

[ckreibich]

Ah, right! I now remember reading over this and going "huh", but it was too early in my logging framework career. 😄 Thanks for the cluebatting!
This seems to have a deficiency in that there's no particularly graceful way of handling the presence of multiple such global functions, but that will be easy to fix.
It's a pity though that this moves pretty far from the type-oriented extension mechanism (via redef) that we have elsewhere ... ideally I'd want a conditional redef for adding to the Info record, depending on what else has been added to it. The fact that this is about logging-related records would then become secondary. We tried if ( record_fields(x) ) — it's nearly there but doesn't quite work right atm.
Definitely fun stuff!

[sethhall]

@dcode Yeah, the original intent of the log extension mechanism was to figure out a way to get access to the log record but we couldn't do it at the time. We might be able to revisit that now with some new features that have been added to Zeek.

[defensivedepth]

Hello there!

Was wondering what the status of this issue is? Any progress?

[ckreibich]

I'd still like to have a way to do this in a controlled yet general way in the logging framework. But others have put in the elbow grease to do it manually for all of Zeek's logs, see here if you prefer that: https://github.com/DynamiteAI/publish-community_id

[dougburks]

@ckreibich Has anything changed since your last comment in 2020? Thanks!

[ckreibich]

I'm afraid no. There are two potential approaches, and we've not found the time for either:

• Make conn_id extensible so the ID could become part of it, and thus automatically appear in the logs
• Give the ext_func approach more context about the log it is extending, so it can tuck the ID on where needed.

Duly noted though that folks still want to see this capability. 👍