zmq format for ntopng incorrect? (openflow support)

[level20peon]

I am trying to push some data into ntopng 6.0, but it doesn't receive any flows, despite receiving zmq messages:

So I went ahead and compared the output of netflow2ng (0.0.5) with the output of nprobe (10.5) and they seem to be very different:
netflow2ng output

{
    "1": 1421,
    "10": 0,
    "11": 33923,
    "12": "61.250.232.215",
    "130": "192.168.0.1",
    "14": 0,
    "15": "61.250.232.215",
    "16": 0,
    "17": 0,
    "2": 6,
    "21": 1706022328,
    "22": 1706022318,
    "29": 0,
    "30": 0,
    "32": 0,
    "4": 6,
    "44": 0,
    "45": 0,
    "5": 0,
    "52": 0,
    "54": 0,
    "58": 0,
    "59": 0,
    "6": 27,
    "60": 4,
    "61": 0,
    "7": 22,
    "8": "89.0.69.55",
    "80": "00:00:00:00:00:00",
    "81": "00:00:00:00:00:00",
    "88": 0,
    "89": 0
}

nprobe output

{
    "56": "00:00:00:00:00:00",
    "57": "00:00:00:00:00:00",
    "10": 0,
    "14": 0,
    "58": 0,
    "8": "192.168.0.1",
    "12": "192.168.0.4",
    "7": 53,
    "11": 51996,
    "27": "::",
    "28": "::",
    "60": 4,
    "4": 17,
    "35632.118": "5",
    "35632.560": 0,
    "1": 64,
    "2": 1,
    "23": 0,
    "24": 0,
    "22": 1706022259,
    "21": 1706022259,
    "35632.78": 0,
    "35632.79": 0,
    "35632.509": 0,
    "35632.527": 0,
    "130": "192.168.0.1",
    "61": 2,
    "34": 1,
    "42": 18,
    "35632.471": "192.168.3.28",
    "35632.122": "",
    "58051": 1,
    "225": "0.0.0.0",
    "226": "0.0.0.0",
    "227": 0,
    "228": 0
}

I am using the @NTOPNG@ template on nprobe, which expands to
%IN_SRC_MAC %OUT_DST_MAC %INPUT_SNMP %OUTPUT_SNMP %SRC_VLAN %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %L7_PROTO %L7_CONFIDENCE %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %CLIENT_TCP_FLAGS %SERVER_TCP_FLAGS %L7_PROTO_RISK %L7_RISK_SCORE %EXPORTER_IPV4_ADDRESS %DIRECTION %SAMPLING_INTERVAL %TOTAL_FLOWS_EXP %NPROBE_IPV4_ADDRESS %NPROBE_INSTANCE_NAME %FLOW_SOURCE %POST_NAT_SRC_IPV4_ADDR %POST_NAT_DST_IPV4_ADDR %POST_NAPT_SRC_TRANSPORT_PORT %POST_NAPT_DST_TRANSPORT_PORT

Looking at https://github.com/cloudflare/goflow/tree/master/decoders/netflow, I can see how I could manipulate the JSON output by incorporating those additional fields into https://github.com/synfinatic/netflow2ng/blob/main/cmd/zmq.go. However, I am not sure how those "combined" fields (i.e. "35632.527") are created / what they mean. Also, I am not sure why for example the MAC addresses use different identifiers on the nprobe string (for example 56 - IPFIX_FIELD_sourceMacAddress, 57 - IPFIX_FIELD_postDestinationMacAddress) compared to the identifiers netflow2ng uses (for example 80 - IPFIX_FIELD_destinationMacAddress, 81 - IPFIX_FIELD_postSourceMacAddress). Same goes for a couple of the other fields - I guess the template used by netflow2ng at the moment is an old version, compatible with previous versions of ntopng!?

So, for now at least, ntopng doesn't receive any flows from netflow2ng. Could you give any pointers on how to incorporate the changes neccessary to support a more current ntopng version?

[synfinatic]

Have you looked at the NetFlow spec? That's mostly what I used to map things. If I had to guess, the .XXX indicates a VLAN tag/sub-interface.... But that's just a guess.

[level20peon]

the .XXX is actually a IANA PEN specifically for nprobe (see https://www.ntop.org/nprobe/iana-assigned-a-pen-to-ntop/). But it turns out that this isn't actually required for sending flows to ntopng.

I ran a few tests and stripped down the amount of fields sent by nprobe to this:
%IN_BYTES %INPUT_SNMP %L4_DST_PORT %IPV4_DST_ADDR %EXPORTER_IPV4_ADDRESS %OUTPUT_SNMP %IN_PKTS %LAST_SWITCHED %FIRST_SWITCHED %SAMPLING_INTERVAL %PROTOCOL %IN_SRC_MAC %OUT_DST_MAC %SRC_VLAN %IP_PROTOCOL_VERSION %DIRECTION %L4_SRC_PORT %IPV4_SRC_ADDR

I also modified https://github.com/synfinatic/netflow2ng/blob/main/cmd/zmq.go to output exactly the same (note: omitting IPV6 here).

This is an example of a zmq message from nprobe:

{
    "1": 636,
    "10": 0,
    "11": 80,
    "12": "192.168.0.64",
    "130": "192.168.0.1",
    "14": 0,
    "2": 9,
    "21": 1706087646,
    "22": 1706087642,
    "34": 1,
    "4": 6,
    "56": "00:00:00:00:00:00",
    "57": "00:00:00:00:00:00",
    "58": 0,
    "60": 4,
    "61": 2,
    "7": 53233,
    "8": "192.168.0.4"
}

This is an example of a zmq message from netflow2ng:

{
    "1": 736,
    "10": 0,
    "11": 56358,
    "12": "192.168.0.41",
    "130": "192.168.0.1",
    "14": 0,
    "2": 10,
    "21": 1706088876,
    "22": 1706088856,
    "34": 1,
    "4": 6,
    "56": "00:00:00:00:00:00",
    "57": "00:00:00:00:00:00",
    "58": 0,
    "60": 4,
    "61": 0,
    "7": 143,
    "8": "138.201.223.84"
}

As you can see, the output has the same structure. Still, the nprobe messages result in actual flows in ntopng, and the netflow2ng messages don't result in flows in ntopng, only in zmq messages.

By the way, the ntopng log doesn't complain about anything (no errors, no warnings at debug log level).

Do you have any pointers on how to proceed here? Did you try to use netflow2ng with ntopng 6.0?

[level20peon]

Maybe this is about templates from the netflow source? I am using softflowd on openwrt and I noticed that ntopng only starts parsing flows on a specific nprobe after the flow template is sent by softflowd. It does so automatically periodically, but I can also trigger this by using softflowctl send-template. nprobe then outputs this to the syslog:

Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=1024][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=1]
Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=1025][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=2]
Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=2048][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=3]
Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=2049][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=4]
Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=256][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=5]

I don't see netflow2ng reacting to those template transmissions at all. Could this be relevant?

edit: after having a look at the /templates endpoint of the netflow2ng webserver, it seems that those template definitions are imported after all, so this shouldn't be the issue here!?
{"192.168.0.1":{"9":{"0":{"1024":{"TemplateId":1024,"FieldCount":14,"Fields":[{"Type":8,"Length":4},{"Type":12,"Length":4},{"Type":22,"Length":4},{"Type":21,"Length":4},{"Type":1,"Length":4},{"Type":2,"Length":4},{"Type":10,"Length":4},{"Type":14,"Length":4},{"Type":7,"Length":2},{"Type":11,"Length":2},{"Type":4,"Length":1},{"Type":6,"Length":1},{"Type":60,"Length":1},{"Type":5,"Length":1}]},"1025":{"TemplateId":1025,"FieldCount":11,"Fields":[{"Type":8,"Length":4},{"Type":12,"Length":4},{"Type":22,"Length":4},{"Type":21,"Length":4},{"Type":1,"Length":4},{"Type":2,"Length":4},{"Type":10,"Length":4},{"Type":14,"Length":4},{"Type":32,"Length":2},{"Type":60,"Length":1},{"Type":5,"Length":1}]},"2048":{"TemplateId":2048,"FieldCount":14,"Fields":[{"Type":27,"Length":16},{"Type":28,"Length":16},{"Type":22,"Length":4},{"Type":21,"Length":4},{"Type":1,"Length":4},{"Type":2,"Length":4},{"Type":10,"Length":4},{"Type":14,"Length":4},{"Type":7,"Length":2},{"Type":11,"Length":2},{"Type":4,"Length":1},{"Type":6,"Length":1},{"Type":60,"Length":1},{"Type":5,"Length":1}]},"2049":{"TemplateId":2049,"FieldCount":11,"Fields":[{"Type":27,"Length":16},{"Type":28,"Length":16},{"Type":22,"Length":4},{"Type":21,"Length":4},{"Type":1,"Length":4},{"Type":2,"Length":4},{"Type":10,"Length":4},{"Type":14,"Length":4},{"Type":139,"Length":2},{"Type":60,"Length":1},{"Type":5,"Length":1}]},"256":{"TemplateId":256,"ScopeLength":4,"OptionLength":8,"Scopes":[{"Type":2,"Length":4}],"Options":[{"Type":34,"Length":4},{"Type":35,"Length":1}]}}}}}

[synfinatic]

I haven't looked at this at all to be honest... I'm not even sure if I'm using noting 6 or 5.x off the top of my head since I very rarely even look at it anymore. Probably worth looking at the ntopng code and see what it's doing?

[jamesfreeman959]

Hi! I'm seeing what I think is the same issue with Ntopng 6.1 - seeing collected ZMQ messages, but no collected flows and no traffic. I turned the logging level up to 6 and buried amongst the debug messages, I did see:

No allowed interface found for zmq://x.x.x.x:5556

I've only had time to quickly review the code for Ntopng, but this statement seems to come from Ntop.cpp where there are a few functions that check if an interface is allowed. I must confess I haven't fully understood them yet. I'm not clear what the issue is yet - whether ZMQ interfaces have become a licensable feature and so are not allowed. I can't find any reference in the GUI to allowing interfaces, or in the documentation (which in turn leads me to think this might be license related but I must stress this is an assumption).

It could be a separate issue of course, but it seems rather coincidental that I'm seeing the same counter behaviour as shown in the original post, and it appears that Ntopng is deliberately dropping this traffic because the interface "isn't allowed". The Ntopng documentation does not make is clear that this is/has become a licensable feature.

[ToqQrrl]

I have the same problem with ntopng 6.0 (stable) ---> no traffic being recorded ... all 0

Using ntopng with --interface zmq://x.x.x.x:5556

Would love it if this was corrected :-)

[gitet]

I have the same problem with ntopng 6.0 (stable) ---> no traffic being recorded ... all 0

Using ntopng with --interface zmq://x.x.x.x:5556

Would love it if this was corrected :-)

Hi @ToqQrrl , I think the correct syntax to use is '--interface tcp://x.x.x.x:5556' or I may be missing something?

Anyway, with the above in my config I am able to collect flows, but if I look at the hosts, the "Throughput" and "Total Bytes" columns are both 0 for all entries.

Using netflow2ng 0.0.5 to consume netflow v9 from a opnsense firewall
ntopng community edition 6.0240207 on ubuntu 22.04 (I see the same with ntopng 6.1.x running on my raspberrypi)

[ToqQrrl]

Hi @gitet … in the documentation it says that we should use zmq:// instead of tcp:// … but both give the same result … I have le flows but all throughput and total bytes are 0 just like you

[synfinatic]

I haven't had a chance yet to dig deeper, but looks like https://github.com/ntop/ntopng/blob/dev/src/ZMQParserInterface.cpp has changed significantly from 5.6 to 6.0 which most likely explains what is going on here.

[synfinatic]

Digging further... not sure what your problem is as I can't reproduce it using ntopng v6.0.240305.

Can you share your ntopng.conf. Are you sure your device sending the netflow packets is using v9 and not v10/IPFIX?

FWIW, looks like v5 used tcp:// and v6 is now using zmq://, but both are valid:

[--interface|-i] <interface|pcap>   | Input interface name (numeric/symbolic),
                                    | view or pcap file path, including:
                                    | zmq://<IP address>         [ZMQ flow collection]
                                    | tcp://<IP address>         [DEPRECATED ZMQ flow collection]
                                    | kafka://<brokers list>     [Kafka flow collection]
                                    | kafka-ssl://<brokers list> [Kafka flow collection over SSL/TLS]

[gitet]

@synfinatic Thank you for not giving up on this 👍
I'm trying to collect netflow generated by my OpnSense firewall (version 24.1.3_1) - it can do v5 or v9, I've selected v9 in the UI. Wireshark decodes the received datagrams as "Cisco Netflow/IPFix" version 9.

I'm sending the same netflow data to a raspberrypi running nfcapd - no problems over there, so the data seems to be healthy.

Netflow2ng info:

netflow2ng --version
netflow2ng v0.0.5 -- Copyright 2020-2022 Aaron Turner
c819d0568627c4eaad3982bab0c16c2630bc8ef0 (v0.0.5-7-gc819d05) built at 2024-02-13T20:51:39+0100

Invoked with netflow2ng -a 192.168.10.100:2055 -z tcp://127.0.0.1:5556

Small json sample produced with -l trace added:

{"1":0,"10":2,"11":443,"12":"35.186.227.140","130":"192.168.10.254","14":1,"15":"$MY_ROUTER_IP","16":0,"17":0,"2":0,"21":1710428931,"22":1710428625,"29":24,"30":0,"32":0,"4":6,"44":24,"45":0,"5":0,"52":0,"54":0,"58":0,"59":0,"6":27,"60":4,"61":0,"7":56668,"8":"192.168.10.100","80":"00:00:00:00:00:00","81":"00:00:00:00:00:00","88":0,"89":0} 

My ntopng config:

grep -v '^#.*' /etc/ntopng/ntopng.conf
-G=/var/run/ntopng.pid
-i=wlp4s0
-i=tcp://127.0.0.1:5556
-w=3000
-m=192.168.10.0/24
-q
--community
-v=2

BTW: I'm using the ntopng binaries provided via https://packages.ntop.org/

[synfinatic]

So going back to the beginning... @level20peon tried to show that nprobe is sending different data than netflow2ng. But this is expected. For example, if you look at ntop_flow.h you'll see there are four different ways the MAC address can be specified:

#define IN_SRC_MAC 56
#define OUT_DST_MAC 57

#define IN_DST_MAC 80
#define OUT_SRC_MAC 81

And that makes sense since packets have a direction (in vs. out) and you have two addresses (src and dst). So the fact that two different netflow messages use different keys for the MAC addresses makes sense and if you look, both nprobe and netflow2ng are sending valid values.

That said, nothing I see in the json output of netflow2ng looks wrong or invalid. Digging through the ntopng code, I don't see anything obvious that would cause any issues and my testing has shown it works fine with both ntopng v5 and v6.

If you'd like me to investigate further, frankly I need pcaps of the netflow traffic being sent to netflow2ng and more information about how the netflow traffic being generated. @gitet mentioned using OpnSense which I assume is using softflowd? What version? what is the configuration/CLI arguments?

If it helps, here is how my pfSense box is running softflowd v1.0.0:

/usr/local/bin/softflowd -i 1:lagg0 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.pid -c /var/run/softflowd.lagg0.ctl
/usr/local/bin/softflowd -i 2:lagg0.200 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.200.pid -c /var/run/softflowd.lagg0.200.ctl
/usr/local/bin/softflowd -i 3:lagg0.400 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.400.pid -c /var/run/softflowd.lagg0.400.ctl
/usr/local/bin/softflowd -i 4:lagg0.300 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.300.pid -c /var/run/softflowd.lagg0.300.ctl
/usr/local/bin/softflowd -i 5:lagg0.500 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.500.pid -c /var/run/softflowd.lagg0.500.ctl
/usr/local/bin/softflowd -i 6:em3 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.em3.pid -c /var/run/softflowd.em3.ctl

Furthermore, you'll get debugging of ZMQ messages with ntopng at --verbose=5. you may also want to try not sniffing on wlp4s0 and just relying on ZMQ.

[gitet]

Hi @synfinatic - I dusted off and booted my old asus router which is running softflowd 1.0.0. Flow stats emitted from softflowd is received and ingested perfectly fine by my ntopng, so I may need to look closer at the stuff emitted by my opnsense box and raise an issue with the opnsense team. Btw: Opnsense is not running softflowd, they are instead using flowd version 0.9.1_5.

I've decided to just run ntopng on the opnsense box directly instead (via plugin). Works like a charm. If you're interested in a pcap containing datagrams from both of my routers, please drop me a message. If not, my part in this has come to and end for now. Sorry for leading you on a wild goose chase - I should not have assumed the problem was related to netflow2ng or ntopng.

Thanks!
BR/Espen

[ToqQrrl]

My problem is exactly the same … opensense … sorry I couldn’t comment earlier. So you guys think the problem comes from flowd on the opensense box ?

[gitet]

My problem is exactly the same … opensense … sorry I couldn’t comment earlier. So you guys think the problem comes from flowd on the opensense box ?

@ToqQrrl , this is my guess, but the logic behind it is not rock solid. I've tested the following variants (all using netflow v9):

• opnsense flowd -> raspberrypi running nfcapd : OK
• opnsense flowd -> linux running netflow2ng + ntopng : NOT OK
• asus router w/softflowd -> linux running netflow2ng + ntopng : OK
• ntopng running on same opnsense box : OK (no zmq or netflow2ng here, this is just local capture on the local nic, so not relevant for our discussion)