Issue with Mikrotik RouterOS CHR & macvlan

[evilmonkey19]

Hi,

I found what i believe is an error between containerlab and RouterOS. I want to have a virtualized mikrotik (it is a container in VM I believe, a .vmdk) in this case and make accessible from my LAN. I have the following config:

name: mikrotik-lab

topology:
  nodes:
    R1:
      kind: mikrotik_ros
      image: vrnetlab/mikrotik_routeros:7.19.6
  links:
    - type: macvlan
      endpoint:
        node: R1
        interface: ether2
      host-interface: eth0
      mode: bridge

The issue is that the Macvlan is not created:

NETWORK ID     NAME               DRIVER    SCOPE
5b9f47a7ef89   bridge             bridge    local
8b17af01cedb   clab               bridge    local
4c94a9c5d630   ghost-in-da-edge   bridge    local
365eed732cc2   host               host      local
5676dc84d9aa   none               null      local

I have the NIC in promisc mode just in case:

2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether d8:cb:8a:14:0c:99 brd ff:ff:ff:ff:ff:ff
    altname enp4s0

To test this bug I used a Linux Ubuntu (baremetal PC, not a VM or similar).

The way to see what was happening I did a ping to the gateway and it didn't reply at all. I get the IP (DHCP broadcast is received) but then the ARP table and the MAC are not correct. I add a packet capture I did in the mikrotik just to show it.

Some other data that might be useful:

Also take into account that I can connect via Winbox using MAC Winbox but not IP. It works in the LAN perfect the MAC Winbox.

If you need any more tests let me know please.

[AAm-kun]

Having the same issue a little bit difference tho, instead of using physical interface i'm using a tagged interface.

# this does not work.
topology:
  nodes:
    r1:
      kind: mikrotik_ros
      image: vrnetlab/mikrotik_routeros:7.19.6

  links:
    - type: macvlan
      endpoint:
        node: r1
        interface: ether2
      host-interface: enp1s0.200
      mode: bridge

Mac is right in my case and in the packet capture I do see arp reply but mikrotik container is not getting it for some reason.
arp on gateway : 132 DC 192.168.30.2 0C:00:E2:25:75:01 vlan200 reachable

192.168.30.1 -- gateway
192.168.30.2 -- mikrotik container

Ping from gateway to mikrotik:

tcpdump -i enp1s0.200
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp1s0.200, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:08:17.520552 IP 192.168.30.1 > 192.168.30.2: ICMP echo request, id 43264, seq 43988, length 36
00:08:18.525189 IP 192.168.30.1 > 192.168.30.2: ICMP echo request, id 43264, seq 44244, length 36
00:08:19.508331 IP6 fe80::4a8f:5aff:fee6:5fad > fe80::e00:e2ff:fe25:7501: ICMP6, echo request, id 43264, seq 44500, length 16
00:08:19.533467 IP 192.168.30.1 > 192.168.30.2: ICMP echo request, id 43264, seq 44756, length 36
00:08:20.507629 IP6 fe80::4a8f:5aff:fee6:5fad > fe80::e00:e2ff:fe25:7501: ICMP6, echo request, id 43264, seq 45012, length 16
00:08:20.544994 IP 192.168.30.1 > 192.168.30.2: ICMP echo request, id 43264, seq 45268, length 36
00:08:21.512064 IP6 fe80::4a8f:5aff:fee6:5fad > fe80::e00:e2ff:fe25:7501: ICMP6, echo request, id 43264, seq 45524, length 16
00:08:21.557164 IP 192.168.30.1 > 192.168.30.2: ICMP echo request, id 43264, seq 45780, length 36
00:08:22.511455 IP6 fe80::4a8f:5aff:fee6:5fad > fe80::e00:e2ff:fe25:7501: ICMP6, echo request, id 43264, seq 46036, length 16
00:08:22.572225 IP 192.168.30.1 > 192.168.30.2: ICMP echo request, id 43264, seq 46292, length 36
00:08:23.517178 IP6 fe80::4a8f:5aff:fee6:5fad > fe80::e00:e2ff:fe25:7501: ICMP6, echo request, id 43264, seq 46548, length 16
00:08:23.571958 IP 192.168.30.1 > 192.168.30.2: ICMP echo request, id 43264, seq 46804, length 36
00:08:24.580138 IP 192.168.30.1 > 192.168.30.2: ICMP echo request, id 43264, seq 47316, length 36
00:08:25.589086 IP 192.168.30.1 > 192.168.30.2: ICMP echo request, id 43264, seq 47572, length 36

Ping from mikrotik to gateway:

tcpdump -i enp1s0.200
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp1s0.200, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:12:11.035708 IP 192.168.30.2 > 192.168.30.1: ICMP echo request, id 46080, seq 0, length 36
00:12:11.036081 IP 192.168.30.1 > 192.168.30.2: ICMP echo reply, id 46080, seq 0, length 36
00:12:12.037567 IP 192.168.30.2 > 192.168.30.1: ICMP echo request, id 46080, seq 256, length 36
00:12:12.037830 IP 192.168.30.1 > 192.168.30.2: ICMP echo reply, id 46080, seq 256, length 36
00:12:13.038947 IP 192.168.30.2 > 192.168.30.1: ICMP echo request, id 46080, seq 512, length 36
00:12:13.039191 IP 192.168.30.1 > 192.168.30.2: ICMP echo reply, id 46080, seq 512, length 36
00:12:14.040582 IP 192.168.30.2 > 192.168.30.1: ICMP echo request, id 46080, seq 768, length 36
00:12:14.041422 IP 192.168.30.1 > 192.168.30.2: ICMP echo reply, id 46080, seq 768, length 36
00:12:15.042117 IP 192.168.30.2 > 192.168.30.1: ICMP echo request, id 46080, seq 1024, length 36
00:12:15.042472 IP 192.168.30.1 > 192.168.30.2: ICMP echo reply, id 46080, seq 1024, length 36
00:12:16.043546 IP 192.168.30.2 > 192.168.30.1: ICMP echo request, id 46080, seq 1280, length 36
00:12:16.043967 IP 192.168.30.1 > 192.168.30.2: ICMP echo reply, id 46080, seq 1280, length 36
00:12:16.058869 ARP, Request who-has 192.168.30.1 tell 192.168.30.2, length 28
00:12:16.058989 ARP, Reply 192.168.30.1 is-at 48:8f:5a:e6:5f:ad (oui Unknown), length 42
00:12:16.108564 ARP, Request who-has 192.168.30.2 tell 192.168.30.1, length 42
00:12:16.710349 IP6 fe80::e00:e2ff:fe25:7501.5678 > ip6-allnodes.5678: UDP, length 144
00:12:16.710365 IP 192.168.30.2.5678 > 255.255.255.255.5678: UDP, length 144
00:12:16.710374 CDPv1, ttl: 121s, Device-ID 'r1', length 96
00:12:16.710387 LLDP, length 141: r1
00:12:17.044996 IP 192.168.30.2 > 192.168.30.1: ICMP echo request, id 46080, seq 1536, length 36
00:12:17.045310 IP 192.168.30.1 > 192.168.30.2: ICMP echo reply, id 46080, seq 1536, length 36
00:12:17.098989 ARP, Request who-has 192.168.30.1 tell 192.168.30.2, length 28
00:12:17.099223 ARP, Reply 192.168.30.1 is-at 48:8f:5a:e6:5f:ad (oui Unknown), length 42
00:12:17.148586 ARP, Request who-has 192.168.30.2 tell 192.168.30.1, length 42
00:12:18.046360 IP 192.168.30.2 > 192.168.30.1: ICMP echo request, id 46080, seq 1792, length 36
00:12:18.046573 IP 192.168.30.1 > 192.168.30.2: ICMP echo reply, id 46080, seq 1792, length 36
00:12:18.138886 ARP, Request who-has 192.168.30.1 tell 192.168.30.2, length 28
00:12:18.139018 ARP, Reply 192.168.30.1 is-at 48:8f:5a:e6:5f:ad (oui Unknown), length 42
00:12:18.188691 ARP, Request who-has 192.168.30.2 tell 192.168.30.1, length 42
00:12:19.048184 IP 192.168.30.2 > 192.168.30.1: ICMP echo request, id 46080, seq 2048, length 36
00:12:19.048552 IP 192.168.30.1 > 192.168.30.2: ICMP echo reply, id 46080, seq 2048, length 36
00:12:20.049453 ARP, Request who-has 192.168.30.1 tell 192.168.30.2, length 28
00:12:20.049684 ARP, Reply 192.168.30.1 is-at 48:8f:5a:e6:5f:ad (oui Unknown), length 42
00:12:21.098886 ARP, Request who-has 192.168.30.1 tell 192.168.30.2, length 28
00:12:21.099067 ARP, Reply 192.168.30.1 is-at 48:8f:5a:e6:5f:ad (oui Unknown), length 42
00:12:22.138968 ARP, Request who-has 192.168.30.1 tell 192.168.30.2, length 28
00:12:22.139371 ARP, Reply 192.168.30.1 is-at 48:8f:5a:e6:5f:ad (oui Unknown), length 42

Arp on mikrotik side:

[admin@r1] > ip arp/print 
Flags: D - DYNAMIC; C - COMPLETE
Columns: ADDRESS, MAC-ADDRESS, INTERFACE, STATUS
#    ADDRESS        MAC-ADDRESS        INTERFACE  STATUS   
0 D  192.168.30.1   48:8F:5A:E6:5F:AD  ether2     failed 

EDIT: making the mode mode: passthru solves for me.

# this works.
topology:
  nodes:
    r1:
      kind: mikrotik_ros
      image: vrnetlab/mikrotik_routeros:7.19.6

  links:
    - type: macvlan
      endpoint:
        node: r1
        interface: ether2
      host-interface: enp1s0.200
      mode: passthru

[hellt]

@evilmonkey19 check if passtru mode helps your case

  links:
    - type: macvlan
      endpoint:
        node: r1
        interface: ether2
      host-interface: enp1s0.200
      mode: passthru

also note that macvlan network won't be seen in docker network ls, because this is not a docker network that is created when you use the macvlan link, it is a direct macvlan link that is put into the container netns

[evilmonkey19]

In my case the passthru can help, but I would like to still have internet on the host. Also in this case i'm using only one mikrotik but i would like to have more router for further experiments. I want to have different vendors available in the LAN.

As I understand the macvlan is in the container and not in the host itself right?

I saw this for example:

# ip -d link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0  allmulti 0 minmtu 0 maxmtu 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 524280 tso_max_segs 65535 gro_max_size 65536 
2: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 9e:64:aa:de:31:59 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0  allmulti 0 minmtu 68 maxmtu 65535 
    veth addrgenmode eui64 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 tso_max_size 524280 tso_max_segs 65535 gro_max_size 65536 
3: br-mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether fe:3d:8b:d3:e6:5f brd ff:ff:ff:ff:ff:ff promiscuity 0  allmulti 0 minmtu 68 maxmtu 65535 
    bridge forward_delay 1500 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.fe:3d:8b:d3:e6:5f designated_root 8000.fe:3d:8b:d3:e6:5f root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  118.30 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0x4000 group_address 01:80:c2:00:00:00 mcast_snooping 1 no_linklocal_learn 0 mcast_vlan_snooping 0 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 
4: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-mgmt state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:3d:8b:d3:e6:5f brd ff:ff:ff:ff:ff:ff promiscuity 1  allmulti 1 minmtu 68 maxmtu 65521 
    tun type tap pi off vnet_hdr on persist off 
    bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.fe:3d:8b:d3:e6:5f designated_root 8000.fe:3d:8b:d3:e6:5f hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on bcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off locked off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 
5: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 7a:61:94:a3:20:c3 brd ff:ff:ff:ff:ff:ff promiscuity 0  allmulti 0 minmtu 68 maxmtu 65521 
    tun type tap pi off vnet_hdr on persist off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 
15: eth1@if2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether aa:c1:ab:d4:68:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 1  allmulti 0 minmtu 68 maxmtu 9256 
    macvlan mode bridge bcqueuelen 1000 usedbcqueuelen 1000 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 
    alias ether2
    altname ether2

In the eth1 i can see the macvlan part. To my understanding the macvlan should enable a direct interface in the LAN host simulating it is a different device right? Basically extending an ethernet (for example eth0) so instead of having one NIC we have two: the physical NIC from eth0 and the virtual NIC from ether2 (macvlan) right?

[AAm-kun]

if your personal router supports vlan untaging, you can do what I did, doesn't exactly explains what you are asking but solves the issue.

I have a single interface, my host gets internet using untag eth0 and then I split the eth0 into multiple vlan which I decap again in my router.

In ideal case, macvlan should take care all of these without going through all these, like you said, however, just a temporary fix for you issue.

[evilmonkey19]

As a temporary fix it works, but it's far from ideal... The issue is that then I need to change each time the router config for a new lab. Thanks for the temporary workaround @AAm-kun

[evilmonkey19]

Just in case it helps:

# ps aux | grep qemu    
root          25 17.5  0.8 647876 136556 pts/0   Sl+  16:36   0:22 qemu-system-x86_64 -enable-kvm -display none -machine pc -monitor tcp:0.0.0.0:4000,server,nowait -serial telnet:0.0.0.0:5000,server,nowait -m 256 -cpu host -smp 1 -drive if=ide,file=/chr-7.19.6-overlay.vmdk -boot n -device pci-bridge,chassis_nr=1,id=pci.1 -device pci-bridge,chassis_nr=2,id=pci.2 -device virtio-net,netdev=br-mgmt,mac=0C:00:1e:75:1b:00 -netdev bridge,br=br-mgmt,id=br-mgmt -device virtio-net,netdev=p01,mac=0C:00:17:79:56:01,bus=pci.1,addr=0x2 -netdev tap,id=p01,ifname=tap1,script=/etc/tc-tap-ifup,downscript=no