Docker containers on a host with multiple network interfaces

[shred86]

I run a small mini PC (Odyssey X86) home server running Debian 11 with several Docker containers to include Frigate, Home Assistant, etc. I also have my cameras isolated on a separate VLAN/subnet and what I've been doing is simply using a firewall rule to allow Frigate to access the cameras. While this works fine, it creates a lot of "unnecessary" overhead on the router with constant video streaming/recording being routed from one VLAN to another. One solution is to buy a switch with Layer 3 inter-VLAN routing, but that is really overkill for my home setup and they are fairly pricey for a PoE version. The Odyssey X86 mini PC has a second physical network interface connection, so I wanted to plug this into my switch such that the mini PC is connected to both networks, my "trusted" VLAN/network and camera VLAN/network. However, this is where the problem arises.

By default, Docker will expose your container to all interfaces on your host when you expose a port. For containers using the bridge networking option, you can get around this by specifying the host IP address when defining your ports. Using a docker-compose example of the ports section for Frigate, instead of:

    ports:
      - 5000:5000
      - 935:1935

You can specify (assuming 192.168.1.10 is your Docker host):

    ports:
      - 192.168.1.10:5000:5000
      - 192.168.1.10:1935:1935

Now all of the Docker containers using bridge networking will only be accessible on 192.168.1.10:XXXX. However, all of the containers running the host networking option are still accessible on both networks since there's no way to isolate which network interfaces those containers can use. Originally, I was going to try using the macvlan Docker networking option, but I didn't want to create a separate IP address for the few containers I have that currently use the host networking option. That's where I thought, why not just use the built in iptables firewall in Linux (Debian in my case). iptables is already installed with Debian and you can view the tables by using sudo iptables -L -v. Here's a great blog post that explains iptables. You will likely notice a bunch of entries for Chain FORWARD, Chain DOCKER, and so on, all of which are automatically created by Docker (this is how Docker manages the port forwarding under the hood). Unless you've messed with this before, you'll notice Chain INPUT is empty and all incoming (and outgoing) network connections are allowed by default.

First, you’ll want to add a rule to create a stateful firewall, meaning if a connection is initiated from somewhere that is allowed, it will allow packets to flow to and from that device to whatever device it's connected to on the network. In this case, I want devices on my "trusted" VLAN/network to be able to access the cameras on the camera VLAN/network, and when they do, I want the camera to be able to send back data/packets on that same established connection. This rule is what enables the latter - the cameras will be able to send back data/packets on the connection established by the device on my trusted VLAN/network.

sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Next we’ll want to block all inbound connections on a specific network interface which in this case is the network interface connected to the camera VLAN/network. First, find the network interface name by using ip a. The name will depend on your machine but my first network interface is labeled enp2s0 and the second is labeled enp3s0 which is the one that is connected to the same VLAN/network my cameras are on. So for me, I used the following:

sudo iptables -A INPUT -i enp3s0 -p all -j DROP

Now, there's one more place these same rules need to be added based on some rules Docker automatically created. Without going into too much of the details, there's another section of iptables called PREROUTING on the nat table which is checked before INPUT. These can be viewed using sudo iptables -vL -t nat and as you'll see, there's a rule under PREROUTING:

Chain PREROUTING (policy ACCEPT 67 packets, 10626 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   58  9287 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

This rule will match any packet that will be delivered "locally" on the host and forwarded to the DOCKER chain which can be viewed using sudo iptables -vL. Essentially, some traffic could bypass the rules we added above. To prevent this, the same rule we added above needs to be added to the DOCKER-USER chain which is intended for adding custom iptable rules for Docker (the Docker documentation explains this). Add the same two rules above (same reasoning) but to the DOCKER-USER chain:

sudo iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A DOCKER-USER -i enp3s0 -p all -j DROP

These changes won’t persist through a system restart but you can install iptables-persistent using sudo apt install iptables-persistent which will save your iptables to a file which is referenced on boot.

Anyways, just wanted to share this with anyone else out there looking for a similar solution. I've been slowly searching for a solution these past few months but it wasn't until tonight I realized iptables would work for this specific problem.

Edit: If you use WebRTC (e.g. WebRTC integration in Home Assistant to view your camera streams), you will have to allow all UDP ports (or at least the range you specified in the integration configuration). Note the slightly different syntax below which is adding this rule above the second rule we created above.

sudo iptables -I INPUT 2 -i enp3s0 -p udp -j ACCEPT

[ThreeFN]

I can confirm I was able to get this iptables config working on Unraid. Due to the slackware basis of Unraid you will need to use the User Scripts addon and set said script for 'at first array start only' to re-enable the iptable rules each boot. Also a reminder you may also need to adjust your router settings if you currently connect the docker to the camera subnet via your router. Worth using netdata or similar to monitor you frigate/camera-subnet/etc traffic to make sure your config is working.

my script:

#!/bin/bash
#This script configures iptables so that frigate accesses the security camera subnet 
#directly via an interface rather than via your router/server subnet
#inspired by frigate nvr discussion topic by shred86 at
#https://github.com/blakeblackshear/frigate/discussions/2524
#implimented by ThreeFN
#caveat emptor, I'm not a shell expert, no warranty implied, do at your own risk, assumption of knowledge, etc etc

CAMINTERFACE='YourInterfaceHere'

iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i $CAMINTERFACE -p all -j DROP
iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A DOCKER-USER -i $CAMINTERFACE -p all -j DROP
iptables -I INPUT 2 -i $CAMINTERFACE -p udp -j ACCEPT