race between weave-kube and kube-proxy can allow all traffic through Service VIP

[brb]

kube-proxy pre-appends the iptables rule "-j KUBE-FORWARD" which ACCEPTs all traffic and prevents it entering the "WEAVE-NPC" chain.

In #3210 we introduced a fix which pre-appends "-j WEAVE-NPC" after kube-proxy has inserted "-j KUBE-FORWARD". The fix relies on a premise that weave-kube is started after kube-proxy which follows from a fact that weave-kube depends on api-server (to get a peer list) and api-server is accessible to weave-kube only after kube-proxy has inserted all nat rules.

However, if the nat rules for api-server are present (e.g. from previous k8s installation which failed to flush them), then weave-kube can start before kube-proxy, and thus the WEAVE-NPC rule will be preceded by the KUBE-FORWARD => all traffic will be enabled to Pods through Service Virtual IP.

(Maybe) possible fixes to the problem:

• Try creating a dummy Service before starting weaver in weave-kube/launch.sh to ensure that kube-proxy is running.
• Monitor iptables rules and ensure the required order.
• Wait for "KUBE-FORWARD" chain to appear before inserting "WEAVE-NPC" (requires checking k8s vsn and kube-proxy backend).
• Wait for an answer to Kube-proxy adds forward rules to ensure NodePorts work kubernetes/kubernetes#52569 (comment).

[brb]

Monitoring and maintaining iptables rules would also solve such issues as #3155, #3106.

[murali-reddy]

@brb is this still a problem? I would like to take a shot at this.

[brb]

@murali-reddy I think this is still a problem. I'd be interested in the "Monitor (and maintain) iptables rules and ensure the required order" solution, as it could be used to maintain other iptables rules installed by Net.

Before implementing it, perhaps you could present possible ideas.

[murali-reddy]

One pattern i have seen with Kubernetes controllers is that idea of periodic reconciliation loop which ensures actual state is in sync with desired state. Such reconciliation even can fix out-of-band changes like iptables restart etc and issues as mentioned in #3155, #3106. But that is more significant work but holistic solution IMO. I still don't have good clarity how much of the code (both net and npc) is idempotent.

I will look for other alternatives as well.

I'd be interested in the "Monitor (and maintain) iptables rules and ensure the required order" solution

By Monitor you mean, read the iptables once in a while and check for the order, or is there a callback kind of functionality when there is a change to chain?

[brb]

By Monitor you mean, read the iptables once in a while and check for the order, or is there a callback kind of functionality when there is a change to chain?

I meant what you just described above, i.e. the way k8s does.

[bzillins]

Still a problem, currently worked around by modifying the kube-proxy service to remove weave iptables rules in ExecStartPre and remove weave containers in ExecStartPost for now.