`whitelist`/`allow-list` works intermittently

[mateuszdrab]

Hello

After some recent improvements in my monitoring infrastructure, I've confirmed (an old concern that) my ingresses that have been annotated with a whitelist are intermittently available to non-whitelisted IP addresses.

The annotation I have tried using is:

haproxy.org/allow-list: "192.168.0.0/16, 10.0.0.0/8" (also tried whitelist)

Haproxy service and backing infrastructure are configured so that originating IP addresses for connections are correctly passed through all the way to the pod, so that each connection is logged with correct source IP.

Unfortunately, despite this configuration, I am observing intermittency in enforcement of the whitelist - when working correctly, I observe a 403 on requests from disallowed IPs, when not working, I get standard 200 OK with the content of the page. Tried with curl and blackbox monitor, and haproxy logs confirm that at the time the enforcement isn't working, source IPs are outside of the whitelisted range (initially wanted to rule out an internal network misconfiguration).

I have noticed that inside the generated haproxy.cfg file I can see the line (when grepping for 403):

http-request deny deny_status 403 if { var(txn.path_match) -m dom 5cecb88cc44beded7e19977ae5fdea99 } !{ src -f /etc/haproxy/maps/allowlist-6e90f8b0f4302b42ab5725b46cf65c5a.map }

This line, however, sometimes appears once and sometimes twice. When testing, most of the time it disappears from the https frontend generated config, resulting in https requests to my host not respecting the whitelist, whilst http requests do.

The content of the file /etc/haproxy/maps/allowlist-6e90f8b0f4302b42ab5725b46cf65c5a.map is

10.0.0.0/8
192.168.0.0/16

There is no pattern to this, sometimes the enforcement works for half an hour, then doesn't for 5 minutes, other times, it's the opposite.

This is on the latest version of the ingress controller.

Any ideas as to why this intermittency would happen?

I will try to run an instance of the controller in debug mode to see if I can get more insight on the issue.

Thanks

[hdurand0710]

Hi @mateuszdrab
When you say that it's an old concern, do you know when you started noticing this ? Was it since a specific version ? We will definitively look into this issue.

[mateuszdrab]

Hey @hdurand0710

Apologies, I'd love to be able to answer this, it would the be within the past 2 years. Before then, I was not using the whitelist annotation as this particular service was open 'to the wild'.

I remember a couple of occasions where I loaded up the URL from my phone, and it worked, I thought, that's bizarre, but then it would it would start showing 403s, so I thought it was temporary (or caching from when I was at home on LAN).

However, a few weeks ago I've instrumented blackbox exporter monitoring from outside of my networks, coming over public Internet. This made me realize the probes are flapping, triggering a deeper investigation yesterday.

I did packet captures, correlated logs from pfsense and haproxy, as I have 3 ingress paths (3 public ips in 3 different locations), and routing internally happens via metalLBs BGP announcements.

Some of the flapping is due to DNS round robin load balancing between those ingress IPs, and the particular instance of haproxy that is being connected to. I've then started pinning my curl requests to a single IP, ensuring requests land with a single instance (the haproxy service internally is configured with local externalTrafficPolicy) and discovered the correlation of the disappearing line in the config, resulting in the periods of time when traffic is passed through.

I currently believe this happens on the https backend only, however, my monitoring only touches https not http, I will set something up to monitor the http backend which does the redirect, but so far I've not had any issue with that one as far as I remember.