Is `3.2.1` a proper release?

[mateuszdrab]

Hey guys

Yesterday evening Fluxcd upgraded the Haproxy ingress controller and broke a number of ingresses, all of which started responding with a 502, so I had to roll back.

When I realized that, I came here to check for breaking changes and deprecations, noticing there's no release for version 3.2.1, just a tag. Latest release is 3.1.15.

Was that version supposed to be rolled out through Helm?
For now, I've rolled back and pinned the previous version.

Thanks

[oktalz]

hi @mateuszdrab,

yes, its released and official,
Helm was released yesterday, so that might be an issue,
you can check changes in https://github.com/haproxytech/helm-charts/commits/main/

for not having a release, its some weird Github issue actually, I have a patch for that in queue already.

[mateuszdrab]

Ahh, okay - I need to find out what happened then.
I just realized thanks to Blackbox Exporter that aside of the earlier mentioned issue, one of my ingress resources with basic authentication configured, simply opened itself out to the world, no longer asking for authentication with a 401 and returning a 200 instead.

I still can't find a changelog for 3.2.0 and above, could that be it? https://github.com/haproxytech/kubernetes-ingress/releases/tag/v3.2.0-rc1

[mateuszdrab]

I found some errors that appeared since the upgrade and vanished after a rollback, so I remediated by removing the objectset.rio.cattle.io annotations on the on the error files configmap. Why is it treating an annotation as an error code? I'm a bit confused.
handler/errorfiles.go:42 [transactionID=8d3c9d61-7272-4943-9223-4c60e6fb60ee] HTTP error code 'objectset.rio.cattle.io/applied' not supported

A new set of errors that was not seen before (with various frontends being mentioned):
[WARNING] (98) : frontend 'http' has no 'bind' directive. Please declare it as a backend if this was intended.

Lastly, I upgraded again and reproduced the ingress resource loading with no authentication enforced despite being configured - until I rolled back. The below annotations seem to be ignored.

    haproxy.org/auth-secret: cattle-monitoring-system/alertmanager-ingress-credentials
    haproxy.org/auth-type: basic-auth

This is a security risk, imagine if this went unnoticed? There's no errors or warnings about it.

A number of ingresses backed by externalname services fail to load with error 502, despite haproxy backend being healthy and L4 check being successful.

502 Bad Gateway
The server returned an invalid or incomplete response.

[hdurand0710]

b08d72b will fix the issue of annotation being treated as error code.
It will be included in the next 3.2 release.

[mateuszdrab]

b08d72b will fix the issue of annotation being treated as error code.
It will be included in the next 3.2 release.

Thanks, that's good news.

However, my main concern is that my basic-auth ingresses were left insecure, open to the world until I rolled back.

How can we determine why the annotations were ignored?

Nothing in the logs about it.

Same with the other set of ingresses, which were returning 502 until I rolled back.

[ivanmatmati]

Hi @mateuszdrab , there was a regression concerning annotation prefix. It is solved and the fix will be released. You can test it with master branch if you can build the Docker image or wait for the nightly build to be done (replace tag in your docker image with nighly)

[mateuszdrab]

Hi @mateuszdrab , there was a regression concerning annotation prefix. It is solved and the fix will be released. You can test it with master branch if you can build the Docker image or wait for the nightly build to be done (replace tag in your docker image with nighly)

Got it, I hope the buggy release was pulled since it would have a huge security impact.

Will attempt to test after the nightly build is done.

Thanks