keda-operator-metrics-apiserver /metrics endpoint connection refused

[trampfox]

• Our Prometheus instance is getting a connection refused error while trying to scrape the metrics from the /metrics endpoint after a clean installation (using the default values) via Helm Chart version 2.11.1.
• A Prometheus operator ServiceMonitor is enabled for the Keda Metrics Server
• The keda-operator-metrics-apiserver is deployed and is running without any problems, except for the /metrics endpoint where we get a connection refused error

Expected behavior

Keda Metrics Server service monitor targets are all up and Prometheus is able to scrape the metrics from keda-operator-metrics-apiserver instances.

Actual Behavior

Prometheus is not able to scrape the metrics from the /metrics endpoint of due to the following error

Get "http://10.24.1.112:8080/metrics": dial tcp 10.24.1.112:8080: connect: connection refused

We have also tried to get the metrics by enabling a port-forwarding to the 8080 port of the keda-operator-metrics-apiserver and we got the same connection refused error

$ kubectl port-forward keda-operator-metrics-apiserver-57586d6c44-vtm5s 8080:8080

and when we call the /metrics endpoint we see the following error

E0720 11:05:39.957032   60120 portforward.go:400] an error occurred forwarding 8080 -> 8080: error forwarding port 8080 to pod c7354feb5414f5e29681e7d45354b5b0424053ab52eddead40b18908b9f0773e, uid : failed to execute portforward in network namespace "/var/run/netns/cni-c62a7094-5ffa-c294-3107-6fac86c49f95": failed to connect to localhost:8080 inside namespace "c7354feb5414f5e29681e7d45354b5b0424053ab52eddead40b18908b9f0773e", IPv4: dial tcp4 127.0.0.1:8080: connect: connection refused IPv6 dial tcp6 [::1]:8080: connect: connection refused

Steps to Reproduce the Problem

1. Install Keda via helm chart version 2.11.1

Specifications

• KEDA Version: 2.11.1
• Kubernetes Version: 1.24
• Scaler(s): N/A

[cskinfill]

I'm able to connect when I port-forward, but I get an empty reply from the endpoint.

$ kubectl -n keda port-forward keda-operator-metrics-apiserver-5bb88966d5-6v2mk metrics
Forwarding from 127.0.0.1:9022 -> 9022
Forwarding from [::1]:9022 -> 9022
Handling connection for 9022
Handling connection for 9022
Handling connection for 9022

when I try to use curl, I get an empty reply.

$ curl -v http://localhost:9022/metrics
*   Trying 127.0.0.1:9022...
* Connected to localhost (127.0.0.1) port 9022 (#0)
> GET /metrics HTTP/1.1
> Host: localhost:9022
> User-Agent: curl/7.88.1
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

[hontarenko]

I also have curl: (52) Empty reply from server

[pragmaticivan]

Check this release: https://github.com/kedacore/keda/releases/tag/v2.11.2

[trampfox]

It works now. Many thanks!

[pragmaticivan]

You can probably close this issue since it was fixed in v2.11.2 @trampfox