feat: add nsq scaler

[Ulminator]

This change adds a new scaler for NSQ, enabling auto-scaling based on topic/channel depth. The main components of NSQ that the scaler interacts with are nsqlookupd and nsqd. First the scaler makes /lookup requests to the nsqlookupd instances in the cluster to determine which nsqd instances have messages for the given topic. Then for each nsqd instance the scaler makes /stats requests to and aggregates the results to determine a final depth value.

NSQ consumers create the channel they are reading if they do not exist. As such, if the channel does not exist on an nsqd host, the topic depth is used. The consumer pods that KEDA would be responsible in spawning would then create the channel.

Checklist

• When introducing a new scaler, I agree with the scaling governance policy
• I have verified that my change is according to the deprecations & breaking changes policy
• Tests have been added
• Changelog has been updated and is aligned with our changelog requirements
• A PR is opened to update the documentation on (repo) (if applicable)
• Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #3281

Relates to:

• feat: add nsq e2e image test-tools#175
• fix: adds sleep flag to nsq e2e test image test-tools#184
• feat: nsq scaler docs keda-docs#1485

[semgrep-app]

Semgrep found 7 no-direct-write-to-responsewriter findings:

• pkg/scalers/nsq_scaler_test.go
  • L160 - Triage
  • L169 - Triage
  • L315 - Triage
  • L394 - Triage
  • L463 - Triage
  • L575 - Triage
  • L636 - Triage

Detected directly writing or similar in 'http.ResponseWriter.write()'. This bypasses HTML escaping that prevents cross-site scripting vulnerabilities. Instead, use the 'html/template' package and render data using 'template.Execute()'.

_{Ignore this finding from no-direct-write-to-responsewriter.}

Semgrep found 20 sprintf-host-port findings:

• pkg/scalers/nsq_scaler_test.go
  • L176 - Triage
  • L176 - Triage
  • L176 - Triage
  • L322 - Triage
  • L322 - Triage
  • L402 - Triage
  • L402 - Triage
  • L402 - Triage
  • L470 - Triage
  • L470 - Triage
  • 7 more - Triage
• pkg/scalers/nsq_scaler.go
  • L188 - Triage
  • L188 - Triage
  • L188 - Triage

use net.JoinHostPort instead of fmt.Sprintf(parsedURL.Port(), parsedURL.Hostname())

_{Ignore this finding from sprintf-host-port.}

[semgrep-app]

Semgrep found 7 no-fprintf-to-responsewriter findings:

• pkg/scalers/nsq_scaler_test.go
  • L160 - Triage
  • L169 - Triage
  • L315 - Triage
  • L394 - Triage
  • L463 - Triage
  • L576 - Triage
  • L637 - Triage

Detected 'Fprintf' or similar writing to 'http.ResponseWriter'. This bypasses HTML escaping that prevents cross-site scripting vulnerabilities. Instead, use the 'html/template' package to render data to users.

_{Ignore this finding from no-fprintf-to-responsewriter.}

[Ulminator]

@JorTurFer thanks for merging in kedacore/test-tools#175! I updated the code here to utilize the image that can be created from that PR.

I also wanted to let you know that I have created the corresponding docs PR as well here: kedacore/keda-docs#1485

The last remaining TODO's seem to just be getting these CI checks to pass. Is it safe to ignore the remaining semgrep checks since they just relate to tests? I initially got flagged for no-direct-write-to-responsewriter and now I'm getting no-fprintf-to-responsewriter, yet I do see both of these patterns being used in other test files.

[zroubalik]

/run-e2e nsq
Update: You can check the progress here

[Ulminator]

I see the e2e tests have failed, but I'm wondering if the e2e test image (ghcr.io/kedacore/tests-nsq:latest) was created? 🤔 I don't see it here. I'm thinking the issue was an image pull error due to it not existing yet.

# Local run w output snippet from describing a consumer pod
  Warning  Failed     9s    kubelet            Failed to pull image "ghcr.io/kedacore/tests-nsq:latest": failed to pull and unpack image "ghcr.io/kedacore/tests-nsq:latest": failed to resolve reference "ghcr.io/kedacore/tests-nsq:latest": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Akedacore%2Ftests-nsq%3Apull&service=ghcr.io: 401 Unauthorized
  Warning  Failed     9s    kubelet            Error: ErrImagePull
  Normal   BackOff    8s    kubelet            Back-off pulling image "ghcr.io/kedacore/tests-nsq:latest"
  Warning  Failed     8s    kubelet            Error: ImagePullBackOff

I also noticed I had only updated one of the refs of my custom image to ghcr.io/kedacore/tests-nsq:latest so I went ahead and made that fix.

[JorTurFer]

The image was created but GH creates them as private by default, now it's public

[JorTurFer]

/run-e2e nsq
Update: You can check the progress here

[JorTurFer]

Do we need to update the policy to include any extra permission? https://github.com/kedacore/testing-infrastructure/blob/main/terraform/modules/aws/iam/main.tf#L88-L96

If yes, could you open a PR there?

[Ulminator]

Do we need to update the policy to include any extra permission? https://github.com/kedacore/testing-infrastructure/blob/main/terraform/modules/aws/iam/main.tf#L88-L96

If yes, could you open a PR there?

No additional permissions are required 👍

[Ulminator]

@JorTurFer just wanted to check in and see if you need anything else before merging this in?

[JorTurFer]

Looking good. I've kept some comments inline.
I'm worried about the lifecycle of the project, I closed the issue with the feature request a year and a half ago because there wasn't activity, and it looked as no longer maintained.
@zroubalik @wozniakjan

[Ulminator]

@JorTurFer thanks for taking a look! I believe I addressed all of your comments, but let me know if anything else needs to change of course.

I'm worried about the lifecycle of the project, I closed the issue with the feature request a year and a half ago because there wasn't activity, and it looked as no longer maintained.

I'm an engineer at Bitly, which is where NSQ was initially developed. We use it extensively throughout our stack, and while updates to the project are not as frequent as they were in the past, it is stable. There are plans for updates in the near future and it is under active development.

[JorTurFer]

/run-e2e nsq
Update: You can check the progress here

[JorTurFer]

LGTM! Awesome job! 🙇

[JorTurFer]

@zroubalik @wozniakjan PTAL

[Ulminator]

@JorTurFer thanks for approving the PR and for your prompt feedback!

[JorTurFer]

/run-e2e nsq
Update: You can check the progress here

[wozniakjan]

lgtm, thank you!

not sure what is wrong with Semgrep / Analyze Semgrep action, it doesn't allow me to see anything in there

[Ulminator]

Thanks for the review @wozniakjan !

@JorTurFer, I have updated this PR along with kedacore/docs#1485 to account for the 2.16 release that happened last week, PTAL!