[bug] Opening channel error log from operator

[gamino-brex]

Describe the bug
Increase of error logs on controller-manager like the following:

ssh: rejected: connect failed (error dialing local listener tcp://localhost:8080: dial tcp [::1]:8080: connect: connection refused)

or

ssh: unexpected packet in response to channel open: <nil>

Full logs export was provided through the Slack shared channel.

To Reproduce
Steps to reproduce the behavior (including version, environment)

• Operator v0.15.0
• No explicit actions/steps known to reproduce

Expected behavior
No error logs on controller-manager unless there is user-facing impact and action required for administrators (those who host Signadot operator)

Observed behavior
Bunch of error logs that are hard to understand what they mean, what they are related to, and if should require action from administrators

[scott-cotton]

@gamino-brex these logs do not appear to be part of the controller-manager, which I think is a source of confusion.

The error ssh: rejected: connect failed (error dialing local listener tcp://localhost:8080: dial tcp [::1]:8080: connect: connection refused) occurs in the context of running signadot local connect with a local sandbox: A request is routed to a local workload, but that that local workload is not listening. This error message is generated by the signadot cli when running a local workload and transmitted to the tunnel-proxy deployment.

The error unexpected packet in response to channel open is an error generated by https://pkg.go.dev/golang.org/x/crypto/ssh when trying to establish an ssh channel with a workstation. This looks like a situation where the local workstation went offline and so did not participate in the channel setup as expected by the server side on the tunnel-proxy deployment. This again is triggered when exercising a local workload.

I think this error may also be generated by an overloaded tunnel-proxy, but in light of the first error, I'd guess not.

So, it seems to me like you have generation of traffic to local workloads while they are not being served. We've logged these as errors in light of the use case where one expects to generate the traffic to local workloads only when they are being served. However, from previous discussion I am not sure this is the case for you.

On our end, it would help to know the operator component when discussing what to do with logs.

Does this information help?