GKE: missing authentication support from the poller

[afarbos]

When the poller call the scaler today there is no way (at least documented) to add authentication headers.
It would be nice to either add support, document alternative path to support this or remove the need for http calls.

[henrybell]

Hi @afarbos -- thanks for raising this. One possibility would be to use a service mesh as we discussed; although it would be possible to introduce auth to the autoscaler directly, it feels like a good use-case for a service mesh. I also raised issue #141 to track the investigation of a potential new deployment pattern that would combine the poller and scaler components and so remove the need for auth between the components. I will start looking at this and update #141 with any related new developments/findings/blockers/etc, but will leave this issue open also. Thanks!

[afarbos]

Agree with both. I think the latter would also make processes easier to observe since we would only have one long running service. But for this to work well, we should expose healthcheck verifying that the service is working as expected.

[henrybell]

I raised PR #142 to introduce a unified model for GKE deployment, in which the poller and scaler run as a single deployable unit, run by a single cron job. This should remove the need for poller to scaler auth if this pattern is used. I'll add an update to #137 related specifically to the points on observability. For now I will leave this issue open for any further discussion. Thanks!

[afarbos]

That make sense, I am wondering if that change the deployment pattern or add any limitations. for example, What is the max amount of spanner that can be scale within 1 minutes by this single cronjob?
Yes, I think it will make even more important to get good observability here. Thanks!