added continuous retries to make sure deployment is scaled up

[shubhamrai1993]

With a keda based autoscaling setup.

Two loops run continuously -

• Prometheus recording every ~ 15s
  • pending pod status
  • traffic
• Keda scaling based on prometheus having the pending pod or traffic data - every 30s

Strategies

• Synchronise keda scaling down and elasti scaling up
  • How?
    • Keda must know that a service is in the middle of scale up by elasti, hence, keda will need to listen to a metric that is emitted from elasti. This would mean, elasti will have to implement a scaler through webhook which keda can listen to
  • Pros
    • We prevent scaling down from happening at all
  • Cons
    • Elasti becomes keda aware which is better avoided
• Add retries to elasti to keep scaling up until the request is fulfilled
  • How?
    • Add retry for elasti to keep scaling up the deployment every 5 seconds
  • Pros
    • Elasti and autoscaler remain decoupled
  • Cons
    • Initial scale down will happen by keda. Although the scale up happens as well by elasti

Remaining problem -

• Even after all this there are failure cases -
  • If prometheus is delays the recording of metrics for some reason, keda can still consider a service for scale down when the first request has been fulfilled by elasti, after which no retries happen

[debajyoti-truefoundry]

The above line looks incorrect, as there is no error handling at all.
Do we even need go here?