Allow modification of controller's replica index/type labels

[terrytangyuan]

Per comment in kubeflow/training-operator#1171 (comment), given that each operator may have different labels, we should discuss whether we want to allow each operator's controller that implements JobController to change ReplicaIndexLabel and ReplicaTypeLabel. For example, something like the following for tf-operator:

func (jc *JobController) GetReplicaTypeLabelKey() string {
	return tfReplicaTypeLabel
}

cc @gaocegege @Jeffwan @ChanYiLin @johnugeorge

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/operator	0.92
kind/feature	0.73

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[kf-label-bot-dev]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.73

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[Jeffwan]

I think we can have an interface there, and leave replica-type as default implementation in kubeflow/common, different operators can still override it if there's a need.

[terrytangyuan]

Yes they should be able to override the defaults.

[ChanYiLin]

I will take the issue
/assign @ChanYiLin