Support user‑specified initializers in TrainJob when runtime has none

[kramaranya]

What would you like to be added?
Shall we allow AI Practitioners to set dataset and model initializers directly on a TrainJob even when the selected TrainingRuntime or 'ClusterTrainingRuntime' has none?

This could be exposed via the Trainer SDK options - like WithInitializer() and WithDatasetInitializer(),WithModelInitializer().

This enables running only a data initializer, only a model initializer, or both, without creating multiple runtimes. But AI Practitioners will be responsible to set Volumes / volumeMounts, image etc.

cc @kubeflow/kubeflow-sdk-team @kubeflow/kubeflow-trainer-team

[kramaranya]

/kind discussion

[andreyvelich]

@kramaranya Please can we transfer this issue to the Trainer repo since it requires Controller and API changes if we agree with that?
Right now, there is not way to configure Volume spec, if we don't use PodTemplateOverrides API.

The idea is to allow users configure initializer in the TrainJob API without setting it in the Runtime template.

apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
spec:
  trainer:
    image: docker.io/torch-trainer
    numNodes: 2
  initializer:
    model:
      storageUri: hf://meta-llama/llama-3.2
  runtimeRef:
    apiGroup: trainer.kubeflow.org
    kind: ClusterTrainingRuntime
    name: torch-distributed
---
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
  name: torch-distributed
  labels:
    trainer.kubeflow.org/framework: torch
spec:
  mlPolicy:
    numNodes: 1
    torch:
      numProcPerNode: auto
  template:
    spec:
      replicatedJobs:
        - name: node
          template:
            metadata:
              labels:
                trainer.kubeflow.org/trainjob-ancestor-step: trainer
            spec:
              template:
                spec:
                  containers:
                    - name: node
                      image: pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime

[andreyvelich]

cc @akshaychitneni

[kramaranya]

Sure!
/transfer trainer