Extend the PodTemplateOverrides structure with "manager" field

[mimowo]

What you would like to be added?

I would like to extend the PodSetOverrides for the optional "manager" field.

This is needed for the integration with Kueue done properly.

When Kueue starts a Job it is using the PodSetOverrides to set NodeLabels, Tolerations, based on the selected ResourceFlavor. Then, on suspending the Job Kueue needs to "revert" (remove) the overrides.

However, it is not clear from Kueue perspecive which overrides it did, and which were done by a user, or another controller. So, to overcome this we temporarily have a workaround by injecting the "kueue.x-k8s.io/override-idx" annotation which indicates the modified "override" by Kueue.

Why is this needed?

To make the interface between the Kueue and TrainJob easier to understand.

Probably other controllers or users who modify the "PodSetOverrides" could also benefit by annotating "who" owns the override.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[mimowo]

cc @tenzen-y @andreyvelich @astefanutti @kaisoz

This is also briefly discussed here: kubernetes-sigs/kueue#6865 (comment)

[mimowo]

Actually, now I wonder maybe this is not necessarily needed if we could introspect the field manager by the TrainJob .ManagedFields. This should generally keep the actor already, but I don't have much experience with that.

Also, embedding into spec would make it clear when debugging.

However, this is potentially another path worth exploring. cc @kaisoz

[mimowo]

x-referencing the issue at the Kueue side: kubernetes-sigs/kueue#7000

[astefanutti]

Actually, now I wonder maybe this is not necessarily needed if we could introspect the field manager by the TrainJob .ManagedFields. This should generally keep the actor already, but I don't have much experience with that.

@mimowo yes that would also be my intuition. The changes applied by Kueue should be available in the managed fields of the TrainJob.

Maybe helper functions like this one be helpful: https://pkg.go.dev/k8s.io/apimachinery/pkg/util/managedfields#ExtractInto

[tenzen-y]

AFAIK, in case of client-side update, the field could be overridden by another field manager w/o strict mode.
But in case of SSA, the field could not be overriddeden by another field manager w/o strict mode.

So, in case of both, the field could be updated and owner could be stolen by another one. I think that it will probably be problematic.

[mimowo]

Maybe it is not problematic that much. First of all it would mean buggy integration between two orchestrators. Second it seems preventable if the Override is immutable, then the second orchestrator sending Patch or SSA would be validated against the mutation.

[astefanutti]

First of all it would mean buggy integration between two orchestrators

I would also tend to consider it a bug.

[tenzen-y]

That's a good point. In that case, we have 2 approaches:

1. Introducing immutable restrictions to PodSpecOverride while TrainJob is not suspended.
2. Introduce managed field to each PodSpecOverride field like podSpecOverride[*].spec.fieldManager

I'm ok with either approach.

[kaisoz]

We found that the TrainJob admission webhook rejects TrainJobs with more than one PodSpecOverride targeting the same job. This effectively breaks the integration if the user supplied TrainJob already contains a podSpecOverride. I created an issue in the Trainer repo to discuss it: #2871

[kaisoz]

We found that the TrainJob admission webhook rejects TrainJobs with more than one PodSpecOverride targeting the same job. This effectively breaks the integration if the user supplied TrainJob already contains a podSpecOverride. I created an issue in the Trainer repo to discuss it: #2871

This has been solved and merged into master in #2880

[kaisoz]

Actually, now I wonder maybe this is not necessarily needed if we could introspect the field manager by the TrainJob .ManagedFields. This should generally keep the actor already, but I don't have much experience with that.

Also, embedding into spec would make it clear when debugging.

However, this is potentially another path worth exploring. cc @kaisoz

Unfortunately, we can't use the ManagedFields here because it only shows that Kueue modified the podSpecOverrides, not thet values it added:

- apiVersion: trainer.kubeflow.org/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kueue.x-k8s.io/trainjob-override-idx: {}
      f:spec:
        f:podSpecOverrides: {}
        f:suspend: {}
    manager: kueue
    operation: Update
    time: "2025-10-02T13:34:49Z"

So it seems we'll need to go with the "manager" field in the PodSpecOverrides. I'll take care of this if there's no objection

/assign