[backend] Backwards compatibility error with POD_NAMES v1 for v1 runs

[112358fn]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
  Using kubeflow/manifests
• KFP version:
  1.9.1 (but I think it impacts 1.10 too)
• KFP SDK version:
  1.8.22 and 2.8.0

Steps to reproduce

Setting POD_NAMES=v1 in the workflow controller resolves the "ML Metadata not found" issue in kubeflow/pipelines#11457.

However, this change introduces a bug that prevents users from retrying failed pipelines. The sequence is:

1. A user starts a run.
2. The pipeline fails due to one or more failing components.
3. The user clicks "Retry" in the web UI.
4. The pipeline gets stuck in a pending state, and no new pods are scheduled.

This happens because, if the pods associated with failed components are not deleted, the workflow controller logs:

level=info msg="Failed pod ... creation: already exists"

As a result, the Kubeflow pipeline remains stuck in the pending state.

Expected result

The failed pipeline is retried.

Materials and Reference

The reason for this bug is that:

• The GenerateRetryExecution function uses RetrievePodName to collect the list of pods to delete.
• When POD_NAMES=v1 is set, pod names match the ArgoWorkflows Node IDs.
• This causes deletion to fail, leaving pods behind and blocking pipeline retries.

Proposed Solution

To fix this, we should use the annotation workflows.argoproj.io/pod-name-format in the RetrievePodName function.
A similar solution is implement in the frontend in this PR: #11682

---

Impacted by this bug? Give it a 👍.

[juliusvonkohout]

Can you test this with Kubeflow manifests master which has KFP around 2.14 ?

[112358fn]

@juliusvonkohout Today I tested with the Kubeflow manifest from the master branch using kind and I could reproduce the same bug for pipelines v1.

[112358fn]

Today, I tested the manifest from the master branch. When POD_NAMES=v1 is set in the workflow_controller, both v1 and v2 pipelines get stuck if I try to retry them from the web UI.
Applying the changes from @kubeflow/pipelines/pull/12303 resolves this issue for both v1 and v2 pipelines.

[juliusvonkohout]

Today, I tested the manifest from the master branch. When POD_NAMES=v1 is set in the workflow_controller, both v1 and v2 pipelines get stuck if I try to retry them from the web UI. Applying the changes from @kubeflow/pipelines/pull/12303 resolves this issue for both v1 and v2 pipelines.

then this looks like an important regression.