[Housekeeping] Flyte mutli-cluster network issues on k8s submission - kube-api-client - allow retries

[punkerpunker]

Describe the issue

When running Flyte in a multi-cluster environment (separate flyte-controlplane, separate flyte-dataplane), when FlyteAdmin is trying to create a FlyteWorkflow CRD in a dataplane cluster, sometimes it could encounter errors, e.g:

failed to create workflow in propeller Post "https://kube-apiserver.<domain>/apis/flyte.lyft.com/v1alpha1/namespaces/ft-production/flyteworkflows?timeout=30s": net/http: TLS handshake timeout

It happens when K8sWorkflowExecutor is sending a POST to the kube-apiserver of the dataplane cluster:

flyte/flyteadmin/pkg/workflowengine/impl/k8s_executor.go

Lines 78 to 84 in 3b621c1

	_, err = targetCluster.FlyteClient.FlyteworkflowV1alpha1().FlyteWorkflows(data.Namespace).Create(ctx, flyteWf, v1.CreateOptions{})
	if err != nil {
	if !k8_api_err.IsAlreadyExists(err) {
	logger.Debugf(context.TODO(), "Failed to create execution [%+v] in cluster: %s", data.ExecutionID, targetCluster.ID)
	return interfaces.ExecutionResponse{}, errors.NewFlyteAdminErrorf(codes.Internal, "failed to create workflow in propeller %v", err)
	}
	}

And the workflow appears to have failed from the very first attempt, without a good mechanism to retry submission (other than recovering the job manually).

I guess in a multi-cluster setup, network issues are somehow inevitable, so it would be appreciated to have some retry mechanism in there. I saw there's a very comprehensive retry handler for cases where execution node failures occur, but this is happening before that, and there's no retry machinery.

I am happy to implement myself and open a PR, just wanted to hear opinions on that matter. Thanks folks!

What if we do not do this?

Flyte multi-cluster setups would suffer badly from some intermittent issues (esp. related to network)

Related component(s)

flyteadmin
flytepropeller

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

It seems likely that Flyte's current K8sWorkflowExecutor does not implement retries for network errors when submitting a FlyteWorkflow CRD to a remote Kubernetes API server in a multi-cluster setup. The code at the referenced location simply attempts the POST once and returns an error if it fails, with no built-in retry logic for transient network issues such as TLS handshake timeouts. This means that, as you observed, network blips can cause workflow creation to fail immediately, requiring manual recovery. There is no mention in the available data sources of a retry mechanism at this early submission stage, though retries are handled for later execution/node failures (source).

My understanding is that adding a retry mechanism for network-related errors at the CRD submission step would improve Flyte's resilience in multi-cluster environments, and this gap is not currently addressed in the official codebase or documentation. Would you like more detail on how such a retry mechanism could be implemented or on related retry strategies in Flyte?

Sources:

• flyteadmin/pkg/workflowengine/impl/k8s_executor.go

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[Sovietaced]

This seems reasonable. The other APIs you may be referring to are gRPC APIs and gRPC has much better native support for handling retries. The k8s APIs are REST based so retry support typically nonexistent or fragmented.

[kumare3]

We prefer implementing retry for workflow submission outside of Flyte - this is why we have user defined execution names. If we automatically retry then we need a way to store, manage state and retry. We need anti entropy and other mechanisms.

2 reasons we did not implement this

1. This is as much work to do it right. As to implement propeller
2. Increases latency significantly for submissions

In v2 we have a different architecture.

Adding this stuff definitely had the chances of making the code base more complex

Happy to hop on a call to understand more in detail -
Please join the slack

[kumare3]

Cc @EngHabu