Retrying compartmentless containers #1608
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
istio-testing
added
the
size/L
MikeZappa87
reviewed
Aug 18, 2025
| @@ -339,6 +362,98 @@ impl WorkloadProxyManagerState { | |||
| } | |||
| } | |||
|
|
|||
| pub async fn retry_comparmentless(&mut self, poddata: &WorkloadData) -> Result<(), Error> { | |||
keithmattix
reviewed
Aug 18, 2025
| @@ -34,8 +34,7 @@ mod workloadmanager; | |||
| #[cfg(any(test, feature = "testing"))] | |||
| pub mod test_helpers; | |||
|
|
|||
|
|
|||
| #[derive(Debug)] | |||
| #[derive(Debug, Clone)] | |||
There was a problem hiding this comment.
Are we sure we want to clone this?
Comment on lines
+65
to
+66
| pub fn proxy_pending(&self, uid: &crate::inpod::WorkloadUid, workload_info: &WorkloadInfo) { | ||
| let mut state = self.state.write().unwrap(); |
There was a problem hiding this comment.
Does it make sense to merge these?
| @@ -36,14 +34,14 @@ impl InPodConfig { | |||
| ..cfg.socket_config | |||
| }; | |||
| Ok(InPodConfig { | |||
| cur_namespace: InpodNamespace::current()?, | |||
| cur_namespace: NetworkNamespace::current()?, | |||
There was a problem hiding this comment.
+1, the current naming makes no distiction between the specific namespace. Does this rename add value?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
We have an issue with containerd and Windows' HCS where namespaces for pods don't yet have a compartment ID when the CNI signals ztunnel for a new workload that's just been created. Without that ID we can't create ztunnel proxy sockets inside the pod's network compartment. The compartment ID will only be available after ztunnel signals the CNI that the workload has been assimilated by ztunnel (which can't happen without a compartment ID 🫤).
What this PR does
This PR mitigates the stated problem with ztunnel replying an ACK to the CNI during the ADD operation, even though the workload proxies haven't been yet instantiated inside ztunnel. After a timeout, ztunnel tries again to add the workflow, and now the HCS API returns a valid compartment ID, which allows the creation of a proxy without any problems.
A more detailed flow looks like:
2.1 If the compartment ID is available, we create a proxy for the workload and ACK the CNI
2.2 If the compartment ID is not available, we mark the workload as pending inside ztunnel and ACK the CNI.
How the PR does it
It introduces a queue of events that runs in the same thread where ZDS commands are processed. These events are managed in a quite simple way at the moment, but can be expanded in the future for more advanced handling of future "internal events", including the current retries of pending workloads.
Caveats
We'll keep retrying compartmentless workloads even if they're failing in an unretriable way (to be fixed in a different PR). There's no current way to signal CNI after sending an ACK for the ADD command that the workload actually failed to be assimilated by ztunnel.