Clarify windows support for etcd images

[ahrtr]

What would you like to be added?

We start to use etcd officially released image starting from v3.6.5 in Kubernetes. But it doesn't have Windows flavour, and is breaking some Window-based conformance e2e test. Refer to kubernetes/kubernetes#134251 (comment)

Windows is just tier-3 support, see https://etcd.io/docs/v3.6/op-guide/supported-platform/. Currently we (etcd team) only build linux container images. Also Kubernetes doesn't support deploying the control plane on Windows.

$ docker manifest inspect gcr.io/etcd-development/etcd:v3.6.5
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3650,
         "digest": "sha256:28cf8781a30d69c2e3a969764548497a949a363840e1de34e014608162644778",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3650,
         "digest": "sha256:0f87957e19b97d01b2c70813ee5c4949f8674deac4a65f7167c4cd85f7f2941e",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3650,
         "digest": "sha256:2055881a1107b7e1bf7002b48544aed8b7da517d4d8e138c7b3b67abbf73a81d",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3650,
         "digest": "sha256:dfecf781891d331534930c82444272582feb768fb07b51a0cc1e51d7ebbbc170",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

Questions:

• Does the Windows based e2e test really make sense?
• If we revert the etcd: Update etcd to v3.6.5 kubernetes/kubernetes#134251, does it mean that we need to manually remove the etcd v3.6.5 image that previously published to registry.k8s.io/etcd, and rebuild them using the legacy cluster/images/etcd?

cc @ivanvc @joshjms @hakman @serathius @neolit123 @marosset @dims

Why is this needed?

.

[neolit123]

kubernetes/kubernetes#134426 (comment)

[joshjms]

my vote is to just tag them as LnuxOnly, but the bigger question is whether this inclusion of the windows binary resulted in ecosystem usage e.g. someone running etcd on distributed windows server instances for HA. that is not the same as running a windows control plane.

The etcd release process never did build windows images (because we release using linux machines). The windows binary is still available in our releases though and most likely they are using the binaries not the k8s.io images if they are indeed using windows.

my vote is to just tag them as LnuxOnly

i agree :D

[ahrtr]

most likely they are using the binaries not the k8s.io images if they are indeed using windows.

Based on comment kubernetes/kubernetes#134251 (comment), the test is using etcd Windows image. But please @marosset clarify anyway. Also are you comfortable to update the test to tag them linuxOnly?

[joshjms]

What I mean is that it's highly unlikely that anyone else that uses etcd in Windows uses the k8s.io built etcd images. If they are to extend the cluster for HA using a windows server, it's using etcd built binary for Windows. (hope I understood neolit's question correctly)

[marosset]

Based on comment kubernetes/kubernetes#134251 (comment), the test is using etcd Windows image. But please @marosset clarify anyway. Also are you comfortable to update the test to tag them linuxOnly?

Let me bring it up at the next sig-windows meeting (which is later today).

[ahrtr]

@liggitt 's comment kubernetes/kubernetes#134251 (comment)

Don't some conformance tests already require linux nodes be present? If the etcd portion of the aggregated server test pod is not supported on windows (and it is not, according to the etcd project), shouldn't we add an os selector or test tag to steer those pods to a linux node?

That seems like the correct fix to the failing e2e conformance test

[marosset]

@liggitt 's comment kubernetes/kubernetes#134251 (comment)

Don't some conformance tests already require linux nodes be present? If the etcd portion of the aggregated server test pod is not supported on windows (and it is not, according to the etcd project), shouldn't we add an os selector or test tag to steer those pods to a linux node?
That seems like the correct fix to the failing e2e conformance test

#20767

I want to check with @claudiubelu to see if he was aware of anyone running etcd as a Windows container or if we just added the image for test parity.

If we just added the image for test parity I think we can steer the etcd pod to a linux node (for the Windows test passes we taint the linux / control-plane node(s) so a toleration + an os selector added to the deployment in the test should suffice).

[joshjms]

Hi @marosset , how is the decision? Can we tag them as LinuxOnly? 😄

[marosset]

Hi @marosset , how is the decision? Can we tag them as LinuxOnly? 😄

Sorry, there are a few different threads for this.
Folks from sig-cluser-lifecycle commented that this test doesn't make sense to run on Windows so i think the LinuxOnly label is appropriate.

We are actively working towards enabling node_e2e tests on windows (kubernetes/kubernetes#129938) and those will also need to run etcd on Windows (although not in a container image AFAIK). It was expressed that it would be desirable to keep running the tests to ensure there aren't regressions in running etcd on Windows until the node e2e tests are running regularly.

If we can get some confirmation that any new issues that come up related to etcd and running the node_e2e tests on Windows would get addressed promptly then I am OK not running into this test on Windows.

/cc @zylxjtu

[ahrtr]

Thanks for the update.

Sorry, there are a few different threads for this.

This is the main thread for this issue. It needs both sig-etcd and sig-windows to collaborate. Please keep this ticket updated. thx

Folks from sig-cluser-lifecycle commented that this test doesn't make sense to run on Windows so i think the LinuxOnly label is appropriate.

So the action is to update the test to run it on LinuxOnly and resolve any related/new issues?

We also need to close kubernetes/kubernetes#134447 if there is no any objection.

[serathius]

cc @BenTheElder
What is the status of windows support in K8s?