Customizing container image of envoy

[rophy]

It seems the envoy image is hard-coded in source code:

cloud-provider-kind/pkg/loadbalancer/proxy.go

Line 24 in b6c63d0

const proxyImage = "envoyproxy/envoy:v1.30.1"

In air-gapped environments, we'll have private image repositories which will have different image name.

Workaround: pull the private image first, and locally re-tag it as "envoyproxy/envoy:v1.30.1".

[aojea]

thanks for reporting, is this a request for making this configurable or just to document it?

The code depends on envoy and is tested with an specific version, leaving this open will make easier for users to hit new bugs or problems because the image is different

[rophy]

Hope to make this configurable.

[BenTheElder]

@aojea we have this with the haproxy in kind as well, I think we should consider some format to override the host for these to point to a mirror, without allowing overriding the full image name.

Along with a command to dump the list of images required, so they can be mirrored.

[BenTheElder]

I think that will be less error prone than just enabling a full image override and then incompatible versions.

We should settle on an approach and ship it in kind as well.

As a counter point: the node images allow complete image override currently. But we have less well defined compatibility for the other images, and we've had issues with this for the node images as well.

[rophy]

IMHO providing default value is good enough to claim what is the compatible image

It's common that the environment simply cannot access dockerhub. Hard-coding the image name does not prevent compatibility issue, it only enforces that user must try to re-tag certain image to what is hard-coded in code.

[BenTheElder]

It's common that the environment simply cannot access dockerhub. Hard-coding the image name does not prevent compatibility issue, it only enforces that user must try to re-tag certain image to what is hard-coded in code.

providing a fully overridable value is very different from a support perspective versus "the user falsely tagged their own image to match and then is surprised when it didn't work"

FWIW: dockerhub images can be configured to point to a mirror in the short term https://docs.docker.com/docker-hub/mirror/#configure-the-docker-daemon

[aojea]

@BenTheElder do you have some proposal in mind?

[rophy]

Providing command to dump the list of images required is nice.

When we say air-gapped envs, it means no network connectivity with internet, only private registries with "pre-approved images" are allowed. The process to pull images into th registry is likely to involve re-tagging, which could also imply autmatically tagging with suffixes such as build id # or timestamp. So user may not always have the freedom to keep the image name identifal to what was tagged in dockerub.

[BenTheElder]

So user may not always have the freedom to keep the image name identifal to what was tagged in dockerub.

The digest however should be the same, 100% of the time, unless they're not actually using the same image, so we can just not use tags.

@BenTheElder do you have some proposal in mind?

#110 (comment)

Allow overriding the registry host (maybe an env) and add a command so the images to mirror can be identified (kubeadm has something similar to both of these)

[aojea]

this is what clayton did for the e2e framework kubernetes/kubernetes#93510

[BenTheElder]

I think maybe we should pick a common value (though not the one from the e2e tests) and respect it in kind as well for node images. We can parse image references and rewrite the host, maybe all the way up to the last path element and digest/tag.

We should consider any prior art and put a little bit of thought into exactly which portion is rewritten (e.g. in dockerhub you only have docker.io/$user/$image, but in registry.k8s.io and some other services you can have $host/$foo/$bar/$baz/$image, if we use registry.k8s.io in the future in kind then just rewriting docker.io/$kindest could possibly become like rewriting registry.k8s.io/kind/$version/$node:$digest. Something to think about a bit before committing to an API.

[BenTheElder]

In the meantime, just providing a tool to list the images should be enough, because the mirror user can mirror those, pull them and locally fix the tags if for some reason they're not going to make them match in the mirror (... I would really like to hear more about why mirroring necessitates altering the tags, but even if it must, you can alter the tag in local storage to fix it for now).

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten