pants packaging a docker image using a kubernetes buildx driver (and no docker daemon)

[gdfast]

These are lessons from troubleshooting the issue discussed here: https://pantsbuild.slack.com/archives/C046T6T9U/p1751926012583459

tl;dr

To use a kubernetes buildx driver

• pass along KUBERNETES_SERVICE_[HOST|PORT|PORT_HTTPS] env_vars to the [docker] backend config of pants.toml
• consider explicitly setting buildx driver and namespace in the same env_vars config, like BUILDX_BUILDER=<your k8s buildx driver name>, BUILDKIT_NAMESPACE=<your namespace>
• on the docker_image target, set output={"type": "registry"}
• don't run pants publish on that target because pants package will upload to the registry

Premise

The ability of pants to optimally build docker images with pex files is pretty compelling (see this blog). This is made even better with buildx support in the pants docker backend, which enables multi-platform docker image builds.

Normally, with docker desktop installed, pants package and pants publish of a docker_image target using buildx "just works". package will build the image and store it locally, publish can push it up to a registry.

However, I recently hit an issue moving the pants commands that worked on my laptop to a (Gitlab) CI job had some tricky failures to overcome.

More about the CI environment

For our CI jobs, our workers run a Docker image that has the docker CLI installed. There have long been security concerns and complications running docker in docker (dind). For that reason, the team in charge of CI infrastructure at my company does not have a docker daemon running on the CI container (no socket). Instead, there's a kubernetes cluster of BuildKit workers available for building the images triggered by a docker buildx build command.

On my laptop, I have the following build backend

% docker buildx ls 
NAME/NODE           DRIVER/ENDPOINT     STATUS    BUILDKIT   PLATFORMS
default             docker                                   
 \_ default          \_ default         running   v0.23.2    linux/amd64 (+2), linux/arm64, linux/ppc64le, linux/s390x, (2 more)
desktop-linux*      docker                                   
 \_ desktop-linux    \_ desktop-linux   running   v0.23.2    linux/amd64 (+2), linux/arm64, linux/ppc64le, linux/s390x, (2 more)

In CI, we instead have

docker buildx ls
NAME/NODE                     DRIVER/ENDPOINT                                                       STATUS    BUILDKIT   PLATFORMS
ci*                           kubernetes                                                                                 
 \_ buildkit-platform-amd64    \_ kubernetes:///ci?deployment=buildkit-platform-amd64&kubeconfig=   running   v0.21.1    linux/amd64 (+4), linux/386
 \_ buildkit-platform-arm64    \_ kubernetes:///ci?deployment=buildkit-platform-arm64&kubeconfig=   running   v0.21.1    linux/arm64, linux/arm (+2)
default                                                                                             error
Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Notice "default" status is "error" and that last line... there's no default docker daemon

Overcoming pants package failure 1 – "cannot determine Kubernetes namespace"

On buildx version v0.19.2 I got

ERROR: no valid drivers found: cannot determine Kubernetes namespace, specify manually: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

And on the older buildx v0.11.1 I got

ERROR: no valid drivers found: cannot determine Kubernetes namespace, specify manually: stat /root/.kube/config: no such file or directory

There's no kubeconfig file because the CI worker pod is in-cluster, which means it should get its namespace and cluster information from the kubernetes server. To get this to work for docker buildx build launched by pants, I had to pass the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment vars to the docker backend's env_vars. With that info exposed, the docker buildx build executed by pants was able to pull the info it needed to use the ci buildx kubernetes driver.

Overcoming pants package failure 2 - "failed to copy to tar"

Next,pants package <docker_image target> almost succeeded in building the image, but failed with the following error:

#7 ERROR: failed to copy to tar: rpc error: code = Unknown desc = io: read/write on closed pipe

Inspecting with pants -ldebug, we see that pants is running the following command

/usr/bin/docker buildx build \
--platform=linux/amd64 \
--output=type=docker \
--pull=False \
--file ...

type=docker cannot work as an output type for remote docker builds when there's no local docker engine running that can save the image. The solution was to set --output=type=registry on the docker_image.

❗ This means there's no need for a pants publish step: pants publish pushes up an image from the local engine to a registry, but the output of pants package on the docker_image target already was that registry.

Putting it all together (working in pants v2.27.0)

pants.[ci.]toml

[docker]
use_buildx = true
env_vars = [
    "BUILDX_BUILDER=ci",
    "BUILDKIT_NAMESPACE=buildkit-platform",
    "DOCKER_CONFIG=%(homedir)s/.docker",
    "KUBERNETES_SERVICE_HOST",
    "KUBERNETES_SERVICE_PORT",
    "KUBERNETES_SERVICE_PORT_HTTPS",
    ...
]
build_verbose = true

BUILD target

docker_image(
    output={"type": "registry"},
    ...