Failed to append node, Get http://docker.example.com context deadline exceeded #391

longngn · 2024-12-16T10:08:12Z

Contributing guidelines

I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

The documentation does not mention anything about my problem
There are no open or closed issues that are related to my problem

Description

docker/setup-buildx-action@v3 fails when append Buildx builder node

Logs:

Run docker/setup-buildx-action@v3
  with:
    append: - endpoint: ssh://root@docker-builder-arm
    platforms: linux/arm64
  - endpoint: ssh://root@testnet-preprod
    platforms: linux/amd64
  
    driver: docker-container
    install: false
    use: true
    cache-binary: true
    cleanup: true
Docker info
  /usr/bin/docker version
  Client: Docker Engine - Community
   Version:           [2](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:2)6.1.3
   API version:       1.45
   Go version:        go1.21.10
   Git commit:        b72abbb
   Built:             Thu May 16 08:[3](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:3)3:35 2024
   OS/Arch:           linux/amd64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          26.1.3
    API version:      1.[4](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:4)5 (minimum version 1.24)
    Go version:       go1.21.10
    Git commit:       8e96db1
    Built:            Thu May 16 08:33:3[5](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:5) 2024
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          1.7.24
    GitCommit:        88bf19b2105c8b175[6](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:6)0993bee28a01ddc2f97182
   runc:
    Version:          1.2.2
    GitCommit:        v1.2.2-0-g[7](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:7)cb3632
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  /usr/bin/docker info
  Client: Docker Engine - Community
   Version:    26.1.3
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.19.2
      Path:     /usr/libexec/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.27.1
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 0
   Server Version: 26.1.3
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: systemd
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: [8](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:8)8bf19b2105c8b17560993bee28a01ddc2f[9](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:9)7182
   runc version: v1.2.2-0-g7cb3632
   init version: de40ad0
   Security Options:
    apparmor
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.8.0-[10](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:10)17-azure
   Operating System: Ubuntu 24.04.1 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 2
   Total Memory: 7.753GiB
   Name: fv-az814-629
   ID: 50b3d87a-e452-42cc-a596-ad59e86f2b5d
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Experimental: false
   Insecure Registries:
    [12](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:12)7.0.0.0/8
   Live Restore Enabled: false
  
Buildx version
  /usr/bin/docker buildx version
  github.com/docker/buildx v0.19.2 1fc5647dc281ca3c2ad5b451aeff2dce84f1dc49
Inspecting default docker context
  [
    {
      "Name": "default",
      "Metadata": {},
      "Endpoints": {
        "docker": {
          "Host": "unix:///var/run/docker.sock",
          "SkipTLSVerify": false
        }
      },
      "TLSMaterial": {},
      "Storage": {
        "MetadataPath": "<IN MEMORY>",
        "TLSPath": "<IN MEMORY>"
      }
    }
  ]
Creating a new builder instance
  /usr/bin/docker buildx create --name builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa68984 --driver docker-container --buildkitd-flags --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host --use
  builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa68984
Appending node(s) to builder
  /usr/bin/docker buildx create --name builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa68984 --append --platform linux/arm64 ssh://root@docker-builder-arm
  ERROR: failed to initialize builder builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa68984 (builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa6898[41](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:42)): Get "http://docker.example.com/v1.47/containers/buildx_buildkit_builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa689841/json": context deadline exceeded
Error: Failed to append node undefined: ERROR: failed to initialize builder builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa68984 (builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa689841): Get "http://docker.example.com/v1.[47](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:48)/containers/buildx_buildkit_builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa689841/json": context deadline exceeded

Expected behaviour

It should run successfully

Actual behaviour

It has cryptic error: ERROR: failed to initialize builder builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa68984 (builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa6898[41](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:42)): Get "http://docker.example.com/v1.47/containers/buildx_buildkit_builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa689841/json": context deadline exceeded Error: Failed to append node undefined: ERROR: failed to initialize builder builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa68984 (builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa689841): Get "http://docker.example.com/v1.[47](https://github.com/minswap/monorepo/actions/runs/12350039497/job/34462188386#step:5:48)/containers/buildx_buildkit_builder-c0e8be7c-9e64-4e9f-ac0c-2d8d1aa689841/json": context deadline exceeded

Repository URL

No response

Workflow run URL

No response

YAML workflow

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Tailscale
        uses: tailscale/github-action@v2
        with:
          oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
          oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
          tags: tag:github-actions
          version: 1.78.1

      - name: Add host key for SSH
        run: |
          mkdir -p $HOME/.ssh
          echo "ssh_host_keys" >> $HOME/.ssh/known_hosts

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          append: |
            - endpoint: ssh://root@docker-builder-arm
              platforms: linux/arm64
            - endpoint: ssh://root@testnet-preprod
              platforms: linux/amd64

Workflow logs

No response

BuildKit logs

Additional info

No response

The text was updated successfully, but these errors were encountered:

crazy-max · 2024-12-16T10:32:38Z

I think you forgot to set the remote driver:

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver: remote
          append: |
            - endpoint: ssh://root@docker-builder-arm
              platforms: linux/arm64
            - endpoint: ssh://root@testnet-preprod
              platforms: linux/amd64

longngn · 2024-12-16T17:41:29Z

I think you forgot to set the remote driver:

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v3
    with:
      driver: remote
      append: |
        - endpoint: ssh://root@docker-builder-arm
          platforms: linux/arm64
        - endpoint: ssh://root@testnet-preprod
          platforms: linux/amd64

the job is flaky, I added driver: remote and try again and it didn't work (maybe the flag didn't work or it still reads old job)
after a few hours I just tried again and it works now
why would the job request http://docker.example.com in the first place though? I think it's worth looking into

longngn · 2024-12-17T08:51:33Z

I think you forgot to set the remote driver:

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v3
    with:
      driver: remote
      append: |
        - endpoint: ssh://root@docker-builder-arm
          platforms: linux/arm64
        - endpoint: ssh://root@testnet-preprod
          platforms: linux/amd64

actually, adding driver: remote break it:

Creating a new builder instance
  /usr/bin/docker buildx create --name builder-808c5dfd-59f3-495c-a05c-7edea0903ecc --driver remote --use
  ERROR: no remote endpoint provided
Error: ERROR: no remote endpoint provided

if I removed it, it'd work (until docker.example.com error happens)

Appending node(s) to builder
  /usr/bin/docker buildx create --name builder-9be4012c-5852-4a9c-9826-67922b924d60 --append --platform linux/arm64 ssh://root@docker-builder-arm
  builder-9be4012c-5852-4a9c-9826-67922b924d60
  /usr/bin/docker buildx create --name builder-9be4012c-5852-4a9c-9826-67922b924d60 --append --platform linux/amd64 ssh://root@testnet-preprod
  builder-9be4012c-5852-4a9c-9826-67922b924d60

crazy-max · 2024-12-17T09:34:27Z

actually, adding driver: remote break it:

Creating a new builder instance
  /usr/bin/docker buildx create --name builder-808c5dfd-59f3-495c-a05c-7edea0903ecc --driver remote --use
  ERROR: no remote endpoint provided
Error: ERROR: no remote endpoint provided

Ah right don't need to use the remote driver as your intent is to create container builder on remote using the Docker API.

until docker.example.com error happens

docker.example.com seems related to the SSH connection helper in docker CLI: https://github.com/docker/cli/blob/8454e012cadeca7719f170012c5331655ed5e5ec/cli/connhelper/connhelper.go#L59 (cc @thaJeztah)

Are docker-builder-arm and testnet-preprod actual IPs/domains redacted?

Can you try creating a docker ssh context and check with docker info if it works?:

docker context create --docker host=ssh://root@docker-builder-arm mycontext
docker context inspect mycontext
docker -c mycontext info

longngn added the status/triage label Dec 16, 2024

crazy-max added status/needs-investigation and removed status/triage labels Dec 16, 2024

crazy-max added status/needs-more-info and removed status/needs-investigation labels Dec 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to append node, Get http://docker.example.com context deadline exceeded #391

Failed to append node, Get http://docker.example.com context deadline exceeded #391

longngn commented Dec 16, 2024

crazy-max commented Dec 16, 2024

longngn commented Dec 16, 2024

longngn commented Dec 17, 2024

crazy-max commented Dec 17, 2024

Failed to append node, Get http://docker.example.com context deadline exceeded #391

Failed to append node, Get http://docker.example.com context deadline exceeded #391

Comments

longngn commented Dec 16, 2024

Contributing guidelines

I've found a bug, and:

Description

Expected behaviour

Actual behaviour

Repository URL

Workflow run URL

YAML workflow

Workflow logs

BuildKit logs

Additional info

crazy-max commented Dec 16, 2024

longngn commented Dec 16, 2024

longngn commented Dec 17, 2024

crazy-max commented Dec 17, 2024