On-Prem Omni behind Traefik (or other reverse proxy)

[kenlasko]

I've been successfully running Omni on-prem for the past year using the suggested approach. Its currently running fine when using network_mode: host, but I'm trying to simplify things a bit, because I have to run a separate container to ensure the LetsEncrypt certificates generated by Traefik are available as files for Omni. That container has failed from time to time, which means the Omni certs eventually go stale and the cluster reacts badly.

Instead of using network_mode: host, I'm trying to proxy everything through Traefik in a somewhat similar approach to How to expose Omni with Nginx. I'm trying to do things that cause the least amount of interruption to my multiple clusters, which means that I'm not using separate DNS names for the API and KProxy endpoints and sending everything through 443.

I've got Traefik listeners for ports 8090 and 8100, which are using the LetsEncrypt certs. I've added all the other ports that appear when running ss -tulpn when the Omni container is running in host networking mode to the Omni container (5001, 10000-10001, 50180, 8091-8093). The siderolink.ManagerController connects properly over 8090, but I keep getting these errors, which prevents updates from going through:

[talos] controller failed {"component": "controller-runtime", "controller": "v1alpha1.EventsSinkController", "error": "error publishing event: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [fdae:41e4:649b:9303::1]:8091: i/o timeout\""} 

As soon as I revert back to my normal config with network_mode: host, all machines connect properly. I presume that the :8091 connection is being done over the Wireguard tunnel and not directly against the node address. What am I missing with my configuration to allow this to work?

Here's my docker-compose.yaml:

services:
  omni:
    image: ghcr.io/siderolabs/omni:v0.47.1
    container_name: omni
    restart: always
#    network_mode: host  # disable for new method
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun
    ports:          # disable for old method
      - 5001
      - 10000-10001:10000-10001
      - 50180:50180
      - 8091-8093:8091-8093
    command:
      - --account-id=<REDACTED>
      - --name=onprem-omni
#      - --cert=/tls.crt              # disable for new method
#      - --key=/tls.key               # disable for new method
#      - --machine-api-cert=/tls.crt  # disable for new method
#      - --machine-api-key=/tls.key   # disable for new method
      - --private-key-source=file:///omni.asc
      - --event-sink-port=8091
      - --bind-addr=0.0.0.0:4443
      - --machine-api-bind-addr=0.0.0.0:8090
      - --k8s-proxy-bind-addr=0.0.0.0:8100
      - --advertised-api-url=https://omni.ucdialplans.com/
      - --siderolink-api-advertised-url=https://omni.ucdialplans.com:8090/
      - --siderolink-wireguard-advertised-addr=192.168.1.17:50180
      - --advertised-kubernetes-proxy-url=https://omni.ucdialplans.com:8100/
      - --auth-auth0-enabled=true
      - --auth-auth0-domain=<REDACTED>
      - --auth-auth0-client-id=<REDACTED>
      - --workload-proxying-subdomain=omni
      - [email protected]
#      - --siderolink-use-grpc-tunnel=true   # hopefully not required
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.omni.rule=HostRegexp(`omni.ucdialplans.com`) || HostRegexp(`.+.omni.ucdialplans.com`)"
      - "traefik.http.routers.omni.tls.domains[0].main=ucdialplans.com"
      - "traefik.http.routers.omni.tls.domains[0].main=*.ucdialplans.com"
      - "traefik.http.routers.omni.tls.domains[1].main=omni.ucdialplans.com"
      - "traefik.http.routers.omni.tls.domains[1].main=*.omni.ucdialplans.com"
      - "traefik.http.routers.omni.entrypoints=websecure"
      - "traefik.http.routers.omni.tls=true"
      - "traefik.http.routers.omni.tls.certresolver=letsencrypt"
      - "traefik.http.routers.omni.service=omni"
      - "traefik.http.services.omni.loadbalancer.server.port=4443"
      - "traefik.http.services.omni.loadbalancer.server.scheme=https"
      - "traefik.http.services.omni.loadbalancer.server.scheme=h2c"    # disable for old method
      # Omni API routing rule for consolidated routing
      - "traefik.http.routers.mapi.rule=Host(`omni.ucdialplans.com`) || Host(`192.168.1.17`)"
      - "traefik.http.routers.mapi.entrypoints=mapi"
      - "traefik.http.routers.mapi.tls=true"
      - "traefik.http.routers.mapi.tls.domains[0].main=omni.ucdialplans.com"
      - "traefik.http.routers.mapi.tls.certresolver=letsencrypt"
      - "traefik.http.routers.mapi.service=mapi"
      - "traefik.http.services.mapi.loadbalancer.server.port=8090"
      - "traefik.http.services.mapi.loadbalancer.server.scheme=h2c"
      # Omni Kube routing rule for consolidated routing
      - "traefik.http.routers.kproxy.rule=Host(`omni.ucdialplans.com`)"
      - "traefik.http.routers.kproxy.entrypoints=kproxy"
      - "traefik.http.routers.kproxy.tls=true"
      - "traefik.http.routers.kproxy.tls.domains[0].main=omni.ucdialplans.com"
      - "traefik.http.routers.kproxy.tls.certresolver=letsencrypt"
      - "traefik.http.routers.kproxy.service=kproxy"
      - "traefik.http.services.kproxy.loadbalancer.server.port=8100"
      - "traefik.http.services.kproxy.loadbalancer.server.scheme=http"
    volumes:
      - ./omni:/_out
      - ./certs/star_ucdialplans_com.pem:/tls.crt   # disable for new method
      - ./certs/star_ucdialplans_com.key:/tls.key   # disable for new method
      - ./omni/omni.asc:/omni.asc
    networks:  # disable for old method
      - ipv6   # disable for old method

  traefik:
    container_name: traefik
    image: traefik:latest
    restart: always
    command:
      - --providers.docker=true
      - --api.dashboard=true
      - --log.level=INFO
      - --log.filePath=/var/log/traefik.log
      - --serversTransport.insecureSkipVerify=true
      # Set up LetsEncrypt
      - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.letsencrypt.acme.email=ken.lasko@gmail.com
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
      # Set up an insecure listener that redirects all traffic to TLS
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      # Set up the TLS configuration for our websecure listener
      - --entrypoints.websecure.http.tls=true
      - --entrypoints.websecure.http.tls.certResolver=letsencrypt
      - --entrypoints.websecure.http.tls.domains[0].main=*.ucdialplans.com
      # Set up the Omni endpoints
      - --entryPoints.mapi.address=:8090
      - --entryPoints.kproxy.address=:8100
#      - --entryPoints.wireguard.address=:51820
    environment:
      - [email protected]
      - CLOUDFLARE_DNS_API_TOKEN=<REDACTED>
    ports:
      - 80:80
      - 443:443
      - 8090:8090
      - 8100:8100
#      - 51820:51820
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./certs:/letsencrypt"
      - "./traefik/logs/traefik.log:/var/log/traefik.log"
    labels:
      # Dashboard
      - "traefik.enable=true"
      - 'traefik.http.routers.traefik.rule=Host(`traefik-rpi1.ucdialplans.com`)'
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik.service=api@internal"
      - 'traefik.http.routers.traefik.middlewares=strip'
      - 'traefik.http.middlewares.strip.stripprefix.prefixes=/traefik'
      # middleware redirect
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    extra_hosts:
      - host.docker.internal:172.19.0.1
    networks:
      - ipv6

[smira]

We can't guess correct Traefik configuration, but you can look at the Talos logs to guess between two scenarios:

• if SideroLink API is not exposed properly, you will see Talos failing to setup SideroLink connection repeatedly and retrying
• if SideroLink API is exposed properly, but Wireguard port is not, you will see Talos successfully setting up SideroLink connection followed in 3 minutes by another attempt to reconnect (as the tunnel is down).

Wireguard UDP port should be exposed directly from the host network without any modification to the packets.

[kenlasko]

SideroLink API is done over 8090, right? If so, then that is working if I'm reading this correctly:

[talos] siderolink connection configured {"component": "controller-runtime", "controller": "siderolink.ManagerController", "endpoint": "https://omni.ucdialplans.com:8090/?jointoken=<REDACTED>", "node_uuid": "4e211131-23e7-f477-9437-94c691a76e7c", "node_address": "fdae:41e4:649b:9303:eac0:efba:125e:6545/64"} 

What port should be opened for Wireguard UDP?

[smira]

Whatever port you configured, siderolink-wireguard-advertised-addr=192.168.1.17:50180 ?

[kenlasko]

Yes, that port is open:

    ports:          # disable for old method
      - 5001
      - 10000-10001:10000-10001
      - 50180:50180
      - 8091-8093:8091-8093

[kenlasko]

Looking at that closer, it might be because I'm not explicitly setting /udp on the 50180 port. Trying that now.

[kenlasko]

The solution was to explicitly set the Docker port to /udp. Here's the complete docker-compose.yml for those who come across this in the future (not sure if the other ports are explicitly required):

services:
  omni:
    image: ghcr.io/siderolabs/omni:v0.47.1
    container_name: omni
    restart: always
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun
    ports: 
      - 5001
      - 10000-10001:10000-10001
      - 50180:50180/udp
      - 8091-8093:8091-8093
    command:
      - --account-id=<REDACTED>
      - --name=onprem-omni
      - --private-key-source=file:///omni.asc
      - --event-sink-port=8091
      - --bind-addr=0.0.0.0:4443
      - --machine-api-bind-addr=0.0.0.0:8090
      - --k8s-proxy-bind-addr=0.0.0.0:8100
      - --advertised-api-url=https://omni.ucdialplans.com/
      - --siderolink-api-advertised-url=https://omni.ucdialplans.com:8090/
      - --siderolink-wireguard-advertised-addr=192.168.1.17:50180
      - --advertised-kubernetes-proxy-url=https://omni.ucdialplans.com:8100/
      - --auth-auth0-enabled=true
      - --auth-auth0-domain=<REDACTED>
      - --auth-auth0-client-id=<REDACTED>
      - --workload-proxying-subdomain=omni
      - [email protected]
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.omni.rule=HostRegexp(`omni.ucdialplans.com`) || HostRegexp(`.+.omni.ucdialplans.com`)"
      - "traefik.http.routers.omni.tls.domains[0].main=ucdialplans.com"
      - "traefik.http.routers.omni.tls.domains[0].main=*.ucdialplans.com"
      - "traefik.http.routers.omni.tls.domains[1].main=omni.ucdialplans.com"
      - "traefik.http.routers.omni.tls.domains[1].main=*.omni.ucdialplans.com"
      - "traefik.http.routers.omni.entrypoints=websecure"
      - "traefik.http.routers.omni.tls=true"
      - "traefik.http.routers.omni.tls.certresolver=letsencrypt"
      - "traefik.http.routers.omni.service=omni"
      - "traefik.http.services.omni.loadbalancer.server.port=4443"
      - "traefik.http.services.omni.loadbalancer.server.scheme=https"
      - "traefik.http.services.omni.loadbalancer.server.scheme=h2c"    # disable for old method
      # Omni API routing rule for consolidated routing
      - "traefik.http.routers.mapi.rule=Host(`omni.ucdialplans.com`) || Host(`192.168.1.17`)"
      - "traefik.http.routers.mapi.entrypoints=mapi"
      - "traefik.http.routers.mapi.tls=true"
      - "traefik.http.routers.mapi.tls.domains[0].main=omni.ucdialplans.com"
      - "traefik.http.routers.mapi.tls.certresolver=letsencrypt"
      - "traefik.http.routers.mapi.service=mapi"
      - "traefik.http.services.mapi.loadbalancer.server.port=8090"
      - "traefik.http.services.mapi.loadbalancer.server.scheme=h2c"
      # Omni Kube routing rule for consolidated routing
      - "traefik.http.routers.kproxy.rule=Host(`omni.ucdialplans.com`)"
      - "traefik.http.routers.kproxy.entrypoints=kproxy"
      - "traefik.http.routers.kproxy.tls=true"
      - "traefik.http.routers.kproxy.tls.domains[0].main=omni.ucdialplans.com"
      - "traefik.http.routers.kproxy.tls.certresolver=letsencrypt"
      - "traefik.http.routers.kproxy.service=kproxy"
      - "traefik.http.services.kproxy.loadbalancer.server.port=8100"
      - "traefik.http.services.kproxy.loadbalancer.server.scheme=http"
    volumes:
      - ./omni:/_out
      - ./omni/omni.asc:/omni.asc
    networks:  
      - ipv6 

  traefik:
    container_name: traefik
    image: traefik:latest
    restart: always
    command:
      - --providers.docker=true
      - --api.dashboard=true
      - --log.level=INFO
      - --log.filePath=/var/log/traefik.log
      - --serversTransport.insecureSkipVerify=true
      # Set up LetsEncrypt
      - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.letsencrypt.acme.email=ken.lasko@gmail.com
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
      # Set up an insecure listener that redirects all traffic to TLS
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      # Set up the TLS configuration for our websecure listener
      - --entrypoints.websecure.http.tls=true
      - --entrypoints.websecure.http.tls.certResolver=letsencrypt
      - --entrypoints.websecure.http.tls.domains[0].main=*.ucdialplans.com
      # Set up the Omni endpoints
      - --entryPoints.mapi.address=:8090
      - --entryPoints.kproxy.address=:8100
    environment:
      - [email protected]
      - CLOUDFLARE_DNS_API_TOKEN=<REDACTED>
    ports:
      - 80:80
      - 443:443
      - 8090:8090
      - 8100:8100
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./certs:/letsencrypt"
      - "./traefik/logs/traefik.log:/var/log/traefik.log"
    labels:
      # Dashboard
      - "traefik.enable=true"
      - 'traefik.http.routers.traefik.rule=Host(`traefik-rpi1.ucdialplans.com`)'
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik.service=api@internal"
      - 'traefik.http.routers.traefik.middlewares=strip'
      - 'traefik.http.middlewares.strip.stripprefix.prefixes=/traefik'
      # middleware redirect
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    extra_hosts:
      - host.docker.internal:172.19.0.1
    networks:
      - ipv6

networks:
  ipv6:
    enable_ipv6: false
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.19.0.0/24
          gateway: 172.19.0.1