Kubernetes - "the node is offline or has disconnected"

[thibvatin]

Hi,

I've setup a Teleport single node cluster which is able to retrieve my different nodes and i'm able to connect through ssh on them.
But now, I want my Teleport cluster to act as a gateway to allow my Kubernetes to connect to it over reverse tunnel. I have deployed a Teleport agent in my Kubernetes cluster and configured the Teleport cluster to enable the kube service.

Unfortunately, i've always the same issue when I try to retrieve the pods for example :
"Error from server: the node is offline or has disconnected, if the issue persists, restart the node or re-register it in the cluster"

Here is my Teleport config file :

teleport:
  nodename: bastion.domain.com
  data_dir: /var/lib/teleport
  auth_token: 1aedec7eae94144984512ghfacbopfdfghertyt8fa8c970b12ae0d49e5a304
  auth_servers:
  - 172.17.1.4:3025
  - bastion.domain.com:3025
[...]
  ca_pin: sha256:4cc16614d00f940acc82c8c6c48a53b3af414
  log:
    output: /var/lib/teleport/teleport.log
    severity: DEBUG
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  public_addr: bastion.domain.com:3025
  tokens:
  - proxy,node:3bedtede9dac7eae859444e0068fa8c93670b12ae
  - kube:b7aa0a4f4e770a7213be0a8591321fe8
ssh_service:
[...]
  public_addr: teleport.domain.com:3022
[...]
proxy_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3023
  web_listen_addr: 0.0.0.0:3080
  tunnel_listen_addr: 0.0.0.0:3024
  public_addr: teleport.domain.com
  https_key_file: /etc/letsencrypt/live/teleport.domain.com/privkey.pem
  https_cert_file: /etc/letsencrypt/live/teleport.domain.com/fullchain.pem
  kube_listen_addr: 0.0.0.0:3027
  kube_public_addr: teleport.domain.com:3027

And here is the cmds and the logs when attempting to retrieve my PODs for example :

tsh kube ls                                                               
Kube Cluster Name Selected
----------------- --------
my-cluster-aks-01     *
tsh kube login my-cluster-aks-01
Logged into kubernetes cluster "my-cluster-aks-01"
kubectl get pods -n development                                                   
### and after 1/2 mins, this error log appears :
Error from server: the node is offline or has disconnected, if the issue persists, restart the node or re-register it in the cluster

My logs from teleport.log (debug):

DEBU [PROXY:PRO] Kubernetes session for user: tvatin, users: map[tvatin:{}], groups: map[development-access:{} system:authenticated:{}], teleport cluster: bastion.domain.com, kube cluster: my-cluster-aks-01 forwarded to remote kubernetes_service instance. kubernetes_service.addr:X.X.X.215:3027 kubernetes_service.name:ad01110f-574c-4d0f-a773-399647b89631 proxy/forwarder.go:1453
DEBU [PROXY:SER] Dialing from: "185.25.195.19:56951" to: "X.X.X.215:3027". trace.fields:map[cluster:bastion.domain.com] reversetunnel/localsite.go:184
DEBU [PROXY:SER] Error occurred while dialing through a tunnel. address:X.X.X.215:3027 error:[
ERROR REPORT:
Original Error: *trace.NotFoundError no tunnel connection found: no kube reverse tunnel for ad01110f-574c-4d0f-a773-399647b89631.bastion.domain.com found
Stack Trace:
	/go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:264 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).dialTunnel
	/go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:288 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).getConn
	/go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:186 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).DialTCP
	/go/src/github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:516 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).setupContext.func2
	/go/src/github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:309 github.com/gravitational/teleport/lib/kube/proxy.(*teleportClusterClient).DialWithContext
	/go/src/github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:305 github.com/gravitational/teleport/lib/kube/proxy.(*teleportClusterClient).Dial
	/go/src/github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:1290 github.com/gravitational/teleport/lib/kube/proxy.(*clusterSession).Dial
	/opt/go/src/net/http/transport.go:1144 net/http.(*Transport).dial
	/opt/go/src/net/http/transport.go:1575 net/http.(*Transport).dialConn
	/opt/go/src/net/http/transport.go:1421 net/http.(*Transport).dialConnFor
	/opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
User Message: no tunnel connection found: no kube reverse tunnel for ad01110f-574c-4d0f-a773-399647b89631.bastion.domain.com found] trace.fields:map[cluster:bastion.domain.com] reversetunnel/localsite.go:297
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:221
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:136
DEBU [PROXY:SER] Error occurred while dialing directly. address:X.X.X.215:3027 error:[dial tcp X.X.X.215:3027: i/o timeout] trace.fields:map[cluster:bastion.domain.com] reversetunnel/localsite.go:321
ERRO [PROXY:PRO] Error forwarding to https://X.X.X.215:3027/api?timeout=32s, err: the node is offline or has disconnected, if the issue persists, restart the node or re-register it in the cluster forward/fwd.go:179
DEBU [PROXY:PRO] Kubernetes session for user: tvatin, users: map[tvatin:{}], groups: map[development-access:{} system:authenticated:{}], teleport cluster: bastion.domain.com, kube cluster: my-cluster-aks-01 forwarded to remote kubernetes_service instance. kubernetes_service.addr:X.X.X.215:3027 kubernetes_service.name:ad01110f-574c-4d0f-a773-399647b89631 proxy/forwarder.go:1453
DEBU [PROXY:SER] Dialing from: "185.25.195.19:56951" to: "X.X.X.215:3027". trace.fields:map[cluster:bastion.domain.com] reversetunnel/localsite.go:184
DEBU [PROXY:SER] Error occurred while dialing through a tunnel. address:X.X.X.215:3027 error:[
ERROR REPORT:
Original Error: *trace.NotFoundError no tunnel connection found: no kube reverse tunnel for ad01110f-574c-4d0f-a773-399647b89631.bastion.domain.com found
Stack Trace:
	/go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:264 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).dialTunnel
	/go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:288 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).getConn
	/go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:186 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).DialTCP
	/go/src/github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:516 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).setupContext.func2
	/go/src/github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:309 github.com/gravitational/teleport/lib/kube/proxy.(*teleportClusterClient).DialWithContext
	/go/src/github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:305 github.com/gravitational/teleport/lib/kube/proxy.(*teleportClusterClient).Dial
	/go/src/github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:1290 github.com/gravitational/teleport/lib/kube/proxy.(*clusterSession).Dial
	/opt/go/src/net/http/transport.go:1144 net/http.(*Transport).dial
	/opt/go/src/net/http/transport.go:1575 net/http.(*Transport).dialConn
	/opt/go/src/net/http/transport.go:1421 net/http.(*Transport).dialConnFor
	/opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
User Message: no tunnel connection found: no kube reverse tunnel for ad01110f-574c-4d0f-a773-399647b89631.bastion.domain.com found] trace.fields:map[cluster:bastion.domain.com] reversetunnel/localsite.go:297
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:221
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:136

Thank you for your help

[awly]

@thibvatin how did you deploy the teleport agent in the kubernetes cluster?
If possible, please paste its config and logs.

[thibvatin]

Hi @awly,
I deployed the teleport agent with Helm as below :

helm install teleport-agent teleport/teleport-kube-agent \
--namespace auth \
--set proxyAddr=teleport.domain.com \
--set authToken=b7aa0a4f4e770a7213be0a8591321fe8 \
--set kubeClusterName=my-cluster-aks-01

These are the only logs i have from the POD where the teleport agent has been deployed :
[KUBERNETES] Kubernetes service 5.2.1:v5.2.1-0-g81218b38a is starting on [::]:3027.

Here is its teleport.yaml :

teleport:
  auth_token: "/etc/teleport-secrets/auth-token"
  auth_servers: ["teleport.domain.com"]

kubernetes_service:
  enabled: true
  kube_cluster_name: my-cluster-aks-01
  listen_addr: 0.0.0.0:3027

app_service:
  enabled: false

auth_service:
  enabled: false
ssh_service:
  enabled: false
proxy_service:
  enabled: false

[awly]

@thibvatin can you try changing --set proxyAddr=teleport.domain.com to --set proxyAddr=teleport.domain.com:3080?
Run something like:

helm upgrade teleport-agent teleport/teleport-kube-agent \
  --namespace auth \
  --set proxyAddr=teleport.domain.com:3080 \
  --set authToken=b7aa0a4f4e770a7213be0a8591321fe8 \
  --set kubeClusterName=my-cluster-aks-01

I believe when auth_servers doesn't specify a port, it will default to the standard auth_service port 3025.
For a kube agent to create a reverse tunnel, it needs to go through the proxy_service instead, using port 3080.

[thibvatin]

Hi @awly
Indeed that fixed my issue. I'm now able to display my K8s resources from CLI !

Do you know if there is any solution to access K8s resources from the Teleport UI ?

[awly]

Teleport UI doesn't implement k8s access and we don't have plans for that yet.

You may be able to use different software for k8s UI, if it supports connecting via a kubeconfig file.
Teleport can generate a long-lived kubeconfig with tctl auth sign --format=kubernetes.
I'm not familiar with k8s UI software out there to recommend anything specific/