Failed to add node to teleport cluster in AWS EKS Cluster. #8607

GuangTianLi · 2021-10-14T03:55:16Z

GuangTianLi
Oct 14, 2021

teleport start log:

sudo teleport start --roles=node --token=[token] --auth-server=https://teleport.dbkops.com:443 --ca-pin=[pin] --debug
DEBU [SQLITE]    Connected to: file:/var/lib/teleport/proc/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:172
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:217
DEBU [KEYGEN]    SSH cert authority started with no keys pre-compute. native/native.go:106
DEBU [PROC:1]    Adding service to supervisor. service:register.node service/supervisor.go:201
DEBU [PROC:1]    Adding service to supervisor. service:ssh.node service/supervisor.go:201
DEBU [PROC:1]    Adding service to supervisor. service:ssh.shutdown service/supervisor.go:201
DEBU [PROC:1]    Adding service to supervisor. service:common.rotate service/supervisor.go:201
DEBU [PROC:1]    No signal pipe to import, must be first Teleport process. service/service.go:818
DEBU [PROC:1]    Service has started. service:register.node service/supervisor.go:262
DEBU [PROC:1]    Connected state: never updated. service/connect.go:103
INFO [PROC:1]    Connecting to the cluster teleport.dbkops.com with TLS client certificate. service/connect.go:132
DEBU [PROC:1]    Attempting to connect to Auth Server directly. auth-addrs:[teleport.dbkops.com:443] service/connect.go:819
DEBU [PROC:1]    Service has started. service:ssh.node service/supervisor.go:262
DEBU [PROC:1]    Service has started. service:ssh.shutdown service/supervisor.go:262
DEBU [PROC:1]    Service has started. service:common.rotate service/supervisor.go:262
DEBU [PROC:1]    Failed to connect to Auth Server directly. auth-addrs:[teleport.dbkops.com:443] service/connect.go:825
DEBU [PROC:1]    Attempting to discover reverse tunnel address. auth-addrs:[teleport.dbkops.com:443] service/connect.go:834
DEBU             Attempting GET teleport.dbkops.com:443/webapi/find webclient/webclient.go:62
DEBU [PROC:1]    Attempting to connect to Auth Server through tunnel. proxy-addr:teleport.dbkops.com:3024 service/connect.go:843
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
DEBU [PROC:1]    Connected to Auth Server through tunnel. proxy-addr:teleport.dbkops.com:3024 service/connect.go:855
DEBU [PROC:1]    Connected client: Identity(Node, cert(f4cce78a-c63a-4464-abee-dc79e3a5a61b.teleport.dbkops.com issued by teleport.dbkops.com:299909333634806521418197115113145938577),trust root(teleport.dbkops.com:299909333634806521418197115113145938577)) service/connect.go:86
DEBU [PROC:1]    Connected server: Identity(Node, cert(f4cce78a-c63a-4464-abee-dc79e3a5a61b.teleport.dbkops.com issued by teleport.dbkops.com:299909333634806521418197115113145938577),trust root(teleport.dbkops.com:299909333634806521418197115113145938577)) service/connect.go:87
INFO [PROC:1]    Node: features loaded from auth server: Kubernetes:true App:true DB:true  service/connect.go:58
DEBU [PROC:1]    Broadcasting event. event:SSHIdentity service/supervisor.go:370
DEBU [PROC:1]    Service is completed and removed. service:register.node service/supervisor.go:239
DEBU [NODE:1]    Received event "SSHIdentity". service/service.go:1670
DEBU [BPF]       Enhanced session recording is not enabled, skipping. bpf/bpf.go:120
DEBU [RESTRICTE] Restricted session is not enabled, skipping. restrictedsession/restricted.go:92
DEBU [NODE:1]    Starting watch. resource-kind:lock retry:Linear(attempt=0, duration=0s) services/watcher.go:177
DEBU [SSH:NODE]  Supported ciphers: ["[email protected]" "[email protected]" "aes128-ctr" "aes192-ctr" "aes256-ctr"]. sshutils/server.go:229
DEBU [SSH:NODE]  Supported KEX algorithms: ["[email protected]" "ecdh-sha2-nistp256" "ecdh-sha2-nistp384" "ecdh-sha2-nistp521"]. sshutils/server.go:239
DEBU [SSH:NODE]  Supported MAC algorithms: ["[email protected]" "hmac-sha2-256"]. sshutils/server.go:249
DEBU [NODE:BEAT] Starting Node heartbeat with announce period: 1m0s, keep-alive period 5m7.325353698s, poll period: 5s srv/heartbeat.go:143
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:1976
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:1976
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions. service/service.go:1976
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions/default. service/service.go:1976
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:1976
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:1976
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming. service/service.go:1976
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming/default. service/service.go:1976
DEBU [PROC:1]    Adding service to supervisor. service:uploader.service service/supervisor.go:201
DEBU [PROC:1]    Adding service to supervisor. service:uploader.shutdown service/supervisor.go:201
DEBU [PROC:1]    Adding service to supervisor. service:fileuploader.service service/supervisor.go:201
DEBU [PROC:1]    Adding service to supervisor. service:fileuploader.shutdown service/supervisor.go:201
DEBU [PROXY:AGE] Starting agent pool f4cce78a-c63a-4464-abee-dc79e3a5a61b.teleport.dbkops.com.teleport.dbkops.com... cluster:teleport.dbkops.com reversetunnel/agentpool.go:167
INFO [NODE:1]    Service is starting in tunnel mode. service/service.go:1888
DEBU [PROC:1]    Broadcasting event. event:NodeReady service/supervisor.go:370
DEBU [PROC:1]    Broadcasting mapped event. in:NodeReady out:EventMapping(in=[NodeReady], out=TeleportReady) service/supervisor.go:395
INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status with period 10m0s. service/connect.go:445
DEBU [PROC:1]    Service has started. service:uploader.service service/supervisor.go:262
DEBU [PROC:1]    Service has started. service:uploader.shutdown service/supervisor.go:262
DEBU [PROC:1]    Service has started. service:fileuploader.service service/supervisor.go:262
DEBU [PROC:1]    Service has started. service:fileuploader.shutdown service/supervisor.go:262
DEBU [PROXY:AGE] Seeking: {Addr:teleport.dbkops.com:3024 AddrNetwork:tcp Path:}. cluster:teleport.dbkops.com reversetunnel/agentpool.go:197
DEBU [PROXY:AGE] Adding agent(leaseID=1,state=connecting) -> teleport.dbkops.com:teleport.dbkops.com:3024. cluster:teleport.dbkops.com reversetunnel/agentpool.go:284
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
INFO [NODE:PROX] Connected. addr:192.168.29.2:33106 remote-addr:52.198.217.70:3024 leaseID:1 target:teleport.dbkops.com:3024 reversetunnel/agent.go:354
DEBU [NODE:PROX] Agent connected to proxy: [da9fd063-fe1e-4f36-998c-ed3194200c05.teleport.dbkops.com da9fd063-fe1e-4f36-998c-ed3194200c05 teleport-7df976d769-ftqjb.teleport.dbkops.com teleport-7df976d769-ftqjb localhost 127.0.0.1 ::1 teleport.dbkops.com remote.kube.proxy.teleport.cluster.local]. leaseID:1 target:teleport.dbkops.com:3024 reversetunnel/agent.go:359
DEBU [NODE:PROX] Changing state connecting -> connected. leaseID:1 target:teleport.dbkops.com:3024 reversetunnel/agent.go:193
DEBU [NODE:PROX] Discovery request channel opened: teleport-discovery. leaseID:1 target:teleport.dbkops.com:3024 reversetunnel/agent.go:470
DEBU [NODE:PROX] handleDiscovery requests channel. leaseID:1 target:teleport.dbkops.com:3024 reversetunnel/agent.go:488
WARN [NODE:PROX] Unable to continue processesing requests: heartbeat: connection closed. leaseID:1 target:teleport.dbkops.com:3024 reversetunnel/agent.go:378
DEBU [NODE:PROX] Changing state connected -> disconnected. leaseID:1 target:teleport.dbkops.com:3024 reversetunnel/agent.go:193
DEBU [PROXY:AGE] Seeking: {Addr:teleport.dbkops.com:3024 AddrNetwork:tcp Path:}. cluster:teleport.dbkops.com reversetunnel/agentpool.go:197
DEBU [PROXY:AGE] Adding agent(leaseID=2,state=connecting) -> teleport.dbkops.com:teleport.dbkops.com:3024. cluster:teleport.dbkops.com reversetunnel/agentpool.go:284
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
INFO [NODE:PROX] Connection closed, returning leaseID:1 target:teleport.dbkops.com:3024 reversetunnel/agent.go:498
WARN [PROC:1]    Sync rotation state cycle failed: watcher has disconnected, going to retry after ~10s. service/connect.go:462
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
WARN [NODE:1]    Restart watch on error: watcher is closed. resource-kind:lock services/watcher.go:194
DEBU [NODE:1]    Starting watch. resource-kind:lock retry:Linear(attempt=1, duration=1s) services/watcher.go:177
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122

Teleport Cluster log:

2021-10-14T03:46:34Z WARN [MXWEB]     Handshake failed. error:[remote error: tls: bad certificate] multiplexer/web.go:137

bergmannf · 2021-10-29T13:05:48Z

bergmannf
Oct 29, 2021

I think i have the similar problem, when deploying teleport in a local kubernetes cluster.

I install it using the helm-chart and I can join the node (the teleport cluster event acknowledges the join), but it is immediately followed by the tls: bad certificate warnings as well:

2021-10-29T13:03:26Z DEBU [AUTH[]      ClientCertPool -> cert(teleport.dev.my.domain issued by teleport.dev.my.domain:212991665884250844065797333951560948156) auth/middleware.go:590
2021-10-29T13:03:26Z INFO [AUTH[]      Node "jumphost-0001" [c54ae28d-ddcb-49d2-8b8e-8df28533704e[] is trying to join with role: Node. auth/auth.go:1711
2021-10-29T13:03:26Z DEBU [KEYGEN[]    Generated SSH host certificate for role Node with principals: [c54ae28d-ddcb-49d2-8b8e-8df28533704e.teleport.dev.my.domain c54ae28d-ddcb-49d2-8b8e-8df28533704e jumphost-0001.teleport.dev.my.domain jumphost-0001 localhost 127.0.0.1 ::1[]. native/native.go:231
2021-10-29T13:03:26Z INFO [CA[]        Generating TLS certificate {0x668fe78 0xc0013542d0 1.3.9999.1.7=#131e74656c65706f72742e6465762e617468656e652e647671612e636c6f7564,CN=c54ae28d-ddcb-49d2-8b8e-8df28533704e.teleport.dev.my.domain,O=Node,POSTALCODE=null,STREET= 2031-10-27 13:03:26.86665402 +0000 UTC [jumphost-0001 c54ae28d-ddcb-49d2-8b8e-8df28533704e[]}. common_name:c54ae28d-ddcb-49d2-8b8e-8df28533704e.teleport.dev.my.domain dns_names:[jumphost-0001 c54ae28d-ddcb-49d2-8b8e-8df28533704e[] locality:[] not_after:2031-10-27 13:03:26.86665402 +0000 UTC org:[Node[] org_unit:[] tlsca/ca.go:640
2021-10-29T13:03:26Z INFO [AUTH[]      Node "jumphost-0001" [c54ae28d-ddcb-49d2-8b8e-8df28533704e[] has joined the cluster. auth/auth.go:1752
2021-10-29T13:03:26Z DEBU [AUTH[]      ClientCertPool -> cert(teleport.dev.my.domain issued by teleport.dev.my.domain:91488424854950793482707354896901456673) auth/middleware.go:590
2021-10-29T13:03:26Z DEBU [AUTH[]      ClientCertPool -> cert(teleport.dev.my.domain issued by teleport.dev.my.domain:212991665884250844065797333951560948156) auth/middleware.go:590
2021-10-29T13:03:26Z DEBU [AUTH[]      ClientCertPool -> cert(teleport.dev.my.domain issued by teleport.dev.my.domain:91488424854950793482707354896901456673) auth/middleware.go:590
2021-10-29T13:03:26Z DEBU [AUTH[]      ClientCertPool -> cert(teleport.dev.my.domain issued by teleport.dev.my.domain:212991665884250844065797333951560948156) auth/middleware.go:590
2021-10-29T13:03:26Z WARN [MXWEB[]     Handshake failed. error:[remote error: tls: bad certificate[] multiplexer/web.go:137
2021-10-29T13:03:26Z WARN [MXWEB[]     Handshake failed. error:[remote error: tls: bad certificate[] multiplexer/web.go:137

After that the node does not show up in the web interface as well.

0 replies

webvictim · 2021-10-29T19:37:20Z

webvictim
Oct 29, 2021
Collaborator

Is port 3024 open on your Teleport proxy so the node can establish a reverse tunnel?
Is the reverse tunnel address correct? You can check with curl https://teleport-proxy:443/webapi/ping (replace teleport-proxy with your FQDN)

4 replies

bergmannf Nov 2, 2021

I assume this is correct in my case - I ran this command on the machine I want to add to teleport:

curl https://teleport.my.fqdn:443/webapi/ping
{"auth":{"type":"local","second_factor":"otp","u2f":{"app_id":""},"has_motd":false},"proxy":{"kube":{"enabled":true,"listen_addr":"0.0.0.0:3026"},"ssh":{"listen_addr":"[::]:3023","tunnel_listen_addr":"0.0.0.0:3024","public_addr":"teleport.my.fqdn:443"},"db":{"mysql_listen_addr":"0.0.0.0:3036"}},"server_version":"7.3.2","min_client_version":"6.0.0"}

bergmannf Nov 2, 2021

I found a way to make it work, I am not sure if it is the way it should be done though:

I had to add listen_addr for the auth_service:

auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  cluster_name: teleport-0001.dev.athene.dvqa.cloud

I had to add the 3025 port to the kubernetes service to also be forwarded to the 3025 port on the pod:

kubectl edit service -n teleport teleport

  - name: auth
    port: 3025
    protocol: TCP
    targetPort: 3025

After doing this the nodes were shown in the web ui and I can connect to them.

webvictim Nov 2, 2021
Collaborator

This will work, but joining via the auth server (port 3025) rather than the proxy (port 443) means that your node will need to have port 3022 open and exposed on a public IP for any client to be able to connect - it won't be using Teleport's reverse tunnel system (which doesn't require any open ports). It also means that you wouldn't be able to join database, application or Kubernetes services to your Teleport cluster - only SSH nodes.

To get it working via the proxy, try shutting down Teleport on your node, deleting /var/lib/teleport on its filesystem, then rerunning the start/join command without the https on the auth server URL:
sudo teleport start --roles=node --token=[token] --auth-server=teleport.dbkops.com:443 --debug

(the CA pin hash is also not needed when joining via the proxy server - the chain of trust on the TLS certificates is used to ensure that you're connecting to the right server)

bergmannf Nov 5, 2021

This is strange, because I actually needed the 3025 port, but I still used the auth-server with port 443.

I could also add kubernetes clusters and applications to the cluster without problems.

In the Web UI the SSH node also shows the connection as tunnel.

As an example the command to join a node to the cluster was:

teleport start --roles=node --auth-server=teleport.dev.my.domain:443 --labels role=jumphost

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to add node to teleport cluster in AWS EKS Cluster. #8607

{{title}}

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Failed to add node to teleport cluster in AWS EKS Cluster. #8607

GuangTianLi Oct 14, 2021

Replies: 2 comments · 4 replies

bergmannf Oct 29, 2021

webvictim Oct 29, 2021 Collaborator

bergmannf Nov 2, 2021

bergmannf Nov 2, 2021

webvictim Nov 2, 2021 Collaborator

bergmannf Nov 5, 2021

GuangTianLi
Oct 14, 2021

Replies: 2 comments 4 replies

bergmannf
Oct 29, 2021

webvictim
Oct 29, 2021
Collaborator

webvictim Nov 2, 2021
Collaborator