Authentication issue while accessing pods from kubernetes cluster. Raised issue on the same : https://github.com/gravitational/teleport/issues/12356

[madhureddy08]

Hello Team,
We have configured teleport in AWS k8 cluster. We have installed teleport agents in k8 clusters which we are looking to access from teleport CLI. I have installed teleport CLI on my local machine. I am able to log in to teleport and access the cluster list. Once I accessed a particular cluster and try to access pods, I receive the following error "Error from server: access denied: failed to authenticate with auth server" I have configured the CLI and teleport agents as per the documentation not sure what I am doing wrong here. Looking for recommendations.
Thank you,

Please find the Proxy DEBUG logs:

DEBU [CLIENT]    open /Users/anthony/.tsh/teleport-proxy-kubernetes-teleport-demo.testpoc.com.yaml: no such file or directory client/api.go:781
INFO [CLIENT]    No teleport login given. defaulting to anthony client/api.go:1122
INFO [CLIENT]    no host login given. defaulting to anthony client/api.go:1132
INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/private/tmp/com.apple.launchd.wH8KgidgvO/Listeners" client/api.go:3075
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport-proxy-kubernetes-teleport-demo.testpoc.com:443 client/api.go:3036
DEBU             Attempting GET teleport-proxy-kubernetes-teleport-demo.testpoc.com:443/webapi/ping/ad-sso webclient/webclient.go:105
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport-proxy-kubernetes-teleport-demo.testpoc.com:443 client/api.go:3036
DEBU [CLIENT]    HTTPS client init(proxyAddr=teleport-proxy-kubernetes-teleport-demo.testpoc.com:443, insecure=false) client/weblogin.go:223
INFO [CLIENT]    Waiting for response at: http://127.0.0.1:64341. client/redirect.go:137
If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:64341/ad273db6-c6d1-45c4-8489-8161a6451cbe
DEBU [CLIENT]    Got response from browser. client/weblogin.go:316
DEBU [KEYAGENT]  Adding CA key for teleport-proxy-kubernetes-teleport-demo.testpoc.com client/keyagent.go:313
DEBU [KEYSTORE]  Adding known host teleport-proxy-kubernetes-teleport-demo.testpoc.com with proxy teleport-proxy-kubernetes-teleport-demo.testpoc.com and key: SHA256:N086bK3P7IOd1i4yZ1+eTFJwAzm9x0jXYv7MjCz4oJ0 client/keystore.go:573
ERRO [KEYSTORE]  open /Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]: no such file or directory client/keystore.go:269
INFO [KEYAGENT]  Loading SSH key for user "[email protected]" and cluster "teleport-proxy-kubernetes-teleport-demo.testpoc.com". client/keyagent.go:191
INFO [CLIENT]    Connecting proxy=teleport-proxy-kubernetes-teleport-demo.testpoc.com:3023 login="root" client/api.go:2266
DEBU             No valid environment variables found. client/proxy.go:116
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:268
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]" valid until "2022-04-28 20:35:01 +0000 UTC". client/keystore.go:307
DEBU [KEYAGENT]  "Checking key: [email protected] AAAAHHNzaC1yc2EtY2VydC12MDFAb3BlbnNzaC5jb20AAAAgJijksIFCSGz/KQ2x44l+Kl1l+3jsoOHT8Qe4+n7cwtMAAAADAQABAAABAQDVsNbzl5wh9Ckq9ZqBEG5eLuip24BClP26wEqFmt+4/VuxwgYxW1JNMW+/qYjytESHKp0xWoymnnFPdq1esMtDyiDPnR+I+Yk5Jj8JlusS4hDNy4FQFgwYvngAWckz+oOLNv6viLyq/ZdI0/qgvqIKMiTyg7ftD1GHFaJexDxNXirOfFRmiIuPYWbTmgPYHHzcaooQcXkOgANEz9hNb2SHC1lFQd+cKyiHkNtbNyBPYTKABfKNfH0a+Ygk7qf6pUbvtn9Y28mOyhIVtomgi017di25wpPxzw6C2fqulhgZFxHgv0nENJHLSmnXaacQpvi/CZTvPcgY19YJ7cRGpTnngNmJlMzhkNTUtOGJkMi00NWIwLWEwZGItMjYwM2U0NmQwNmU5LnRlbGVwb3J0LXByb3h5LWt1YmVybmV0ZXMtdGVsZXBvcnQtZGVtby5zYW1zdW5naW90YWRtaW4uY29tFV0ZWxlcG9ydC03Njg4ZjRiZDY4LW5nZjZwLnRlbGVwb3J0LXByb3h5LWt1YmVybmV0ZXMtdGVsZXBvcnQtZGVtby5zYW1zdW5naW90YWRtaW4uY29tAAAAGXRlbGVwb3J0LTc2ODhmNGJkNjgtbmdmNnAAAAAJbG9jYWxob3N0AAAACTEyNy4wLjAuMQAAAAM6OjEAAAA7dGVsZXBvcnQtcHJveHkta3ViZXJuZXRlcy10ZWxlcG9ydC1kZW1vLnNhbXN1bmdpb3RhZG1pbi5jb20AAAAocmVtb3RlLmt1YmUucHJveHkudGVsZXBvcnQuY2x1c3Rlci5sb2NhbAAAAABiYyvv//////////8AAAAAAAAAewAAABR4LXRlbGVwb3J0LWF1dGhvcml0eQAAAD8AAAA7dGVsZXBvcnQtcHJveHkta3ViZXJuZXRlcy10ZWxlcG9ydC1kZW1vLnNhbXN1bmdpb3RhZG1pbi5jb20AAAAPeC10ZWxlcG9ydC1yb2xlAAAACQAAAAVQcm94eQAAAAAAAAEXAAAAB3NzaC1yc2EAAAADAQABAAABAQDOMdQoCKeJQnYUPLbnITCW0UpOOmQIBVF1NNYTaWl6AUtEvT6jM2Q4SUt7RM7y31gn/GY6RFIqQv9jOH0nJws8evB17VtGq9+LADOrnpAEj0exLdh+P9sumbYvFxpjyslDeEjd2SPRP8XueIJSBtkbGa4akoHHzjJzRQFG2xS4oFUBuEV3A6LD8ip/pL7FZS+kiVOQRqkc2pNQk9tQ/GYwP/Qz83UpT41BkfsbGhB7Mlu/o0LO72sv+ciiFmJIOnOWlHvHsKqFOJ+DdYnlvv9+8VCXGXR0O26YNyYdtkNxpjZjM9F+DokefWF3TMa0KjqVRDAusJtHsMqJqsJa3ChTAAABFAAAAAxyc2Etc2hhMi01MTIAAAEAYQ6+loB7Paln3tnu59qB5jHHUzoAz9+w5VvM+djSniUleWuoCMyiiVP/+Y4uI2n/mbfLX2naU7YEeBYzdODtbgt6zISNH7tVJbdstjhKzmKQQF5+fGWrXmWG7H6y0+vnrkuQUQ8yPv4nWx4pvQYzP3rW5ZIDcOd25Xhjab8cvaASt9dLYJ3KFtWmCSYib1iaOierVQ7Zo/bGxss8PxeL6uV2m+bFWmj/JwJYSmmZsBmCuvQwz3W1KU8MEMR2a2UABn3ABvcaJQWrL5swDRhcjM7w7UdOInuW96cXdQbD+ZLKoW4oAS3ZNL2091QhqglZ5rDFSiOlTu0UEsOjRacYgQ==\n." client/keyagent.go:365
DEBU [KEYAGENT]  Validated host teleport-proxy-kubernetes-teleport-demo.testpoc.com:3023. client/keyagent.go:371
INFO [CLIENT]    Successful auth with proxy teleport-proxy-kubernetes-teleport-demo.testpoc.com:3023. client/api.go:2273
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]" valid until "2022-04-28 20:35:01 +0000 UTC". client/keystore.go:307
DEBU [CLIENT]    Client  is connecting to auth server on cluster "teleport-proxy-kubernetes-teleport-demo.testpoc.com". client/client.go:969
DEBU [CLIENT]    Client  is connecting to auth server on cluster "teleport-proxy-kubernetes-teleport-demo.testpoc.com". client/client.go:969
DEBU [KEYAGENT]  Adding CA key for teleport-proxy-kubernetes-teleport-demo.testpoc.com client/keyagent.go:313
DEBU [KEYSTORE]  Adding known host teleport-proxy-kubernetes-teleport-demo.testpoc.com with proxy teleport-proxy-kubernetes-teleport-demo.testpoc.com and key: SHA256:N086bK3P7IOd1i4yZ1+eTFJwAzm9x0jXYv7MjCz4oJ0 client/keystore.go:573
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]" valid until "2022-04-28 20:35:01 +0000 UTC". client/keystore.go:307
INFO [CLIENT]    Connecting proxy=teleport-proxy-kubernetes-teleport-demo.testpoc.com:3023 login="root" client/api.go:2266
DEBU             No valid environment variables found. client/proxy.go:116
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:268
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]" valid until "2022-04-28 20:35:01 +0000 UTC". client/keystore.go:307
DEBU [KEYAGENT]  "Checking key: [email protected]/KQ2x44l+Kl1l+3jsoOHT8Qe4+n7cwtMAAAADAQABAAABAQDVsNbzl5wh9Ckq9ZqBEG5eLuip24BClP26wEqFmt+4/VuxwgYxW1JNMW+/qYjytESHKp0xWoymnnFPdq1esMtDyiDPnR+I+Yk5Jj8JlusS4hDNy4FQFgwYvngAWckz+oOLNv6viLyq/ZdI0/qgvqIKMiTyg7ftD1GHFaJexDxNXirOfFRmiIuPYWbTmgPYHHzcaooQcXkOgANEz9hNb2SHC1lFQd+cKyiHkNtbNyBPYTKABfKNfH0a+Ygk7qf6pUbvtn9Y28mOyhIVtomgi017di25wpPxzw6C2fqulhgZFxHgv0nENJHLSmnXaacQpvi/CZTvPcgY19YJ7cRGpTnnNTUtOGJkMi00NWIwLWEwZGItMjYwM2U0NmQwNmU5LnRlbGVwb3J0LXByb3h5LWt1YmVybmV0ZXMtdGVsZXBvcnQtZGVtby5zYW1zdW5naW90YWRtaW4uY29tAAAAJDZiZTM4ZDU1LThiZDItNDViMC1hMGRiLTI2MDNlNDZkMDZlOQAAAFV0ZWxlcG9ydC03Njg4ZjRiZDY4LW5nZjZwLnRlbGVwb3J0LXByb3h5LWt1YmVybmV0ZXMtdGVsZXBvcnQtZGVtby5zYW1zdW5naW90YWRtaW4uY29tAAAAGXRlbGVwb3J0LTc2ODhmNGJkNjgtbmdmNnAAAAAJbG9jYWxob3N0AAAACTEyNy4wLjAuMQAAAAM6OjEAAAA7dGVsZXBvcnQtcHJveHkta3ViZXJuZXRlcy10ZWxlcG9ydC1kZW1vLnNhbXN1bmdpb3RhZG1pbi5jb20AAAAocmVtb3RlLmt1YmUucHJveHkudGVsZXBvcnQuY2x1c3Rlci5sb2NhbAAAAABiYyvv//////////8AAAAAAAAAewAAABR4LXRlbGVwb3J0LWF1dGhvcml0eQAAAD8AAAA7dGVsZXBvcnQtcHJveHkta3ViZXJuZXRlcy10ZWxlcG9ydC1kZW1vLnNhbXN1bmdpb3RhZG1pbi5jb20AAAAPeC10ZWxlcG9ydC1yb2xlAAAACQAAAAVQcm94eQAAAAAAAAEXAAAAB3NzaC1yc2EAAAADAQABAAABAQDOMdQoCKeJQnYUPLbnITCW0UpOOmQIBVF1NNYTaWl6AUtEvT6jM2Q4SUt7RM7y31gn/GY6RFIqQv9jOH0nJws8evB17VtGq9+LADOrnpAEj0exLdh+P9sumbYvFxpjyslDeEjd2SPRP8XueIJSBtkbGa4akoHHzjJzRQFG2xS4oFUBuEV3A6LD8ip/pL7FZS+kiVOQRqkc2pNQk9tQ/GYwP/Qz83UpT41BkfsbGhB7Mlu/o0LO72sv+ciiFmJIOnOWlHvHsKqFOJ+DdYnlvv9+8VCXGXR0O26YNyYdtkNxpjZjM9F+DokefWF3TMa0KjqVRDAusJtHsMqJqsJa3ChTAAABFAAAAAxyc2Etc2hhMi01MTIAAAEAYQ6+loB7Paln3tnu59qB5jHHUzoAz9+w5VvM+djSniUleWuoCMyiiVP/+Y4uI2n/mbfLX2naU7YEeBYzdODtbgt6zISNH7tVJbdstjhKzmKQQF5+fGWrXmWG7H6y0+vnrkuQUQ8yPv4nWx4pvQYzP3rW5ZIDcOd25Xhjab8cvaASt9dLYJ3KFtWmCSYib1iaOierVQ7Zo/bGxss8PxeL6uV2m+bFWmj/JwJYSmmZsBmCuvQwz3W1KU8MEMR2a2UABn3ABvcaJQWrL5swDRhcjM7w7UdOInuW96cXdQbD+ZLKoW4oAS3ZNL2091QhqglZ5rDFSiOlTu0UEsOjRacYgQ==\n." client/keyagent.go:365
DEBU [KEYAGENT]  Validated host teleport-proxy-kubernetes-teleport-demo.testpoc.com:3023. client/keyagent.go:371
INFO [CLIENT]    Successful auth with proxy teleport-proxy-kubernetes-teleport-demo.testpoc.com:3023. client/api.go:2273
DEBU [CLIENT]    Found clusters: [{"name":"teleport-proxy-kubernetes-teleport-demo.testpoc.com","lastconnected":"2022-04-28T12:35:04.730479396Z","status":"online"}] client/client.go:127
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]" valid until "2022-04-28 20:35:01 +0000 UTC". client/keystore.go:307
DEBU [CLIENT]    Client  is connecting to auth server on cluster "teleport-proxy-kubernetes-teleport-demo.testpoc.com". client/client.go:969
DEBU [CLIENT]    Client  is connecting to auth server on cluster "teleport-proxy-kubernetes-teleport-demo.testpoc.com". client/client.go:969
INFO [CLIENT]    Connecting proxy=teleport-proxy-kubernetes-teleport-demo.testpoc.com:3023 login="root" client/api.go:2266
DEBU             No valid environment variables found. client/proxy.go:116
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:268
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]" valid until "2022-04-28 20:35:01 +0000 UTC". client/keystore.go:307
DEBU [KEYAGENT]  "Checking key: [email protected] AAAAHHNzaC1yc2EtY2VydC12MDFAb3BlbnNzaC5jb20AAAAgJijksIFCSGz/KQ2x44l+Kl1l+3jsoOHT8Qe4+n7cwtMAAAADAQABAAABAQDVsNbzl5wh9Ckq9ZqBEG5eLuip24BClP26wEqFmt+4/VuxwgYxW1JNMW+/qYjytESHKp0xWoymnnFPdq1esMtDyiDPnR+I+Yk5Jj8JlusS4hDNy4FQFgwYvngAWckz+oOLNv6viLyq/ZdI0/qgvqIKMiTyg7ftD1GHFaJexDxNXirOfFRmiIuPYWbTmgPYHHzcaooQcXkOgANEz9hNb2SHC1lFQd+cKyiHkNtbNyBPYTKABfKNfH0a+Ygk7qf6pUbvtn9Y28mOyhIVtomgi017di25wpPxzw6C2fqulhgZFxHgv0nENJHLSmnXaacQpvi/CZTvPcgY19YJ7cRGpTnnBgNmJlMzhkNTUtOGJkMi00NWIwLWEwZGItMjYwM2U0NmQwNmU5LnRlbGVwb3J0LXByb3h5LWt1YmVybmV0ZXMtdGVsZXBvcnQtZGVtby5zYW1zdW5naW90YWRtaW4uY29tAAAAJDZiZTM4ZDU1LThiZDItNDViMC1hMGRiLTI2MDNlNDZkMDZlOQAAAFV0ZWxlcG9ydC03Njg4ZjRiZDY4LW5nZjZwLnRlbGVwb3J0LXByb3h5LWt1YmVybmV0ZXMtdGVsZXBvcnQtZGVtby5zYW1zdW5naW90YWRtaW4uY29tAAAAGXRlbGVwb3J0LTc2ODhmNGJkNjgtbmdmNnAAAAAJbG9jYWxob3N0AAAACTEyNy4wLjAuMQAAAAM6OjEAAAA7dGVsZXBvcnQtcHJveHkta3ViZXJuZXRlcy10ZWxlcG9ydC1kZW1vLnNhbXN1bmdpb3RhZG1pbi5jb20AAAAocmVtb3RlLmt1YmUucHJveHkudGVsZXBvcnQuY2x1c3Rlci5sb2NhbAAAAABiYyvv//////////8AAAAAAAAAewAAABR4LXRlbGVwb3J0LWF1dGhvcml0eQAAAD8AAAA7dGVsZXBvcnQtcHJveHkta3ViZXJuZXRlcy10ZWxlcG9ydC1kZW1vLnNhbXN1bmdpb3RhZG1pbi5jb20AAAAPeC10ZWxlcG9ydC1yb2xlAAAACQAAAAVQcm94eQAAAAAAAAEXAAAAB3NzaC1yc2EAAAADAQABAAABAQDOMdQoCKeJQnYUPLbnITCW0UpOOmQIBVF1NNYTaWl6AUtEvT6jM2Q4SUt7RM7y31gn/GY6RFIqQv9jOH0nJws8evB17VtGq9+LADOrnpAEj0exLdh+P9sumbYvFxpjyslDeEjd2SPRP8XueIJSBtkbGa4akoHHzjJzRQFG2xS4oFUBuEV3A6LD8ip/pL7FZS+kiVOQRqkc2pNQk9tQ/GYwP/Qz83UpT41BkfsbGhB7Mlu/o0LO72sv+ciiFmJIOnOWlHvHsKqFOJ+DdYnlvv9+8VCXGXR0O26YNyYdtkNxpjZjM9F+DokefWF3TMa0KjqVRDAusJtHsMqJqsJa3ChTAAABFAAAAAxyc2Etc2hhMi01MTIAAAEAYQ6+loB7Paln3tnu59qB5jHHUzoAz9+w5VvM+djSniUleWuoCMyiiVP/+Y4uI2n/mbfLX2naU7YEeBYzdODtbgt6zISNH7tVJbdstjhKzmKQQF5+fGWrXmWG7H6y0+vnrkuQUQ8yPv4nWx4pvQYzP3rW5ZIDcOd25Xhjab8cvaASt9dLYJ3KFtWmCSYib1iaOierVQ7Zo/bGxss8PxeL6uV2m+bFWmj/JwJYSmmZsBmCuvQwz3W1KU8MEMR2a2UABn3ABvcaJQWrL5swDRhcjM7w7UdOInuW96cXdQbD+ZLKoW4oAS3ZNL2091QhqglZ5rDFSiOlTu0UEsOjRacYgQ==\n." client/keyagent.go:365
DEBU [KEYAGENT]  Validated host teleport-proxy-kubernetes-teleport-demo.testpoc.com:3023. client/keyagent.go:371
INFO [CLIENT]    Successful auth with proxy teleport-proxy-kubernetes-teleport-demo.testpoc.com:3023. client/api.go:2273
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]" valid until "2022-04-28 20:35:01 +0000 UTC". client/keystore.go:307
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]" valid until "2022-04-28 20:35:01 +0000 UTC". client/keystore.go:307
DEBU [CLIENT]    Client  is connecting to auth server on cluster "teleport-proxy-kubernetes-teleport-demo.testpoc.com". client/client.go:969
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]" valid until "2022-04-28 20:35:01 +0000 UTC". client/keystore.go:307
DEBU [KEYSTORE]  Reading certificates from path "/Users/anthony/.tsh/keys/teleport-proxy-kubernetes-teleport-demo.testpoc.com/[email protected]/teleport-proxy-kubernetes-teleport-demo.testpoc.com-cert.pub". `client/keystore.go:330

These are the logs from kubernetes :

2022-04-28T00:02:42Z INFO [KUBERNETE] Connection closed, returning leaseID:126 target:teleport-proxy-kubernetes-teleport-demo.testpocadmin.com:3024 reversetunnel/agent.go:558
2022-04-28T00:02:42Z WARN [KUBERNETE] Failed to close discovery channel:: EOF leaseID:126 target:teleport-proxy-kubernetes-teleport-demo.testpocadmin.com:3024 reversetunnel/agent.go:547
2022-04-28T00:02:42Z WARN [KUBERNETE] Unable to continue processioning requests: heartbeat: connection closed. leaseID:126 target:teleport-proxy-kubernetes-teleport-demo.testpocadmin.com:3024 reversetunnel/agent.go:433
2022-04-28T00:02:42Z WARN [KUBERNETE] Failed to close remote tunnel: close tcp 10.25.151.34:58508->X.XXX.XX.XXX:XXXX: use of closed network connection, local addr: 10.25.151.34:58508 remote addr: 3.130.61.196:3024 leaseID:126 target:[teleport-proxy-kubernetes-teleport-demo.testpocadmin.com](http://teleport-proxy-kubernetes-teleport-demo.testpocadmin.com/):XXXXreversetunnel/agent.go:401
2022-04-28T00:02:43Z INFO [KUBERNETE] Connected. addr:10.25.151.34:34652 remote-addr:1XXXXX leaseID:233 target:teleport-proxy-kubernetes-teleport-demo.testpocadmin.com:3024 reversetunnel/agent.go:409
2022-04-28T00:02:50Z INFO [KUBERNETE] Connected. addr:10.25.151.34:34752 remote-addr:XXXXX leaseID:234 target:teleport-proxy-kubernetes-teleport-demo.testpocadmin.com:3024 reversetunnel/agent.go:409