Allow Configurable Default for Connection Upgrade Method in tsh

[programmerq]

Expected Behavior

Cluster administrators should be able to configure the default connection upgrade method for tsh on a per-cluster basis. That, or tsh should gracefully handle cases where the dual upgrade request fails and fallback to a single upgrade method like WebSocket, without requiring manual end-user configuration.

Current Behavior

The tsh client sends both Upgrade: websocket and Upgrade: alpn-ping headers for connection upgrades by default. This dual header approach can lead to failures in environments with strict network policies or non-supportive proxies. For example, a web application firewall or a strict layer 7 load balancer that rejects all non-WebSocket connection upgrade requests.

Currently, end-users must manually set the TELEPORT_TLS_ROUTING_CONN_UPGRADE_MODE environment variable for their Teleport client to use WebSocket-only upgrades. This is too cumbersome to expect every Teleport user of this cluster to do. The cluster should have a way to advertise that websocket upgrades are preferred.

Bug Details

Teleport Version

First reported by customer when using Teleport v15.4.19 (cluster) with tsh client v15.4.4.

The latest versions of tsh have this same behavior (current tsh is 17.2.6 at time of writing)

Recreation Steps

1. Set up a Teleport Cluster using Layer 7 LB with strict policies against dual upgrade headers. (Reject any connection with Upgrade: alpn-ping or multiple Upgrade headers.)
2. Attempt to use tsh to establish a connection without setting the TELEPORT_TLS_ROUTING_CONN_UPGRADE_MODE variable.
3. Note the failure of connection upgrade due to dual headers.
4. Attempt to use tsh to establish a connection when setting TELEPORT_TLS_ROUTING_CONN_UPGRADE_MODE=websocket
5. Note success when only the Upgrade: websocket header is sent by tsh.

Debug Logs

2025-02-07T16:33:07-05:00 DEBU [TSH] ...
ERROR REPORT:
Original Error: trace.aggregate connection error: desc = "transport: Error while dialing: failed to dial: failed to switch Protocols 403"
Stack Trace:
    github.com/gravitational/teleport/lib/client/api.go:4364 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToRootCluster
...
User Message: connection error: desc = "transport: Error while dialing: failed to dial: failed to switch Protocols 403"

[greedy52]

We have decided to drop the legacy upgrade mode in v18.

#52534 will be backported to v17.3.x so that there is only one Upgrade: websocket header unless the env var override for TELEPORT_TLS_ROUTING_CONN_UPGRADE_MODE=legacy is used.

In v18, client will only do WebSocket upgrade. TELEPORT_TLS_ROUTING_CONN_UPGRADE_MODE will have no effect. Server side legacy upgrade is kept to be compatible with v17 clients. Then server side legacy upgrade will get deprecated in v19 to complete the migration to WebSocket upgrade