Configuring Break Glass Root Access via SSH in the Event of Teleport Agent or Control Plane Failure

[deusxanima]

This guide will walk you through configuring "break glass" access to your servers via SSH, to be used in emergency scenarios when:

1. The Teleport agent on a server has crashed, gone offline, or become unusable.
2. The Teleport control plane is down and cannot be used to access systems, and this procedure is necessary to fix it.

Prerequisites:

• OpenSSH version 6.9 or above on your local machine. You can verify your OpenSSH version by running:

ssh -V

• Teleport version 16.4.2 or above in a running cluster. If you're new to Teleport, sign up for a free trial or set up a demo environment.
• The tctl admin tool and tsh client tool. Visit Teleport Installation for instructions.
• A Linux host with OpenSSH server (sshd) version 7.4 or above.

Step 1/5. Configure sshd to trust the Teleport CA

For break glass access, the OpenSSH server must be configured to trust client certificates issued by the Teleport Certificate Authority (CA).

1. Export the Teleport CA public key:

On the OpenSSH server, run the following command. Replace proxy.example.com with the address of your Teleport Proxy Service:

export KEY=$(curl 'https://proxy.example.com/webapi/auth/export?type=openssh' | sed "s/cert-authority\ //")

Make the public key accessible to sshd:

sudo bash -c "echo \"$KEY\" > /etc/ssh/teleport_openssh_ca.pub"
sudo bash -c "echo 'TrustedUserCAKeys /etc/ssh/teleport_openssh_ca.pub' >> /etc/ssh/sshd_config"

Restart the sshd service:

sudo systemctl restart sshd

Now, sshd will trust users who present a Teleport-issued SSH certificate.

Step 2/5. Create a limited breakglass user on OpenSSH server

1. Create the breakglass user:

sudo adduser breakglass

Set a password for breakglass when prompted.

2. Open the sudoers File for Editing:

sudo visudo

3. Grant limited sudo permissions:

Add the following line to allow the breakglass user to run only specific commands with sudo (e.g.,cat, ls, journalctl, and systemctl):

breakglass ALL=(ALL) /bin/cat, /bin/ls, /bin/journalctl, /bin/systemctl

ALL: This allows the rule to apply on all hosts (for local systems, this will be fine).
(ALL): This allows the user to run the commands as any user (typically root by default).
The paths of the commands (/bin/cat, /bin/ls, /bin/journalctl, /bin/systemctl) must be exact and point to the full binary locations.

4. Save and Exit visudo:

If you're using nano as your sudo editor, press Ctrl+X, then Y, and press Enter.

Step 3/5. Create breakglass role & user in Teleport

1. Create breakglass Teleport role:

Define the role in a file named breakglass-role.yaml

kind: role
metadata:
  name: breakglass
spec:
  allow:
    logins:
    - breakglass
    node_labels:
      '*': '*'
  deny: {}
  options:
    cert_format: standard
    forward_agent: false
    max_session_ttl: 1h0m0s
    port_forwarding: true
    ssh_file_copy: true
version: v7

Create the role by applying with:

tctl create -f breakglass-role.yaml

2. Create the breakglass Teleport user:

tctl users add --roles=breakglass breakglass

We don't need to worry about setting a password for this user as it'll only ever be used in the context of OpenSSH break glass authentication.

3. Grant impersonate permissions to ssh cert issuer user:

To avoid having to directly log into Teleport as breakglass, grant impersonation permissions to an admin or automation user:

impersonate:
    roles:
    - breakglass
    users:
    - breakglass

Step 4/5. Generate breakglass ssh key and cert

Run the following after logging into Teleport via tsh login as the impersonating user:

tctl auth sign --ttl 10h --user=breakglass --out=/safe/path/breakglass --format openssh

This command will create an ssh private key and a Teleport CA-signed certificate to be used by your client when authenticating with the OpenSSH server.

Store the private key securely, such as in a vault, only access if needed to implement break glass procedures. Rotate both keys and certificates regularly.

Step 5/5: Access OpenSSH server using Teleport CA

1. Add the following to your ~/.ssh/config (or equivalent), making sure to update with your HostName and cert paths:

Host sshd-server
    HostName sshd-server.example.com
    User breakglass
    IdentityFile /secure/path/breakglass
    CertificateFile /secure/path/breakglass-cert.pub

2. Access the server

Now, you can securely access the OpenSSH server using the breakglass user:

ssh sshd-server

---

The method described above ensures you can securely access your servers in emergency scenarios using the Teleport CA, even if Teleport's agents or control plane are down.

[Zathiel]

Hey @deusxanima

Thanks for this one

Is it an approved method by the team ?

All security aspect of using this in production have been tested ?

Regards

[deusxanima]

Hi @Zathiel , meant to reply last week but fat-fingered it. After some discussion internally we've agreed to take this guide, clean it up some more, add a few additional caveats, and publish it as part of our official documentation to address your questions.

[Zathiel]

Hey @deusxanima

Thanks a lot let me know where it's gonna be publish so I can have a look deeper

You can confirm it's an approved and secured solution during this time