How to change the schematic ID of a Talos node with secure boot enabled?

[astro-stan]

I am running a single node Talos (v1.11) test cluster with SecureBoot enabled and TPM encrypted STATE and EPHEMERAL partitions.

I am using official images generated from the Talos image factory, that use Siderolabs keys.

I wanted to remove a system extension from the install and I just found out (the hard way) that changing the schematic ID of the Talos install causes a secure boot violation. This left the node unable to boot into Talos, and while booting from a flash drive worked, it prevented the TPM from unsealing the keys for the disk encryption... I tried fiddling around with it, but in the end I had to nuke everything and start over, as I couldn't figure out how to recover from this situation.

If I understand correctly, this is by design and the documentation states that the UKI signing key and the PCR signing key must preserved, otherwise the node will not be able to boot with the new UKI and unlock the encrypted partitions.

However, what i do not understand is how to preserve these keys when upgrading talos or changing the talos schematic ID.

My questions are these:

• What is the procedure for upgrading or changing the schematic ID of a Talos node with SecureBoot enabled, without using custom keys?
• How to recover from a Talos upgrade which caused a secure boot violation without data loss?

[smira]

You just upgrade to another image from the Image Factory with a different schematic (SecureBoot one), they are all signed properly.

Talos does A/B on upgrades, so you can boot previous image if you have access to bootloader screen

[smira]

No, this is not expected.

You're doing something wrong, but I can't guess from your description.

Either the keys you rolled in are wrong, or you never actually rolled them in correctly.

Or the initial boot image has different keys.

[astro-stan]

Interesting.

I got the key from the docs (specifically this link). The only issue I had was that the UEFI interface did not accept the .pem, so I converted it into a .cer using an online tool (it was about a month ago, so I don't remember which).

Github does not let me attach a .cer directly so here is a base64 of it:

$ cat talos-signing-key.cer | base64
MIIFmzCCA4OgAwIBAgIUT6JayQaxpvpq2nc1KydtI/BUe+owDQYJKoZIhvcNAQELBQAwXTELMAkG
A1UEBhMCVVMxGTAXBgNVBAoMEFNpZGVybyBMYWJzIEluYy4xHzAdBgNVBAsMFlNpZGVybyBMYWJz
IFNlY3VyZUJvb3QxEjAQBgNVBAMMCXRhbG9zLmRldjAeFw0yMzExMjExNjE0MDVaFw0zMzExMTgx
NjE0MDVaMF0xCzAJBgNVBAYTAlVTMRkwFwYDVQQKDBBTaWRlcm8gTGFicyBJbmMuMR8wHQYDVQQL
DBZTaWRlcm8gTGFicyBTZWN1cmVCb290MRIwEAYDVQQDDAl0YWxvcy5kZXYwggIiMA0GCSqGSIb3
DQEBAQUAA4ICDwAwggIKAoICAQCtjFhOjgDjJHz/FV7SvlRoREWowRJMMNIoJQXLA3oAuAlv/JYx
+DzYTfAO8Wctn94nirkSf8iA3hByiXTluZjM8/4A+BQ3o/nj+/8b2seBxrURaSaSazDRWFMKRz/3
HhnhbSYPV/IT/10c24efdXzFGcU3bd8Ax6GbntPSJohWd6Teok+yZz0oLg3vdJEWu+3vDTUgz0jI
3Tj9NCZT7WiyY0RTNv5u7aca7l1tKoq7BV7gF1j1iV1C7AUI+rZPZn5mK/7oijcn1eDRs/QmbqSZ
q7z9U7WqhGtVyYhVyVs60zHLhRksd8vZRLNZqfRybbliujLfXiyzFt/TdZ09IR3HFQGmWa6iIien
xHYCrpROMhFO5wne6Y+2jZaP6Zy2+bk7GHESXYGqgVPWfSe6RWWHdR1JBruSuANi0wS7IMsCiFTC
yiILPfc6WmnNyfJagmqTF6OJVRHGnTfowYNTUk0mTBgWeFZNc4j6v6ftFG0UHlnnJ8jxM1vxGoYG
Fs+sIjSWoeRix+VZNbgBy/OrA8hgN+PwpOeVkIgWQbFIsPZiQgnYFEnggLKdtYZdcVhIHsLUKmcb
mcNdFwTEYo5a0SrOEHP4FO8C0f1cC2DpUX+TcwK0soWuReQk0nPf2rlUB4CqR+EYwecpywRFVQrM
/JcTpzqJhRZ3JdSkSZuMJa6WRwIDAQABo1MwUTAdBgNVHQ4EFgQU1G2G+J+BD4fYm/kP5kfnfCPL
lwUwHwYDVR0jBBgwFoAU1G2G+J+BD4fYm/kP5kfnfCPLlwUwDwYDVR0TAQH/BAUwAwEB/zANBgkq
hkiG9w0BAQsFAAOCAgEAjapogp9J1Lx60ZPi7E8hLpGRTebYH2GZzMSQzgHKvobOYDSbwATkzQus
F4fJd4mvX7e8bZClg+yCGp2mdCQRs9JSi6opxSdFhenct8Z4Rij1B+L6z4+ycysu9PVYEplgg2YX
o1PsuXB68R7FLnTbiIOtb356cIGevQLu4ubYpJYAuLLS4kGngGd0XYqqyYTLmV8JCKadKLFAWCLN
M8ufD5cX08utnGowE7vvwK/C2PhtrCyvUWjF9Sz2o0Y3uUlE2kN8tf/PMO8y92KH3HM340FIuj44
ZJbh9h5sLY7q59tIzNx4INQfzb4QVg8QHQUGuyD9wweWd8waNRDusJlg42SeAjoH+uvRhINAKTva
9Xjp1H/z1/JRZUeP+HniP/TzlueHUWPE1W8o79q7wNMO89fu3I+0VVgsLtuRi2v+ODD+4bGhn0q3
uzHN9FHuRW7Vc464EJd1VuVk9TSkF4cUJx4XnEK5M/3rBVV6jGjRjKQEBVIh8RJRDbJICJ72NXB4
iqLcu1YLDS/PsLBaz2WooFvxF/QiwTt8AehtweR9aeGfY5NeDJMZ+ikOn3jdebCmHTnXnb/j1FMy
KFe5gN/agCXK8mtHK36KqQe3ylOJWdQDu2IoCuTiip65XZn9Z414QfCOt/kmlfG3FXZuEtBcLLSg
HYUDlKj6MNlMhZSp00I=

Or the initial boot image has different keys.

I got the ISO from the image factory. Not sure what extensions I selected but I definitely clicked the "secure boot" checkmark as the ISO is called metal-amd64-secureboot.iso. Is there a way to get the schematic ID from the ISO?

[smira]

It doesn't matter which schematic ID you did. Talos ISOs with SecureBoot enabled can self-enroll the keys (there's a boot option).

But this discussion is not going anywhere - Talos SecureBoot works and it's tested, but you did some mistake/have some problem, and we can't guess why.

[astro-stan]

I think there is a misunderstanding here. I was not suggesting there is a bug in the Talos SecureBoot implementation, just stating that it didn't work for me and asked for help in troubleshooting the issue.

In any case, after several more hours of trial and error I have found the issue.

I decided to retry the reproduction steps with the latest available version - v1.11.3 (so far I've been using v1.11.0). I encountered the same issue, but I also noticed this during the upgrade process:

 user: warning: [2025-11-04T23:15:31.830092821Z]: 2025/11/04 23:15:32 sd-boot: found existing UKIs during install: [Talos-v1.11.3.efi]      
 user: warning: [2025-11-04T23:15:31.830097821Z]: 2025/11/04 23:15:32 copying /usr/install/amd64/vmlinuz.efi to /boot/EFI/EFI/Linux/Talos-v1.11.0.efi  

Huh? Why is it downgrading to v1.11.0? Well it turns out its not a bug, I should have just RTFM:

$ talosctl upgrade --help
Upgrade Talos on the target node
...
 -i, --image string               the container image to use for performing the install (default "ghcr.io/siderolabs/installer:v1.11.0")
...

So it seems talos upgrade will by default install an image without secure boot support regardless of what is specified in the node's machine config.

In contrast, my assumption/expectation of the upgrade workflow was:

• Apply a config with the image I want
• Run talos upgrade
• Done

In hindsight, manually adding the SideroLabs key as the PK, KEK, and Auth DB key—or enrolling it via the ISO in setup mode—worked for booting the ISO and initial install but failed after a talosctl upgrade. It should have been an obvious clue to the real issue, but I missed it at the time.

@smira 2 questions:

• Is there a reason why talos upgrade installs a seemingly random image unless the --image option is provided, given that the image is already recorded in the machine config for the node?

  I might be misunderstanding how the upgrade process is supposed to work but if --image must have a default, I'd expect it to be the image specified in the node's machine config, and if --insecure is provided, I'd expect to be mandatory to also provide --image.

• Should the Siderolabs key be enrolled in just Auth DB or does it also need to be in KEK and/or set as PK?

  I can confirm it works by having it just added to the Auth DB. Just in case I also added it as a KEK key. However, I noticed the ISO enrolls it as as the PK key in addition to adding it to the KEK and Auth DBs. Is this on purpose?

[smira]

talosctl upgrade always uses the image you specify in the command line, .machine.install.image is only used for the initial install, and yes, it should point to your proper SecureBoot image.

See documentation.

As for the auth DB, Talos provides a simple way, but you're free to enroll the keys the way it works for you. SecureBoot database is up to your UEFI firmware, not Talos.

[astro-stan]

Thanks for replying.

Out of curiosity, if this wasn't a test cluster, what would the recovery steps be in this situation:

• Talos node runs a secure boot image
• STATE and EPHEMERAL are TPM encrypted (assuming PCR 7 only)
• talos upgrade is executed with non-secure boot image
• Assuming there is only 1 node in the cluster or that all nodes were broken at the same time

In this situation:

• Node will not boot at all, so cannot use Talos's A/B recovery
• Booting from USB will boot the node but won't unseal the disks, thus node will be stuck in "booting" state

Is there a way to put the node into maintenance mode and upgrading it to a secure boot image without wiping the STATE and EPHEMERAL partitions? Or at least without having to wipe EPHEMERAL.

[smira]

you can always recover by selecting previous image in the bootloader menu, you can boot a different one from USB, etc.

as long as you boot a proper image, everything would unlock/decrypt without any intervention.

[astro-stan]

When i accidentally installed a non-secure boot image the node failed to start at all. And by that I mean the secure boot violation happens before it gets to the bootloader menu, thus cannot select the older image. And when booted from USB, the USB bootloader has 2 options:

• Boot Talos - this gets stuck in a booting state because the disks don't unseal
• Wipe system disk - wipes STATE and EPHEMERAL

I couldn't find a way to boot the node into maintenance without also wiping the partitions. Thus, I had to start over and accept the data loss.