Is machine.time.disabled safe on bare-metal machines?

[mi-bauer]

We are using Talos in environments that are air-gapped most of the time.
We had issues booting Talos even when using the bootTimeout setting (I am currently trying to reproduce this in a VM for a bug ticket). To make Talos boot, we set machine.time.disabled = true a few months ago. However, the RTC drifts out of sync over time. Since authentication is time-based, we have already encountered issues where we could not authenticate with talosctl on the first try due to a time mismatch. We were able to connect after several retries (I don’t know why, maybe because certificate refresh cycles sometimes align and sometimes don’t). But as time goes on, we will eventually be locked out permanently from systems with machine.time.disabled = true unless we correct it manually.

While we evaluate this issue, I have the following questions:

• Is machine.time.disabled actually intended for bare-metal installations?
• If so, is there any mechanism that prevents being locked out after some time (for example, a sync mechanism outside the boot process)?

[smira]

We do recommend running NTP sync one way or another.

The disabled setting might be fine if you run your own NTP daemon.

Talos allows you to configure an NTP server, so you can have one inside the air-gapped environment.

If the time gets out of sync, you might certainly encounter random issues across the cluster.

[smira]

I'm not sure from your question what you are looking for.

Hardware clock is RTC, but it's not related directly to time sync process. RTC is synced as part of time sync, but current time is ephemeral in general.

Talos allows to run your own NTP if you disable the Talos one.

[mi-bauer]

I understood from your reply that I have two options::

Option 1

• Run an NTP server on another device in the environment that must always be reachable.
• Point Talos to that NTP server using time.servers = ....

Option 2

• In Talos, set time.disabled = true.
• Set up a custom DaemonSet that syncs the hardware clock with time.cloudflare.com if internet access is available.

Hardware clock is RTC, but it's not related directly to time sync process. RTC is synced as part of time sync, but current time is ephemeral in general.
Talos allows to run your own NTP if you disable the Talos one.

If I understand you correctly, regarding Option 2, it would be sufficient to:

• Set time.disabled = true.
• Set up a custom DaemonSet that just syncs system time.
  • Talos will notice the system time change and will therefore automatically:
    • Update the time on the hardware clock.
    • Rotate certificates.

Did I understand you correctly?

[smira]

In general yes, but I don't understand whether you're air-gapped or having connectivity from time to time.

For option 2, you can also make Talos not block on NTP sync on boot, and it will try to do NTP sync best effort (when connectivity is available).

[mi-bauer]

In general yes, but I don't understand whether you're air-gapped or having connectivity from time to time.

It is air-gapped most of the time and must also be able to boot air-gapped. Internet access may occasionally be established for maintenance and updates (let’s say twice a year, though the timing can vary). This is when a time sync must occur.

For option 2, you can also make Talos not block on NTP sync on boot, and it will try to do NTP sync best effort (when connectivity is available).

I assume you are referring to the bootTimeout setting, correct? That would be a good option, but this is the part that is currently not working as expected. I am working on a setup where I can reproduce the issue.

[smira]

Yes, I mean bootTimeout if you want it to boot without networking/NTP available, but still keep NTP running in the background.