node_time_zone_offset_seconds - time_zone label does not update until node-exporter is restarted

[charliekang]

Host operating system: output of uname -a

Linux xxxxxxx 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.8.2 (branch: HEAD, revision: f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)
  build user:       root@03d440803209
  build date:       20240714-11:53:45
  go version:       go1.22.5
  platform:         linux/amd64
  tags:             unknown

node_exporter command line flags

default install

node_exporter log output

collector=time
collector=timex
collector=udp_queues
collector=uname
collector=vmstat
collector=watchdog
collector=xfs
collector=zfs

Are you running node_exporter in Docker?

k8s & baremetal

What did you do that produced an error?

Changed the timezone after node-exporter was started. Appears node-exporter needs to be restarted to re-read the timezone.

What did you expect to see?

After changing the timezone, node-exporter should show updated timezone.

Also waited 15+mins to see if there was delay to update

Have to restart node-exporter for it re-read the timezone change

note: yes, timezone should not change but debugging another issue we noticed node-exporter timezone does not update

What did you see instead?

## Initial check - UTC

# timedatectl
               Local time: Tue 2024-12-03 17:57:17 UTC
           Universal time: Tue 2024-12-03 17:57:17 UTC
                 RTC time: Tue 2024-12-03 17:57:38
                Time zone: UTC (UTC, +0000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

# curl localhost:9100/metrics | grep time_zone
# HELP node_time_zone_offset_seconds System time zone offset in seconds.
# TYPE node_time_zone_offset_seconds gauge
node_time_zone_offset_seconds{time_zone="UTC"} 0

## Change timezone - PST

# timedatectl set-timezone America/Los_Angeles

# timedatectl
               Local time: Tue 2024-12-03 09:58:42 PST
           Universal time: Tue 2024-12-03 17:58:42 UTC
                 RTC time: Tue 2024-12-03 17:59:03
                Time zone: America/Los_Angeles (PST, -0800)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

# curl -s localhost:9100/metrics | grep time_zone
# HELP node_time_zone_offset_seconds System time zone offset in seconds.
# TYPE node_time_zone_offset_seconds gauge
node_time_zone_offset_seconds{time_zone="UTC"} 0


# systemctl restart node_exporter

# curl -s localhost:9100/metrics | grep time_zone
# HELP node_time_zone_offset_seconds System time zone offset in seconds.
# TYPE node_time_zone_offset_seconds gauge
node_time_zone_offset_seconds{time_zone="PST"} -28800

[discordianfish]

We take this from time.Now():

node_exporter/collector/time.go

Line 68 in ff97e35

now := time.Now()

- surprised that this caches(?) this somehow

[charliekang]

Ahh, found an old issue: #2273

pointing to golang issue: golang/go#28020

[discordianfish]

OH right, well nothing we can (reasonably) do about it until this is fixed upstream.. :-/