Update always fails on wake from suspend

[accron-1]

After waking up the computer from a suspend, the update service always fails.

Output from systemctl status rpm-ostreed-automatic.service:

rpm-ostreed-automatic.service - rpm-ostree Automatic Update
     Loaded: loaded (/usr/lib/systemd/system/rpm-ostreed-automatic.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
             /usr/lib/systemd/system/rpm-ostreed-automatic.service.d
             └─override.conf
     Active: failed (Result: exit-code) since Sat 2025-05-31 11:57:32 CEST; 33s ago
   Duration: 655ms
 Invocation: a3832540cd6244e4bb04c540687820f2
TriggeredBy: ● rpm-ostreed-automatic.timer
       Docs: man:rpm-ostree(1)
             man:rpm-ostreed.conf(5)
    Process: 101305 ExecCondition=/bin/bash -c [[ "$(busctl get-property org.freedesktop.NetworkManager /org/freedesktop/NetworkManager org.freedesktop.NetworkManager Metered | cut -c 3-)" == @(2|4) ]] (code=exited, status=0/SUCCESS)
    Process: 101331 ExecStart=rpm-ostree upgrade --quiet --trigger-automatic-update-policy (code=exited, status=1/FAILURE)
   Main PID: 101331 (code=exited, status=1/FAILURE)
   Mem peak: 9.4M
        CPU: 58ms

May 31 11:57:31 aurora systemd[1]: Starting rpm-ostreed-automatic.service - rpm-ostree Automatic Update...
May 31 11:57:31 aurora systemd[1]: Started rpm-ostreed-automatic.service - rpm-ostree Automatic Update.
May 31 11:57:32 aurora rpm-ostree[101331]: error: Creating importer: failed to invoke method OpenImage: failed to invoke method OpenImage: pinging container registry ghcr.io: Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: Temporary failure in name resolution
May 31 11:57:32 aurora systemd[1]: rpm-ostreed-automatic.service: Main process exited, code=exited, status=1/FAILURE
May 31 11:57:32 aurora systemd[1]: rpm-ostreed-automatic.service: Failed with result 'exit-code'.

The above output is just a few seconds after waking up my PC, as shown in the 'Active' field.

Uupd tries to make a network request before an internet connection has been established.
Note that I'm on a wired connection, not Wifi.

This means that the auto update service never runs successfully for me as it always scheduled for 4am, while my PC is asleep.

[gerblesh]

Hi! Sorry for the late response here, rpm-ostreed-automatic.service is not supposed to be enabled, instead the service for uupd is simply uupd.service or uupd.timer. Let me know if you're able to send the logs for these systemd services, uupd should not be shipping with this service/timer enabled as they serve the same purpose. Thanks!

[gerblesh]

systemd preconditions would probably also largely solve this issue, forcing a wake from suspend when the service starts maybe or making sure the computer isn't suspended during the update

[accron-1]

Thanks for the reply!
As far as I remember I didn't manually enable rpm-ostreed-automatic.service, I think it was automatically enabled when the "old" service was deprecated.

Output from systemctl status uupd.service:

uupd.service - Universal Blue Update Oneshot Service
     Loaded: loaded (/usr/lib/systemd/system/uupd.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead) since Fri 2025-06-20 10:05:13 CEST; 5min ago
 Invocation: ea52d2ba031944e09296a2f0c5f8c55e
TriggeredBy: ● uupd.timer
    Process: 81790 ExecStart=/usr/bin/uupd --disable-module-distrobox --log-level debug --json --hw-check (code=exited, status=0/SUCCESS)
   Main PID: 81790 (code=exited, status=0/SUCCESS)
   Mem peak: 2.3M
        CPU: 9ms

Jun 20 10:05:13 aurora systemd[1]: Starting uupd.service - Universal Blue Update Oneshot Service...
Jun 20 10:05:13 aurora uupd[81790]: {"level":"ERROR","msg":"Hardware checks failed","error":"Network, returned error: network not online"}
Jun 20 10:05:13 aurora systemd[1]: uupd.service: Deactivated successfully.
Jun 20 10:05:13 aurora systemd[1]: Finished uupd.service - Universal Blue Update Oneshot Service.

Note that the same network error is there.

Output from systemctl status uupd.timer:

uupd.timer - Auto Update System Timer For Universal Blue
     Loaded: loaded (/usr/lib/systemd/system/uupd.timer; enabled; preset: disabled)
     Active: active (waiting) since Mon 2025-06-16 08:53:06 CEST; 4 days ago
 Invocation: b2f0978828324dc78822709c24875cb0
    Trigger: Fri 2025-06-20 16:05:13 CEST; 5h 53min left
   Triggers: ● uupd.service

Jun 16 08:53:06 aurora systemd[1]: Started uupd.timer - Auto Update System Timer For Universal Blue.

Let me know if you need any other logs!

EDIT: forcing a wake from suspend is probably not a good idea, as it would leave the PC on unattended for potentially many hours.
I have no idea how this works, but is it possible to set a delay for a few seconds (say 30 sec), when the trigger runs, before the actual code is executed, to make sure everything has "settled"?

[gerblesh]

Ok so I made the service auto-retry on failure which means that it will try 3 times before stopping, let me know if this solves your issue, you can try the updated service file from the PR

[gerblesh]

I'll probably also make a PR that disables rpm-ostreed-automatic.service for Bluefin/Aurora if needed

[inffy]

I'll probably also make a PR that disables rpm-ostreed-automatic.service for Bluefin/Aurora if needed

Aurora should habe the old timers disabled already
https://github.com/ublue-os/aurora/blob/31e718510beb29af6a0073dd35d55a3691efac65/build_files/base/17-cleanup.sh#L34

[accron-1]

I'm not sure what's the correct way of testing it, but I created an override file with the settings in the PR and restarted the service.
After override the output of systemctl cat uupd.service was:

[Unit]
Description=Universal Blue Update Oneshot Service

[Service]
Type=oneshot
# Configure the checks using env vars
#Environment="UUPD_BATTERY_MIN_PERCENT=20"
#Environment="UUPD_NETWORK_MAX_BYTES=500000"
#Environment="UUPD_MEMORY_MAX_PERCENT=90"
#Environment="UUPD_CPU_MAX_LOAD_PERCENT=50"
ExecStart=/usr/bin/uupd --disable-module-distrobox --log-level debug --json --hw-check
SELinuxContext=system_u:unconfined_r:unconfined_t:s0

# /usr/lib/systemd/system/service.d/10-timeout-abort.conf
# This file is part of the systemd package.
# See https://fedoraproject.org/wiki/Changes/Shorter_Shutdown_Timer.
#
# To facilitate debugging when a service fails to stop cleanly,
# TimeoutStopFailureMode=abort is set to "crash" services that fail to stop in
# the time allotted. This will cause the service to be terminated with SIGABRT
# and a coredump to be generated.
#
# To undo this configuration change, create a mask file:
#   sudo mkdir -p /etc/systemd/system/service.d
#   sudo ln -sv /dev/null /etc/systemd/system/service.d/10-timeout-abort.conf

[Service]
TimeoutStopFailureMode=abort

# /etc/systemd/system/uupd.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/uupd --log-level debug --json --hw-check
Restart=on-failure
RestartSec=60s
StartLimitIntervalSec=600
StartLimitBurst=3
SELinuxContext=system_u:unconfined_r:unconfined_t:s0

I just woke up my PC and did systemctl status uupd.service less than a min after wake up, and again several minutes later. The output didn't change:

     Loaded: loaded (/usr/lib/systemd/system/uupd.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead) since Mon 2025-06-23 06:10:19 CEST; 22min ago
 Invocation: 1886524c8b4a4232aed2423319b39d77
TriggeredBy: ● uupd.timer
   Main PID: 131558 (code=exited, status=0/SUCCESS)
   Mem peak: 2.7M
        CPU: 8ms

Jun 23 06:10:19 aurora systemd[1]: Starting uupd.service - Universal Blue Update Oneshot Service...
Jun 23 06:10:19 aurora uupd[131558]: {"level":"ERROR","msg":"Hardware checks failed","error":"Network, returned error: network not online"}
Jun 23 06:10:19 aurora systemd[1]: uupd.service: Deactivated successfully.
Jun 23 06:10:19 aurora systemd[1]: Finished uupd.service - Universal Blue Update Oneshot Service.

I double checked with journalctl -u uupd.service after 10 min but no updates to the log was made after the above.

So if I tested it correctly, I don't think it worked.

EDIT:
Just a quick thought for a faster feedback loop: wouldn't it be possible to confirm this on your side by disconnecting from Wifi/unplugging the cable, then manually invoking uupd (which should fail with the same/similar error), and then connecting to Wifi again before the next attempt 60s later?

[gerblesh]

tested locally and it works for me, in the override you sent it seems like the ExecStart= is set to a blank value? Not sure if this affecting things but disconnecting and reconnecting works for me.

[accron-1]

If the test I suggested works for you then it seems to me the fix should resolve my issue too. As I stated I'm not sure I tested it correctly.

Since it makes the service more robust overall I think this feature should be merged, regardless. I can then check with the released version.

How long would it be until it's released?

[gerblesh]

it'll be released as a minor version since it's such a small change to the service, it may take a little bit to propagate down to aurora however. My tests just tested it when manually disconnecting and reconnecting from wifi, it's a little hard for me to totally replicate your setup and since you were the original issue reporter I figured it might be worth you testing. I agree it does make the service more robust but it would be nice to get some confirmation that the issue is solved. I'll go ahead and merge it, but I'll keep this issue open until it's solved. Thanks for helping me work this out

[accron-1]

Great, thanks for keeping the issue open! Please ping me here when it's released and I'll immediately check it.

And thanks for looking into this!