Upgrade uyuni 2024.12 -> 2025.05 - postgresql container fails to start

[bjin01]

Problem description

Customer did
OS is opensuse leap 15.6

zypper up
reboot
mgradm upgrade podman

The password for CA key they entered was wrong, it was not the original CA Key pwd they used for the initial uyuni installation and first attempt of Upgrade failed.

Customer re-run

mgradm upgrade podman

The upgrade was successful but db container could not be started.

mgradm restart 
12:17PM INF Welcome to mgradm
12:17PM INF Executing command: restart
Error: failed to restart systemd uyuni-db.service: exit status 1; failed to restart systemd uyuni-server.service: exit status 1

In journal logs of uyuni-db.service I found those:

Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Scheduled restart job, restart counter is at 3.
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Starting Uyuni database container service...
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p (sh)[28613]: uyuni-db.service: Referenced but unset environment variable evaluates to an empty string: PODMAN_EXTRA_ARGS
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p sh[28613]: Error: no secret with name or id "uyuni-db-ca": no such secret
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p podman[28613]: 2025-06-12 11:50:48.839527341 +0000 UTC m=+0.019391392 image pull c6e8579ef92ed1b64525c1876cf5fb846d3129db23b01a33bd1ff7630216b>
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Control process exited, code=exited, status=125/n/a
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Failed with result 'exit-code'.
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Failed to start Uyuni database container service.
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Scheduled restart job, restart counter is at 4.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Starting Uyuni database container service...
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p (sh)[28718]: uyuni-db.service: Referenced but unset environment variable evaluates to an empty string: PODMAN_EXTRA_ARGS
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p sh[28718]: Error: no secret with name or id "uyuni-db-ca": no such secret
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p podman[28718]: 2025-06-12 11:50:49.068508073 +0000 UTC m=+0.018436119 image pull c6e8579ef92ed1b64525c1876cf5fb846d3129db23b01a33bd1ff7630216b>
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Control process exited, code=exited, status=125/n/a
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Failed with result 'exit-code'.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Failed to start Uyuni database container service.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Scheduled restart job, restart counter is at 5.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Start request repeated too quickly.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Failed with result 'exit-code'.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Failed to start Uyuni database container service.
Jun 12 12:04:20 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Starting Uyuni database container service...
Jun 12 12:04:20 vm-uyuni-app-01-prod-axsa-m04p (sh)[676]: uyuni-db.service: Referenced but unset environment variable evaluates to an empty string: PODMAN_EXTRA_ARGS
Jun 12 12:04:20 vm-uyuni-app-01-prod-axsa-m04p podman[676]: 2025-06-12 12:04:20.487354596 +0000 UTC m=+0.032102513 image pull c6e8579ef92ed1b64525c1876cf5fb846d3129db23b01a33bd1ff7630216b76>
Jun 12 12:04:20 vm-uyuni-app-01-prod-axsa-m04p sh[676]: Error: no secret with name or id "uyuni-db-ca": no such secret

uyuni-db.service unit file:

# /etc/systemd/system/uyuni-db.service
# uyuni-db-server.service, generated by mgradm
# Use an uyuni-db-server.service.d/local.conf file to override

[Unit]
Description=Uyuni database container service
Wants=network.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStartPre=/bin/rm -f %t/%n.pid %t/%n.ctr-id
ExecStartPre=/usr/bin/podman rm --ignore --force -t 10 uyuni-db
ExecStart=/bin/sh -c '/usr/bin/podman run \
        --conmon-pidfile %t/%n.pid \
        --cidfile=%t/%n.ctr-id \
        --cgroups=no-conmon \
        --shm-size=0 \
        --shm-size-systemd=0 \
        --sdnotify=conmon \
        -d \
        --name uyuni-db \
        --hostname uyuni-db.mgr.internal \
        --network-alias db \
        --network-alias reportdb \
        --secret uyuni-db-ca,type=mount,target=/etc/pki/trust/anchors/DB-RHN-ORG-TRUSTED-SSL-CERT \
        --secret uyuni-db-key,type=mount,uid=999,mode=0400,target=/etc/pki/tls/private/pg-spacewalk.key \
        --secret uyuni-db-cert,type=mount,target=/etc/pki/tls/certs/spacewalk.crt \
        --secret uyuni-db-admin-user,type=env,target=POSTGRES_USER \
        --secret uyuni-db-admin-pass,type=env,target=POSTGRES_PASSWORD \
        --secret uyuni-db-user,type=env,target=MANAGER_USER \
        --secret uyuni-db-pass,type=env,target=MANAGER_PASS \
        --secret uyuni-reportdb-user,type=env,target=REPORT_DB_USER \
        --secret uyuni-reportdb-pass,type=env,target=REPORT_DB_PASS \
        -p 5432:5432 \
        -v var-pgsql:/var/lib/pgsql/data \
        --network uyuni \
        ${PODMAN_EXTRA_ARGS} ${UYUNI_IMAGE}'
ExecStop=/usr/bin/podman stop \
        --ignore -t 10 \
        --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
        -f \
        --ignore -t 10 \
        --cidfile=%t/%n.ctr-id

PIDFile=%t/%n.pid
TimeoutStopSec=180
TimeoutStartSec=900
Type=forking

[Install]
WantedBy=multi-user.target default.target

# /etc/systemd/system/uyuni-db.service.d/generated.conf
# This file is generated by mgradm and will be overwritten during upgrades.
# Custom configuration should go in another .conf file in the same folder.

[Service]
Environment=UYUNI_IMAGE=registry.opensuse.org/uyuni/server-postgresql:latest

Steps to reproduce

1. mgradm restart

...

Uyuni version

2025.05

Uyuni proxy version (if used)

NA

uyuni-tools version used to deploy/manage the server or proxy (if relevant)

latest

Useful logs

Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Scheduled restart job, restart counter is at 3.
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Starting Uyuni database container service...
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p (sh)[28613]: uyuni-db.service: Referenced but unset environment variable evaluates to an empty string: PODMAN_EXTRA_ARGS
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p sh[28613]: Error: no secret with name or id "uyuni-db-ca": no such secret
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p podman[28613]: 2025-06-12 11:50:48.839527341 +0000 UTC m=+0.019391392 image pull c6e8579ef92ed1b64525c1876cf5fb846d3129db23b01a33bd1ff7630216b>
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Control process exited, code=exited, status=125/n/a
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Failed with result 'exit-code'.
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Failed to start Uyuni database container service.
Jun 12 11:50:48 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Scheduled restart job, restart counter is at 4.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Starting Uyuni database container service...
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p (sh)[28718]: uyuni-db.service: Referenced but unset environment variable evaluates to an empty string: PODMAN_EXTRA_ARGS
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p sh[28718]: Error: no secret with name or id "uyuni-db-ca": no such secret
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p podman[28718]: 2025-06-12 11:50:49.068508073 +0000 UTC m=+0.018436119 image pull c6e8579ef92ed1b64525c1876cf5fb846d3129db23b01a33bd1ff7630216b>
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Control process exited, code=exited, status=125/n/a
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Failed with result 'exit-code'.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Failed to start Uyuni database container service.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Scheduled restart job, restart counter is at 5.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Start request repeated too quickly.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: uyuni-db.service: Failed with result 'exit-code'.
Jun 12 11:50:49 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Failed to start Uyuni database container service.
Jun 12 12:04:20 vm-uyuni-app-01-prod-axsa-m04p systemd[1]: Starting Uyuni database container service...
Jun 12 12:04:20 vm-uyuni-app-01-prod-axsa-m04p (sh)[676]: uyuni-db.service: Referenced but unset environment variable evaluates to an empty string: PODMAN_EXTRA_ARGS
Jun 12 12:04:20 vm-uyuni-app-01-prod-axsa-m04p podman[676]: 2025-06-12 12:04:20.487354596 +0000 UTC m=+0.032102513 image pull c6e8579ef92ed1b64525c1876cf5fb846d3129db23b01a33bd1ff7630216b76>
Jun 12 12:04:20 vm-uyuni-app-01-prod-axsa-m04p sh[676]: Error: no secret with name or id "uyuni-db-ca": no such secret

Additional information

No response

[akupreev]

Hello, my friend. We are facing the same problem

[Rumpel1318]

Same here, unfortunately, backuped VM didn't change anything.

[unixadmn]

Hello, We are facing the same problem

[rjmateus]

This was a scenario that we detected when a user enters the wrong CA password, we make the database migration and no certificates are set for the database and the migration fails.
We have corrected this issue and the next version will have it.
You can check the PR's that improves this situation:

• Check for CA validity before doing any real upgrade work uyuni-tools#598
• During migration, improve info about usage on ssl-password and logs(bsc#1243009) uyuni-tools#584

One option to fix the issue is to revert to a backup and restart the migration with the correct password for the CA.
Another option can be to manually set the missing CA's and Certificates (database migration should have happened, and in theory the only missing part for the migration is the certificate creation and deployments)

You can see the comment at: #10413 (comment)

The Uyuni CA, key and cert can be extracted from the "Big" container, uyuni-db-ca can be the same as the one for uyuni-ca, and uyuni-db-cert and uyuni-db-key needs to be generated.

@aaannz anything else to add?

[mcalmer]

closing as duplicate