Outage after pgbouncer 1.25 update

[pat-s]

pgbouncer 1.25 (released two days ago) caused a full outage of my clusters today.

The reason was the system package manager initiated update task.

Apparently, this task changed the ownership on the /var/log/pgbouncer dir from postgres to pgbouncer, letting the service fail to start.

OS: AlmaLinux 10

I see the following options:

1. Don't use the custom user postgres for all pgbouncer related tasks but go with its default one pgbouncer (not sure if easily possible, you know best)
2. Find a way to ensure that system package manager updates don't change permissions on important dirs /var/log/*, /etc/*

[vitabaks]

I would prefer to disable automatic updates, to update the servers only during the special time allotted for this.

Example:

sudo systemctl stop apt-daily.timer apt-daily-upgrade.timer
sudo systemctl disable apt-daily.timer apt-daily-upgrade.timer
sudo systemctl mask apt-daily.service apt-daily-upgrade.service

P.S. Perhaps we should automate disabling auto-updates to system packages.

[pat-s]

I would prefer to disable automatic updates, to update the servers only during the special time allotted for this.

I'd (personally) strongly prefer to have daily updates AND to stay in control how system updates are handled.

Please don't confuse the issue at hand with a general update issue for the tool: the new version itself is fine, the issue comes from the ownership override of the system package manager because autobase changed it to a beforehand.

Pinning versions for pgbouncer, patroni and other essential tools might also be in scope but this is a different task then. If this is the goal, autobase could let the system package managers ignore those packages and use a custom update mechanism otherwise (as done for etcd already AFAIK). But this isn't trivial and requires omitting the default package manager for all essential packages.

Most of these packages should be super stable and not break due to a version bug. To me, the main focus should be on avoiding such ownership issues which then clash with the system package manager. Maybe setting everything pgbouncer related to be owned and run by the pgbouncer user is directly working OOB.

[vitabaks]

Don't use the custom user postgres for all pgbouncer related tasks but go with its default one pgbouncer

Perhaps it is necessary to correct the owner/group here:

https://github.com/vitabaks/autobase/blob/2.4.1/automation/roles/pgbouncer/tasks/main.yml#L52-L53

But I'm not sure how safe it is yet either, so we need to test it.