replace cronjob with systemd timer

[missytake]

This PR enables us to better inspect if the last backup job went wrong, and why.

• it replaces cronjobs with systemd units + timers
• fixes the return code handling in both backup.sh and the backup-pre.py template
• gets rid of the second backup destination, it was never really supported or used anyway
• uses the BORG_REPO env variable, which is used implicitly by borgbackup
• adds a CI job for linting

[missytake]

Also, we can now just query for the systemd status in grafana to create alerts:

We don't even need an instance-specific alert anymore.

[missytake]

Ugh, borg exits with rc 1 on a warning: https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes

do we want to get alerted by a warning already? In this case it was:

Nov 17 12:04:06 b1 backup.sh[2439766]: /home/link2xt/src/chatmail/core/.flake.nix.swp: stat: [Errno 2] No such file or directory: '.flake.nix.swp'
Nov 17 12:28:41 b1 backup.sh[2439766]: /var/log/journal/1ec4b78cfcad4e9395e37efdeee1f9f6/system.journal: file changed while we backed it up
Nov 17 12:29:34 b1 backup.sh[2439766]: /var/log/syslog: file changed while we backed it up
Nov 17 12:33:00 b1 backup.sh[2439766]: /var/lib/prometheus/metrics2/wal/00006937: file changed while we backed it up

[hpk42]

No, warnings should not cause alerts IMO.

…

On Mon, Nov 17, 2025 at 04:30 -0800, missytake wrote: missytake left a comment (deltachat/pyinfra-borgbackup#4) Ugh, borg exits with rc 1 on a warning: https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes do we want to get alerted by a warning already? In this case it was: ``` Nov 17 12:04:06 b1 backup.sh[2439766]: /home/link2xt/src/chatmail/core/.flake.nix.swp: stat: [Errno 2] No such file or directory: '.flake.nix.swp' Nov 17 12:28:41 b1 backup.sh[2439766]: /var/log/journal/1ec4b78cfcad4e9395e37efdeee1f9f6/system.journal: file changed while we backed it up Nov 17 12:29:34 b1 backup.sh[2439766]: /var/log/syslog: file changed while we backed it up Nov 17 12:33:00 b1 backup.sh[2439766]: /var/lib/prometheus/metrics2/wal/00006937: file changed while we backed it up ``` -- Reply to this email directly or view it on GitHub: #4 (comment) You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

[link2xt]

If current alerting based on time since last successful backup works fine, then we can keep it and not worry about warnings.

[missytake]

No, warnings should not cause alerts IMO.

yeah, after the alerts of last night I agree.

[missytake]

If current alerting based on time since last successful backup works fine, then we can keep it and not worry about warnings.

It doesn't work atm, and systemd-based alerting is much better in any case, as we don't need to configure an alert for every single host. We literally just need to deploy this with pyinfra on more hosts now, and then we get an alert if a backup failed.

And I wrote a wrapper function in bash to not fail if borg has return code 1.

[j4n]

Looks good, we should really remove the set -x in backup.sh.j2 though, see above comment. Test run worked fine here.