Database backup errors goes unnoticed

[alteriks]

Summary

I've been playing with docker-db-backup and I've noticed that pg_dump error is masked when passing wrong password to docker-db-backup container

Steps to reproduce

Use this docker-compose.yml, it's mostly verbatim ./examples/compose.yml
Notable changes are

services:
  example-db:
    hostname: example-postgres-host
    container_name: example-db
    image: postgres:16
    ports:
      - 3306:3306
    volumes:
      - ./db:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=example
      - POSTGRES_PASSWORD=examplepassword
    restart: always
    networks:
      - example-db-network

  example-db-backup:
    container_name: example-db-backup
    image: tiredofit/db-backup
    volumes:
      - ./backups:/backup
    environment:
      - TIMEZONE=America/Vancouver
      # - TIMEZONE=Europe/Warsaw
      - CONTAINER_NAME=example-db-backup
      - CONTAINER_ENABLE_MONITORING=FALSE
      # - DEBUG_MODE=TRUE

      - BACKUP_JOB_CONCURRENCY=1 # Only run one job at a time
      - DEFAULT_CHECKSUM=NONE # Don't create checksums
      - DEFAULT_COMPRESSION=ZSTD # Compress all with ZSTD
      - DEFAULT_BACKUP_INTERVAL=1440 # Backup every 1440 minutes
      - DEFAULT_BACKUP_BEGIN=0000 # Start backing up at midnight
      - DEFAULT_CLEANUP_TIME=8640 # Cleanup backups after a week

      - DB01_TYPE=postgres
      - DB01_HOST=example-postgres-host
      - DB01_NAME=example
      - DB01_USER=example
      - DB01_PASS=examplepassword
      - DB01_BACKUP_INTERVAL=60 # (override) Backup every 60 minutes
      - DB01_BACKUP_BEGIN=+1 # (override) Backup starts in ten minutes
      - DB01_CLEANUP_TIME=240 # (override) Cleanup backups they are older than 240 minutes
      - DB01_CHECKSUM=MD5 # (override) Create a SHA1 checksum
      - DB01_COMPRESSION=BZ # (override) Compress with BZIP

    restart: always
    networks:
      - example-db-network

networks:
  example-db-network:
    name: example-db-network

OK:

What is the expected correct behavior?

All errors should be marked as errors and not masked as success

Relevant logs and/or screenshots

executing pg_dump with wrong password inside container produces error and exitcode=1

[alteriks]

I've tried to replicate this error using wrong password when backuping mariadb, but I've came upon reconnecting logic
Bash function check_availability() uses different means for checking if database is up and accepting connections.
For mysql/mariadb it actually checks status which requires login in with correct password. And availability is checked every 5seconds
Postgresql is checked using pg_isready which doesn't use password . Other databases ie redis/mssql are using simply
netcat

https://github.com/tiredofit/docker-db-backup/blob/a475f7d0f31e68e37c1d1f42cf4ea4d2b71c4ac1/install/assets/functions/10-db-backup#L948C1-L948C19

It seems that exitcode is masked in couple of places:
backup_mysql()

docker-db-backup/install/assets/functions/10-db-backup

Line 739 in a475f7d

run_as_user ${play_fair} ${_mysql_prefix}${_mysql_bin_prefix}dump --max-allowed-packet=${backup_job_mysql_max_allowed_packet} -h ${backup_job_db_host} -P ${backup_job_db_port} -u${backup_job_db_user} ${events} ${single_transaction} ${stored_procedures} ${mysql_tls_args} ${backup_job_extra_opts} ${backup_job_extra_backup_opts} $db | ${compress_cmd} | run_as_user tee "${temporary_directory}"/"${backup_job_filename}" > /dev/null

docker-db-backup/install/assets/functions/10-db-backup

Line 760 in a475f7d

run_as_user ${play_fair} ${_mysql_prefix}${_mysql_bin_prefix}dump --max-allowed-packet=${backup_job_mysql_max_allowed_packet} -h ${backup_job_db_host} -P ${backup_job_db_port} -u${backup_job_db_user} ${events} ${single_transaction} ${stored_procedures} ${mysql_tls_args} ${backup_job_extra_opts} ${backup_job_extra_backup_opts} --databases $(echo ${db_names} | xargs) | ${compress_cmd} | run_as_user tee "${temporary_directory}"/"${backup_job_filename}" > /dev/null

backup_pgsql()

docker-db-backup/install/assets/functions/10-db-backup

Line 783 in a475f7d

run_as_user ${play_fair} pg_dumpall -h "${backup_job_db_host}" -U "${backup_job_db_user}" -p "${backup_job_db_port}" -g ${backup_job_extra_opts} ${backup_job_extra_backup_opts} | ${compress_cmd} | run_as_user tee "${temporary_directory}"/"${backup_job_filename}" > /dev/null

docker-db-backup/install/assets/functions/10-db-backup

Line 832 in a475f7d

run_as_user ${play_fair} pg_dump -h "${backup_job_db_host}" -p "${backup_job_db_port}" -U "${backup_job_db_user}" $db ${backup_job_extra_opts} ${backup_job_extra_backup_opts} | ${compress_cmd} | run_as_user tee "${temporary_directory}"/"${backup_job_filename}" > /dev/null

docker-db-backup/install/assets/functions/10-db-backup

Line 866 in a475f7d

run_as_user ${play_fair} pg_dumpall -h ${backup_job_db_host} -p ${backup_job_db_port} -U ${backup_job_db_user} ${pgexclude_arg} ${backup_job_extra_opts} ${backup_job_extra_backup_opts} | ${compress_cmd} | run_as_user tee "${temporary_directory}"/"${backup_job_filename}" > /dev/null

Whenever pipes are used in bash $? variable stores rightmost exit status

alteriks@moar:~ $ false | false | true
alteriks@moar:~ $ echo $?
0
alteriks@moar:~ $ true | true | false
alteriks@moar:~ $ echo $?
1

So using this code, whenever backup goes wrong or compression goes wrong (ie no space) but tee is successful, exitcode will be set to 0 concealing previous errors

        run_as_user ${play_fair} pg_dumpall -h ${backup_job_db_host} -p ${backup_job_db_port} -U ${backup_job_db_user} ${pgexclude_arg} ${backup_job_extra_opts} ${backup_job_extra_backup_opts} | ${compress_cmd} | run_as_user tee "${temporary_directory}"/"${backup_job_filename}" > /dev/null
        exit_code=$?

Solution is to use PIPESTATUS:

alteriks@moar:~ $ false | false | true
alteriks@moar:~ $ echo "${PIPESTATUS[0]} ${PIPESTATUS[1]} ${PIPESTATUS[2]}" 
1 1 0
alteriks@moar:~ $ true | true | false
alteriks@moar:~ $ echo "${PIPESTATUS[0]} ${PIPESTATUS[1]} ${PIPESTATUS[2]}" 
0 0 1

[alteriks]

I'll quote stackexchange answer https://unix.stackexchange.com/questions/14270/get-exit-status-of-process-thats-piped-to-another/14276#14276 for posterity:

bash and zsh have an array variable that holds the exit status of each element (command) of the last pipeline executed by the shell.

If you are using bash, the array is called PIPESTATUS (case matters!) and the array indicies start at zero:

$ false | true
$ echo "${PIPESTATUS[0]} ${PIPESTATUS[1]}"
1 0

If you are using zsh, the array is called pipestatus (case matters!) and the array indices start at one:

$ false | true
$ echo "${pipestatus[1]} ${pipestatus[2]}"
1 0

To combine them within a function in a manner that doesn't lose the values:

$ false | true
$ retval_bash="${PIPESTATUS[0]}" retval_zsh="${pipestatus[1]}" retval_final=$?
$ echo $retval_bash $retval_zsh $retval_final
1 0

Run the above in bash or zsh and you'll get the same results; only one of retval_bash and retval_zsh will be set. The other will be blank. This would allow a function to end with return $retval_bash $retval_zsh (note the lack of quotes!).

[alteriks]

[tiredofit]

Thanks for the detailed investigation on this! I'll have a peek at your PR shortly.