Skip to content

All repositories are skipped during backup restore #3155

@AAWO

Description

@AAWO

I have a GitLab 15.1.0 backup that I need to restore. When I attempt to restore the backup using the command docker compose run --rm gitlab app:rake gitlab:backup:restore, all of the repositories are skipped. After the restore finishes, I can see all repositories listed in GitLab, but they are all empty (not even initialized).

The shortened output of the restore command (I've included the logs for a single repository, but all the other ones are very similar):

$ docker compose run --rm gitlab app:rake gitlab:backup:restore
(...)
Select a backup to restore: 1755221522_2025_08_15_15.1.0_gitlab_backup.tar
Running raketask gitlab:backup:restore...
2025-08-17 08:20:52 +0200 -- Unpacking backup ... 
2025-08-17 08:24:51 +0200 -- Unpacking backup ... done
2025-08-17 08:24:51 +0200 -- Restoring database ...
2025-08-17 08:24:51 +0200 -- Be sure to stop Puma, Sidekiq, and any other process that
connects to the database before proceeding. For Omnibus
installs, see the following link for more information:
https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations

Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.

Do you want to continue (yes/no)? yes
Removing all tables. Press `Ctrl-C` within 5 seconds to abort
2025-08-17 08:26:16 +0200 -- Cleaning the database ... 
2025-08-17 08:26:26 +0200 -- done
Restoring PostgreSQL database gitlabhq_production ... ERROR:  must be owner of extension pg_trgm
ERROR:  must be owner of extension btree_gist
ERROR:  must be owner of extension btree_gist
ERROR:  must be owner of extension pg_trgm
(...)
2025-08-17 08:34:27 +0200 -- Restoring database ... done
2025-08-17 08:34:27 +0200 -- Restoring repositories ... 
{"command":"restore","gl_project_path":"project_path","level":"info","msg":"started restore","relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git","storage_name":"default","time":"2025-08-17T06:34:33.053Z"}
{"command":"restore","gl_project_path":"project_path.wiki","level":"info","msg":"started restore","relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki.git","storage_name":"default","time":"2025-08-17T06:34:33.205Z"}
{"command":"restore","error":"manager: repository skipped: restore bundle: filesystem sink: get reader for \"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki.bundle\": doesn't exist","gl_project_path":"project_path.wiki","level":"warning","msg":"skipped restore","relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki.git","storage_name":"default","time":"2025-08-17T06:34:34.627Z"}
{"command":"restore","gl_project_path":"project_path","level":"info","msg":"started restore","relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.design.git","storage_name":"default","time":"2025-08-17T06:34:34.627Z"}
{"command":"restore","error":"manager: repository skipped: restore bundle: filesystem sink: get reader for \"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.design.bundle\": doesn't exist","gl_project_path":"project_path","level":"warning","msg":"skipped restore","relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.design.git","storage_name":"default","time":"2025-08-17T06:34:35.178Z"}
{"command":"restore","gl_project_path":"project_path","level":"info","msg":"completed restore","relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git","storage_name":"default","time":"2025-08-17T06:34:37.422Z"}
(...)
2025-08-17 08:52:29 +0200 -- Restoring repositories ... done
2025-08-17 08:52:29 +0200 -- Restoring uploads ... 
2025-08-17 08:52:29 +0200 -- Restoring uploads ... done
2025-08-17 08:52:29 +0200 -- Restoring builds ... 
2025-08-17 08:52:30 +0200 -- Restoring builds ... done
2025-08-17 08:52:30 +0200 -- Restoring artifacts ... 
2025-08-17 08:54:00 +0200 -- Restoring artifacts ... done
2025-08-17 08:54:01 +0200 -- Restoring pages ... 
2025-08-17 08:54:02 +0200 -- Restoring pages ... done
2025-08-17 08:54:02 +0200 -- Restoring lfs objects ... 
2025-08-17 08:54:02 +0200 -- Restoring lfs objects ... done
2025-08-17 08:54:02 +0200 -- Restoring terraform states ... 
2025-08-17 08:54:02 +0200 -- Restoring terraform states ... done
2025-08-17 08:54:02 +0200 -- Restoring packages ... 
2025-08-17 08:54:02 +0200 -- Restoring packages ... done
This task will now rebuild the authorized_keys file.
You will lose any data stored in the authorized_keys file.
Do you want to continue (yes/no)? yes

2025-08-17 09:08:50 +0200 -- Deleting tar staging files ... 
2025-08-17 09:08:50 +0200 -- Cleaning up /home/git/data/backups/backup_information.yml
2025-08-17 09:08:50 +0200 -- Cleaning up /home/git/data/backups/db
2025-08-17 09:08:50 +0200 -- Cleaning up /home/git/data/backups/repositories
2025-08-17 09:08:51 +0200 -- Cleaning up /home/git/data/backups/uploads.tar.gz
2025-08-17 09:08:51 +0200 -- Cleaning up /home/git/data/backups/builds.tar.gz
2025-08-17 09:08:51 +0200 -- Cleaning up /home/git/data/backups/artifacts.tar.gz
2025-08-17 09:08:51 +0200 -- Cleaning up /home/git/data/backups/pages.tar.gz
2025-08-17 09:08:51 +0200 -- Cleaning up /home/git/data/backups/lfs.tar.gz
2025-08-17 09:08:51 +0200 -- Cleaning up /home/git/data/backups/terraform_state.tar.gz
2025-08-17 09:08:51 +0200 -- Cleaning up /home/git/data/backups/packages.tar.gz
2025-08-17 09:08:51 +0200 -- Deleting tar staging files ... done
2025-08-17 09:08:51 +0200 -- Deleting backups/tmp ... 
2025-08-17 09:08:51 +0200 -- Deleting backups/tmp ... done
2025-08-17 09:08:51 +0200 -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need to restore these files manually.
2025-08-17 09:08:51 +0200 -- Restore task is done.

The size of the backup .tar file matches the expected size of all the repositories. I also copied the backup file to a different location and extracted its contents. The hashed directories (in this case @hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce) can be found in the repositories directory. The .bundle and .refs files also contain the names of branches in that project. The files in the directory are listed below:

$ ls -lah repositories/\@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/
total 16K
drwx------ 3 username 1000 4.0K 2025-08-17 08:53 ./
drwx------ 3 username 1000 4.0K 2025-08-15 03:32 ../
drwx------ 2 username 1000 4.0K 2025-08-17 08:53 1755221522_2025_08_15_15.1.0/
-rw------- 1 username 1000   28 2025-08-15 03:32 LATEST
$ ls -lah repositories/\@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/1755221522_2025_08_15_15.1.0/
total 72M
drwx------ 2 username 1000 4.0K 2025-08-17 08:53 ./
drwx------ 3 username 1000 4.0K 2025-08-17 08:53 ../
-rw------- 1 username 1000  72M 2025-08-15 03:32 001.bundle
-rw------- 1 username 1000  23K 2025-08-15 03:32 001.refs
-rw------- 1 username 1000    3 2025-08-15 03:32 LATEST

The docker-compose.yml file contents:

services:
  gitlab:
    environment:
      DEBUG: "false"
      DB_ADAPTER: postgresql
      DB_HOST: postgresql
      DB_PORT: 5432
      DB_USER: xxx
      DB_PASS: xxx
      DB_NAME: gitlabhq_production
      REDIS_HOST: redis
      REDIS_PORT: 6379
      GITLAB_BACKUP_DIR_CHOWN: "true"
      GITLAB_BACKUP_SCHEDULE: daily
      GITLAB_BACKUP_TIME: "03:30"
      GITLAB_NOTIFY_ON_BROKEN_BUILDS: "true"
      GITLAB_PORT: 10080
      GITLAB_SECRETS_DB_KEY_BASE: xxx
      GITLAB_SECRETS_OTP_KEY_BASE: xxx
      GITLAB_SECRETS_SECRET_KEY_BASE: xxx
      GITLAB_SSH_PORT: 10022
    image: sameersbn/gitlab:15.1.0
    links:
    - redis:redisio
    - postgresql:postgresql
    ports:
    - 10080:80
    - "10022:22"
    restart: always
    volumes:
    - /path:/home/git/data/backups
  
  postgresql:
    environment:
      DB_EXTENSION: pg_trgm,btree_gist
      DB_NAME: gitlabhq_production
      DB_PASS: xxx
      DB_USER: xxx
    image: sameersbn/postgresql:12-20200524
    restart: always

  redis:
    command:
    - --loglevel warning
    image: sameersbn/redis:latest
    restart: always
version: "2"

What am I missing? How can I restore the backup including all the projects data?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions