Broken Dependency Proxy / 'Error response from daemon: received unexpected HTTP status: 500 Internal Server Error' / Unable to save "Packages and Registries"

[Gaibhne]

Trying to use the Docker Dependency Proxy and facing above error after a container restart ? This issue is for you. Other symptoms of this error are being unable to save your "Packages and registries" settings after a container restart, and pipelines being unable to pull from the proxy. In the error log, errors like ActiveRecord::Encryption::Errors::Decryption will pop up.

I describe the root cause of this error in #3112 but didn't want to make that issue too big with recovery instructions/error post-mortem, so here goes.

The problem is caused by a changing of the Active Records encryption keys, which lead to your Dependency Proxy settings becoming unreadable, which in turn lead to a buggy state in GitLab that is not recoverable through the normal web interface, as far as I could tell. The broken settings have to be manually cleaned up from the PostgreSQL-Database:

WARNING: obviously, you will lose your settings, and obviously, any direct database manipulations should be preceded by making a backup.

docker exec -it gitlab bash -c "psql -h postgresql -p 5432 -U gitlab -d gitlabhq_production -c \"delete from dependency_proxy_group_settings;\""

This is assuming this repos default settings for username and port - password defaults to password, hopefully you changed that one at least ;-)

In some cases, the whole Dependency Proxy dataset gets corrupted, in which case all proxied images must also be cleaned up:

docker exec -it gitlab bash -c "psql -h postgresql -p 5432 -U gitlab -d gitlabhq_production -c \"delete from dependency_proxy_blobs; delete from dependency_proxy_blob_states; delete from dependency_proxy_manifest_states; delete from dependency_proxy_manifests; delete from dependency_proxy_group_settings;\""

After this, you need to log back into Docker Hub in the "Package and registries" setting again.

I created a GitLab issue as well in the hopes of getting better handling for this error: https://gitlab.com/gitlab-org/gitlab/-/issues/545894