Cant push docker images most of the time

[haarhoff-frs]

Description

I'm trying to use Gitea to host a few code repos and the docker images build from them. Migrating the git repos to Gitea was easy and no problem, but most times I cant seem to push docker images. In some cases it succeeds, in others (and most of the cases) it fails with "error from registry: unknown". My images are quite simple and are single architecture. I tried adding "--provenance=false" to the docker build as I've read that somewhere in the issues, but it does not make a difference. It looks like this:

docker push devops.haarhoff.eu/haarhoff/freddiecard:0.4.17
The push refers to repository [devops.haarhoff.eu/haarhoff/freddiecard]
3720fbd9f073: Layer already exists 
722fd60a3029: Layer already exists 
1b2ead6cb8df: Layer already exists 
63b8e2b4e5f9: Layer already exists 
fd4a6745bcb4: Layer already exists 
4f4fb700ef54: Layer already exists 
ca7817a71dee: Pushed 
232b4483dd99: Layer already exists 
error from registry: unknown
unknown

I'm using a personal access token for logging into docker, and to be on the safe side I selected all available permissions when creating the token. I can see this line multiple times in the logs:
/v2/token for 172.30.0.1:0, 404 Not Found in 0.0ms @ container/container.go:187(container.AuthenticateNotImplemented) which seems odd.

Gitea Version

docker.gitea.com/gitea:1.23.8-rootless

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/haarhoff-frs/221969e6f061398ec81cb203ec513ec9

Screenshots

No response

Git Version

2.34.1 on docker host

Operating System

Docker on Ubuntu 22.04

How are you running Gitea?

I'm using Gitea in with docker compose behind an NGINX Proxy Manager. My first try was to connect to the instance via a Cloudflare Tunnel, but in trying to fix the problem with pushing I switched to NPM over a Tailscale connection. Unfortunately I cannot add a public hostname to that server easily, so I had to go the route over Cloudflare/Tailscale.
The NGINX reverse proxy is configured as described in the documentation.

This is my app.ini:

APP_NAME = Haarhoff
RUN_USER = git
RUN_MODE = prod
WORK_PATH = /var/lib/gitea

[repository]
ROOT = /var/lib/gitea/git/repositories

[repository.local]
LOCAL_COPY_PATH = /tmp/gitea/local-repo

[repository.upload]
TEMP_PATH = /tmp/gitea/uploads

[server]
APP_DATA_PATH = /var/lib/gitea
SSH_DOMAIN = devops.haarhoff.eu
HTTP_PORT = 3000
ROOT_URL = https://devops.haarhoff.eu/
DISABLE_SSH = false
; In rootless gitea container only internal ssh server is supported
START_SSH_SERVER = true
SSH_PORT = 2222
SSH_LISTEN_PORT = 2222
BUILTIN_SSH_SERVER_USER = git
LFS_START_SERVER = true
DOMAIN = devops.haarhoff.eu
LFS_JWT_SECRET = XXXXX
OFFLINE_MODE = true

[database]
PATH = /var/lib/gitea/data/gitea.db
DB_TYPE = postgres
HOST = postgresql-db-1:5432
NAME = gitea
USER = gitea
PASSWD = XXXXX
SCHEMA = 
SSL_MODE = disable
LOG_SQL = false

[session]
PROVIDER_CONFIG = /var/lib/gitea/data/sessions
PROVIDER = file

[picture]
AVATAR_UPLOAD_PATH = /var/lib/gitea/data/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /var/lib/gitea/data/repo-avatars

[attachment]
PATH = /var/lib/gitea/data/attachments

[log]
ROOT_PATH = /var/lib/gitea/data/log
MODE = console
LEVEL = info

[security]
INSTALL_LOCK = true
SECRET_KEY = 
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN = XXXXX
PASSWORD_HASH_ALGO = pbkdf2

[service]
DISABLE_REGISTRATION = true
REQUIRE_SIGNIN_VIEW = true
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = false
DEFAULT_KEEP_EMAIL_PRIVATE = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.haarhoff.eu

[lfs]
PATH = /var/lib/gitea/git/lfs

[mailer]
ENABLED = true
SMTP_ADDR = mailer.haarhoff.eu
SMTP_PORT = 2525
FROM = devops@haarhoff.eu
USER = haarhoff/haarhoff
PASSWD = XXXXX

[openid]
ENABLE_OPENID_SIGNIN = true
ENABLE_OPENID_SIGNUP = false

[cron.update_checker]
ENABLED = false

[repository.pull-request]
DEFAULT_MERGE_STYLE = merge

[repository.signing]
DEFAULT_TRUST_MODEL = committer

[oauth2]
JWT_SECRET = XXXX

Database

PostgreSQL

[wxiaoguang]

The root problem is in "manifest processing", there is no easy fix at the moment ......

• push docker image to container packages failed: failed commit on ref "index-sha256:.....": unexpected status: 404 Not Found #26839
• Error 404 on container image sha when using manifest list #25893

and cc @KN4CK3R

[hiifong]

I tested it on version 1.24.0+rc0-3-g6c5f0af45d. The first push resulted in a 500 error, but the second push was successful.

# first push
➜  ~ docker push xxx/images/gitea-mcp:latest
The push refers to repository [xxx/images/gitea-mcp]
097f0947641a: Pushed
015568903b5c: Pushed
received unexpected HTTP status: 500 Internal Server Error

# push again
➜  ~ docker push xxx/images/gitea-mcp:latest
The push refers to repository [xxx/images/gitea-mcp]
097f0947641a: Layer already exists
015568903b5c: Layer already exists
latest: digest: sha256:2cfd3bea1307b087464e2390d20477277302fecc7d03817a0b84013f7502e311 size: 733

[wxiaoguang]

The first push resulted in a 500 error, but the second push was successful.

It just fails randomly. The root problem is in "manifest processing", see my comments in other issues/PR

[Sebclem]

I also have the 500 error with version 1.24.0+rc0-51-g8defca6d39

#18 pushing git.***.fr/***/jellyfin-node-exporter:main with docker
#18 pushing layer 5f70bf18a086
#18 pushing layer 160d30915720
#18 pushing layer fd2758d7a50e
#18 pushing layer 5f70bf18a086 0.4s done
#18 pushing layer 160d30915720 0.4s done
#18 pushing layer fd2758d7a50e 0.4s done
#18 ERROR: received unexpected HTTP status: 500 Internal Server Error
------
 > pushing git.***.fr/***/jellyfin-node-exporter:main with docker:
------
ERROR: received unexpected HTTP status: 500 Internal Server Error

Gitea logs:

2025/06/06 10:33:11 .../container/container.go:92:apiError() [E] close /var/lib/gitea/data/tmp/package-upload/evugzlykjrvseozjsu85tqvmy: file already closed

But even if I retry it doesn't work. I don't know if this is also linked to the "manifest processing" issue.

You can find the project here if needed: https://git.sebclem.fr/sebclem/jellyfin-node-exporter
Dockerfile: https://git.sebclem.fr/sebclem/jellyfin-node-exporter/src/branch/main/Dockerfile
Latest run: https://git.sebclem.fr/sebclem/jellyfin-node-exporter/actions/runs/30

[wxiaoguang]

2025/06/06 10:33:11 .../container/container.go:92:apiError() [E] close /var/lib/gitea/data/tmp/package-upload/evugzlykjrvseozjsu85tqvmy: file already closed

It's not related to this issue. What's your storage? It seems to be related to the storage sdk.

[Sebclem]

I'm using minio, do you want me to create a new issue with all the information ?

[wxiaoguang]

I'm using minio, do you want me to create a new issue with all the information ?

I created a PR to try to "fix" the problem: Ignore "Close" error when uploading container blob #34620 . Let's discuss there and I will mark the comments in this issue as "off-topic". Thank you very much for the report.