registry: received unexpected HTTP status: 500 Internal Server Error

[SquallHalle]

Running the registry container and try to upload an image to it results in a http 500.

Steps to reproduce:
Server 2019 install edge channel docker for windows, upgrade dockerd to nightly and set to experimental.
run: docker run -it -p 5000:5000 -e DEBUG=true --name registry registry:2

Run docker push on an client (dockerd get the server-host as unsecure registry) and get the error.

Docker Info
Client:
Debug Mode: false

Server:
Containers: 22
Running: 0
Paused: 0
Stopped: 22
Images: 24
Server Version: master-dockerproject-2019-03-12
Storage Driver: windowsfilter (windows) lcow (linux)
Windows:
LCOW:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Standard Evaluation Version 1809 (OS Build 17763.348)
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 32GiB
Name: SERVERHOST
ID: 3932400c-ee07-44d5-a769-922e15029e4b
Docker Root Dir: C:\ProgramData\Docker
Debug Mode: true
File Descriptors: -1
Goroutines: 26
System Time: 2019-03-13T15:15:51.4471111+01:00
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
SERVERHOST:5000
127.0.0.0/8
Live Restore Enabled: false

Docker Version

Client:
Version: master-dockerproject-2019-03-12
API version: 1.40
Go version: go1.11.5
Git commit: 81ac432c
Built: Tue Mar 12 23:51:56 2019
OS/Arch: windows/amd64
Experimental: false

Server:
Engine:
Version: master-dockerproject-2019-03-12
API version: 1.40 (minimum version 1.24)
Go version: go1.11.5
Git commit: 258edd7
Built: Tue Mar 12 23:59:36 2019
OS/Arch: windows/amd64
Experimental: true

Output

WARN[0000] No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable. go.version=go1.11.2 instance.id=4a203afe-2d78-42cd-8f46-e7094c70b9ea service=registry version=v2.7.1
INFO[0000] redis not configured go.version=go1.11.2 instance.id=4a203afe-2d78-42cd-8f46-e7094c70b9ea service=registry version=v2.7.1
INFO[0000] Starting upload purge in 31m0s go.version=go1.11.2 instance.id=4a203afe-2d78-42cd-8f46-e7094c70b9ea service=registry version=v2.7.1
INFO[0000] using inmemory blob descriptor cache go.version=go1.11.2 instance.id=4a203afe-2d78-42cd-8f46-e7094c70b9ea service=registry version=v2.7.1
INFO[0000] listening on [::]:5000 go.version=go1.11.2 instance.id=4a203afe-2d78-42cd-8f46-e7094c70b9ea service=registry version=v2.7.1
INFO[0022] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=3f7e45a6-53b2-46f0-85f8-198d9052bdc6 http.request.method=GET http.request.remoteaddr="10.0.0.2:49695" http.request.uri="/v2/" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.3286ms http.response.status=200 http.response.written=2
10.0.0.2 - - [13/Mar/2019:14:05:51 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
ERRO[0023] response completed with error err.code="blob unknown" err.detail=sha256:d2dd05621072711d90e0ca1ae4bac0f201edcf4ed8c120d5e5d35d0de570c736 err.message="blob unknown to registry" go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=f160a3a4-7043-427a-8677-edf7e3f66888 http.request.method=HEAD http.request.remoteaddr="10.0.0.2:49698" http.request.uri="/v2/hello-world/blobs/sha256:d2dd05621072711d90e0ca1ae4bac0f201edcf4ed8c120d5e5d35d0de570c736" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.1626ms http.response.status=404 http.response.written=157 vars.digest="sha256:d2dd05621072711d90e0ca1ae4bac0f201edcf4ed8c120d5e5d35d0de570c736" vars.name=hello-world
10.0.0.2 - - [13/Mar/2019:14:05:51 +0000] "HEAD /v2/hello-world/blobs/sha256:d2dd05621072711d90e0ca1ae4bac0f201edcf4ed8c120d5e5d35d0de570c736 HTTP/1.1" 404 157 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
ERRO[0023] response completed with error err.code="blob unknown" err.detail=sha256:bdcb489a87baede63d00535be023ca8ab2e4291d081e2d6cdd037d23cefbae89 err.message="blob unknown to registry" go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=79a4c36a-2e21-4d5b-8ddb-09017f8ac030 http.request.method=HEAD http.request.remoteaddr="10.0.0.2:49699" http.request.uri="/v2/hello-world/blobs/sha256:bdcb489a87baede63d00535be023ca8ab2e4291d081e2d6cdd037d23cefbae89" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.1262ms http.response.status=404 http.response.written=157 vars.digest="sha256:bdcb489a87baede63d00535be023ca8ab2e4291d081e2d6cdd037d23cefbae89" vars.name=hello-world
10.0.0.2 - - [13/Mar/2019:14:05:51 +0000] "HEAD /v2/hello-world/blobs/sha256:bdcb489a87baede63d00535be023ca8ab2e4291d081e2d6cdd037d23cefbae89 HTTP/1.1" 404 157 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0023] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=a73d4dd4-d0a0-457f-ad21-9ff4fd722d29 http.request.method=POST http.request.remoteaddr="10.0.0.2:49701" http.request.uri="/v2/hello-world/blobs/uploads/" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.duration=107.7403ms http.response.status=202 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:52 +0000] "POST /v2/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0023] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=3b405fd4-4cca-4a8d-9ffa-c08ee3f16399 http.request.method=POST http.request.remoteaddr="10.0.0.2:49700" http.request.uri="/v2/hello-world/blobs/uploads/" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.duration=132.6731ms http.response.status=202 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:51 +0000] "POST /v2/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0023] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=c089075f-00e7-42f9-b2b2-61d3e43d0423 http.request.method=PATCH http.request.remoteaddr="10.0.0.2:49704" http.request.uri="/v2/hello-world/blobs/uploads/ebd3d1f1-38b2-4d96-8f29-57b56662a45f?_state=5zXxtglENvUOxEAxHKC5kMZHEUhec7EsthL57wzkEhB7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJlYmQzZDFmMS0zOGIyLTRkOTYtOGYyOS01N2I1NjY2MmE0NWYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTIuMDAxNTExOFoifQ%3D%3D" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.duration=87.1482ms http.response.status=202 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:52 +0000] "PATCH /v2/hello-world/blobs/uploads/ebd3d1f1-38b2-4d96-8f29-57b56662a45f?_state=5zXxtglENvUOxEAxHKC5kMZHEUhec7EsthL57wzkEhB7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJlYmQzZDFmMS0zOGIyLTRkOTYtOGYyOS01N2I1NjY2MmE0NWYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTIuMDAxNTExOFoifQ%3D%3D HTTP/1.1" 202 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0023] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=eeb02b58-a0f3-45bf-958e-7a3970628a3a http.request.method=PATCH http.request.remoteaddr="10.0.0.2:49703" http.request.uri="/v2/hello-world/blobs/uploads/bacf1302-cd28-46fd-8ae6-cd9473b0c63c?_state=hS1G5rgcI6OV-VPEZvHw7ZDSpU-1-k1zxRx3f12VGlZ7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJiYWNmMTMwMi1jZDI4LTQ2ZmQtOGFlNi1jZDk0NzNiMGM2M2MiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTIuMDE0MDQ5OFoifQ%3D%3D" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.duration=111.3323ms http.response.status=202 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:52 +0000] "PATCH /v2/hello-world/blobs/uploads/bacf1302-cd28-46fd-8ae6-cd9473b0c63c?_state=hS1G5rgcI6OV-VPEZvHw7ZDSpU-1-k1zxRx3f12VGlZ7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJiYWNmMTMwMi1jZDI4LTQ2ZmQtOGFlNi1jZDk0NzNiMGM2M2MiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTIuMDE0MDQ5OFoifQ%3D%3D HTTP/1.1" 202 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0023] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=96b945e7-97e5-43b3-9344-eccaac4f75f0 http.request.method=PUT http.request.remoteaddr="10.0.0.2:49706" http.request.uri="/v2/hello-world/blobs/uploads/bacf1302-cd28-46fd-8ae6-cd9473b0c63c?_state=3cDeoFSdfi1eXd0_SrNuRJXN8u5uyUY-UG4oQbKImRt7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJiYWNmMTMwMi1jZDI4LTQ2ZmQtOGFlNi1jZDk0NzNiMGM2M2MiLCJPZmZzZXQiOjE2NTUsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTJaIn0%3D&digest=sha256%3Ad2dd05621072711d90e0ca1ae4bac0f201edcf4ed8c120d5e5d35d0de570c736" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.duration=374.032ms http.response.status=201 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:52 +0000] "PUT /v2/hello-world/blobs/uploads/bacf1302-cd28-46fd-8ae6-cd9473b0c63c?_state=3cDeoFSdfi1eXd0_SrNuRJXN8u5uyUY-UG4oQbKImRt7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJiYWNmMTMwMi1jZDI4LTQ2ZmQtOGFlNi1jZDk0NzNiMGM2M2MiLCJPZmZzZXQiOjE2NTUsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTJaIn0%3D&digest=sha256%3Ad2dd05621072711d90e0ca1ae4bac0f201edcf4ed8c120d5e5d35d0de570c736 HTTP/1.1" 201 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0023] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=fe1ef99b-1092-4007-aaa8-e338a3fe6f43 http.request.method=PUT http.request.remoteaddr="10.0.0.2:49705" http.request.uri="/v2/hello-world/blobs/uploads/ebd3d1f1-38b2-4d96-8f29-57b56662a45f?_state=3IRrXvgXWcoYXzap1fhNUBQNIEXeNB7YeBOE2alXzVF7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJlYmQzZDFmMS0zOGIyLTRkOTYtOGYyOS01N2I1NjY2MmE0NWYiLCJPZmZzZXQiOjkyNCwiU3RhcnRlZEF0IjoiMjAxOS0wMy0xM1QxNDowNTo1MloifQ%3D%3D&digest=sha256%3Abdcb489a87baede63d00535be023ca8ab2e4291d081e2d6cdd037d23cefbae89" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.duration=401.0992ms http.response.status=201 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:52 +0000] "PUT /v2/hello-world/blobs/uploads/ebd3d1f1-38b2-4d96-8f29-57b56662a45f?_state=3IRrXvgXWcoYXzap1fhNUBQNIEXeNB7YeBOE2alXzVF7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJlYmQzZDFmMS0zOGIyLTRkOTYtOGYyOS01N2I1NjY2MmE0NWYiLCJPZmZzZXQiOjkyNCwiU3RhcnRlZEF0IjoiMjAxOS0wMy0xM1QxNDowNTo1MloifQ%3D%3D&digest=sha256%3Abdcb489a87baede63d00535be023ca8ab2e4291d081e2d6cdd037d23cefbae89 HTTP/1.1" 201 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0024] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=ca0623eb-b37e-4489-8354-73bfd7b08d72 http.request.method=HEAD http.request.remoteaddr="10.0.0.2:49707" http.request.uri="/v2/hello-world/blobs/sha256:d2dd05621072711d90e0ca1ae4bac0f201edcf4ed8c120d5e5d35d0de570c736" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/octet-stream" http.response.duration=3.6585ms http.response.status=200 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:52 +0000] "HEAD /v2/hello-world/blobs/sha256:d2dd05621072711d90e0ca1ae4bac0f201edcf4ed8c120d5e5d35d0de570c736 HTTP/1.1" 200 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0024] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=3a703d4c-3e97-4137-a86e-733c976db33f http.request.method=HEAD http.request.remoteaddr="10.0.0.2:49708" http.request.uri="/v2/hello-world/blobs/sha256:bdcb489a87baede63d00535be023ca8ab2e4291d081e2d6cdd037d23cefbae89" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/octet-stream" http.response.duration=1.4183ms http.response.status=200 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:52 +0000] "HEAD /v2/hello-world/blobs/sha256:bdcb489a87baede63d00535be023ca8ab2e4291d081e2d6cdd037d23cefbae89 HTTP/1.1" 200 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
ERRO[0024] response completed with error err.code="blob unknown" err.detail=sha256:eb68d2e2f59a9e5ea880ccc5715672ba5238c3f03d0ad596689564c675a986b4 err.message="blob unknown to registry" go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=1da40815-757f-4050-b6b0-6897fb37d475 http.request.method=HEAD http.request.remoteaddr="10.0.0.2:49710" http.request.uri="/v2/hello-world/blobs/sha256:eb68d2e2f59a9e5ea880ccc5715672ba5238c3f03d0ad596689564c675a986b4" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=19.452ms http.response.status=404 http.response.written=157 vars.digest="sha256:eb68d2e2f59a9e5ea880ccc5715672ba5238c3f03d0ad596689564c675a986b4" vars.name=hello-world
10.0.0.2 - - [13/Mar/2019:14:05:52 +0000] "HEAD /v2/hello-world/blobs/sha256:eb68d2e2f59a9e5ea880ccc5715672ba5238c3f03d0ad596689564c675a986b4 HTTP/1.1" 404 157 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0024] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=22be23b5-e929-4fdd-ad52-727a6dafbf71 http.request.method=POST http.request.remoteaddr="10.0.0.2:49711" http.request.uri="/v2/hello-world/blobs/uploads/" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.duration=148.1601ms http.response.status=202 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:52 +0000] "POST /v2/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0024] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=4070a50c-a1ac-4f02-bcca-9a0ac6cbfb5a http.request.method=PATCH http.request.remoteaddr="10.0.0.2:49712" http.request.uri="/v2/hello-world/blobs/uploads/c8597299-fa71-4232-8536-f3f87d0d7190?_state=0Y58NYJuW_I_rdlkU19Y0PI5sBbZ8HRO53ShZKeCKTp7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJjODU5NzI5OS1mYTcxLTQyMzItODUzNi1mM2Y4N2QwZDcxOTAiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTIuOTk3MjI4OFoifQ%3D%3D" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.duration=145.3174ms http.response.status=202 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:53 +0000] "PATCH /v2/hello-world/blobs/uploads/c8597299-fa71-4232-8536-f3f87d0d7190?_state=0Y58NYJuW_I_rdlkU19Y0PI5sBbZ8HRO53ShZKeCKTp7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJjODU5NzI5OS1mYTcxLTQyMzItODUzNi1mM2Y4N2QwZDcxOTAiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTIuOTk3MjI4OFoifQ%3D%3D HTTP/1.1" 202 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0025] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=e44e753d-38d3-426c-992d-42e976cfa7f8 http.request.method=PUT http.request.remoteaddr="10.0.0.2:49713" http.request.uri="/v2/hello-world/blobs/uploads/c8597299-fa71-4232-8536-f3f87d0d7190?_state=ymRwlR7joQYkZXGjGZCW64u9nzUqQnLv1GBddvczsFl7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJjODU5NzI5OS1mYTcxLTQyMzItODUzNi1mM2Y4N2QwZDcxOTAiLCJPZmZzZXQiOjE4ODUsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTJaIn0%3D&digest=sha256%3Aeb68d2e2f59a9e5ea880ccc5715672ba5238c3f03d0ad596689564c675a986b4" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.duration=826.3638ms http.response.status=201 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:53 +0000] "PUT /v2/hello-world/blobs/uploads/c8597299-fa71-4232-8536-f3f87d0d7190?_state=ymRwlR7joQYkZXGjGZCW64u9nzUqQnLv1GBddvczsFl7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJjODU5NzI5OS1mYTcxLTQyMzItODUzNi1mM2Y4N2QwZDcxOTAiLCJPZmZzZXQiOjE4ODUsIlN0YXJ0ZWRBdCI6IjIwMTktMDMtMTNUMTQ6MDU6NTJaIn0%3D&digest=sha256%3Aeb68d2e2f59a9e5ea880ccc5715672ba5238c3f03d0ad596689564c675a986b4 HTTP/1.1" 201 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
INFO[0025] response completed go.version=go1.11.2 http.request.host="HOSTNAME:5000" http.request.id=6b6a65e9-7b85-48f2-a014-daa3b162e2f9 http.request.method=HEAD http.request.remoteaddr="10.0.0.2:49715" http.request.uri="/v2/hello-world/blobs/sha256:eb68d2e2f59a9e5ea880ccc5715672ba5238c3f03d0ad596689564c675a986b4" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/octet-stream" http.response.duration=1.7966ms http.response.status=200 http.response.written=0
10.0.0.2 - - [13/Mar/2019:14:05:54 +0000] "HEAD /v2/hello-world/blobs/sha256:eb68d2e2f59a9e5ea880ccc5715672ba5238c3f03d0ad596689564c675a986b4 HTTP/1.1" 200 0 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"
ERRO[0025] response completed with error err.code=unknown err.message="unknown error" go.version=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="HOSTNAME:5000" http.request.id=09c06ff1-dd51-48d0-9545-e7ca6a13a026 http.request.method=PUT http.request.remoteaddr="10.0.0.2:49716" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.5447ms http.response.status=500 http.response.written=523 vars.name=hello-world vars.reference=latest
ERRO[0025] response completed with error err.code=unknown err.message="invalid URL on layer" go.version=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="HOSTNAME:5000" http.request.id=09c06ff1-dd51-48d0-9545-e7ca6a13a026 http.request.method=PUT http.request.remoteaddr="10.0.0.2:49716" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.2014ms http.response.status=500 http.response.written=523 vars.name=hello-world vars.reference=latest
ERRO[0025] response completed with error err.code="manifest blob unknown" err.detail=sha256:e46172273a4e4384e1eec7fb01091c828a256ea0f87b30f61381fba9bc511371 err.message="blob unknown to registry" go.version=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="HOSTNAME:5000" http.request.id=09c06ff1-dd51-48d0-9545-e7ca6a13a026 http.request.method=PUT http.request.remoteaddr="10.0.0.2:49716" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.263ms http.response.status=500 http.response.written=523 vars.name=hello-world vars.reference=latest
ERRO[0025] response completed with error err.code=unknown err.message="unknown error" go.version=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="HOSTNAME:5000" http.request.id=09c06ff1-dd51-48d0-9545-e7ca6a13a026 http.request.method=PUT http.request.remoteaddr="10.0.0.2:49716" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.314ms http.response.status=500 http.response.written=523 vars.name=hello-world vars.reference=latest
ERRO[0025] response completed with error err.code=unknown err.message="invalid URL on layer" go.version=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="HOSTNAME:5000" http.request.id=09c06ff1-dd51-48d0-9545-e7ca6a13a026 http.request.method=PUT http.request.remoteaddr="10.0.0.2:49716" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.5069ms http.response.status=500 http.response.written=523 vars.name=hello-world vars.reference=latest
ERRO[0025] response completed with error err.code="manifest blob unknown" err.detail=sha256:f493dc3e1d73855439ead197cc94d3bdac81372c5cb171f12b1f29ba58cdc9d9 err.message="blob unknown to registry" go.version=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="HOSTNAME:5000" http.request.id=09c06ff1-dd51-48d0-9545-e7ca6a13a026 http.request.method=PUT http.request.remoteaddr="10.0.0.2:49716" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.5566ms http.response.status=500 http.response.written=523 vars.name=hello-world vars.reference=latest
10.0.0.2 - - [13/Mar/2019:14:05:54 +0000] "PUT /v2/hello-world/manifests/latest HTTP/1.1" 500 523 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.2 \(windows\))"

[ChrML]

Same issue here.

I can successfully push to the same insecure private Docker repo from a Linux computer.
Pushing the hello-world container to the same repo from the newest Docker For Desktop fails.

[yarglah]

Hi,
A workaround is to downgrade registry to 2.6.2.

[ChrML]

I figured it out.

The problem is observed on the registry server with debug logging enabled. Says something about validation failed when pushing from a Windows Docker. Clearly the client should get more info than just "500 - Internal error".

The solution is to change/add the following to config.yml on the registry server:

validation:
disabled: true

NOTE: This property is undocumented, setting "enabled: false" does NOT do the same, you need the "disabled: true".

[pwuertz]

@yarglah 's workaround worked for me too. Simply run registry:2.6 instead of registry:2. Seems this validation error has been introduced somewhere in 2.7.

[AndreasHassing]

See distribution/distribution#2866 (the issue opened in the source code repo for Docker Registry).

[alpharameeztech]

I had the same problem, I fixed it by disabling validation.
docker run -d -p 5000:5000 --name registry -e REGISTRY_VALIDATION_DISABLED=true registry:2

[docdawning]

Thank you @alpharameeztech, I've been beating my head against a wall on this for what feels like days (just a few hours, actually). I set this env var as you suggested and now the registry appears usable from Windows.