How to fix Traefik v2 / v3 (v2.11.x / v3.x.x) stopping Immich uploads that go longer than 60 seconds (502 Bad Gateway)

[othyn]

TLDR, here's the fix:

Add the following to your Traefik config and restart the Traefik instance. For more examples, see the docs: https://doc.traefik.io/traefik/routing/entrypoints/#respondingtimeouts

entryPoints:
  ...
  name:
    ...
    # This is the fix:
    transport:
      respondingTimeouts:
        readTimeout: "0s"

---

What's going on?

I was having issues with Immich not being able to upload large files. The app would stop uploading, whilst the app remained responsive, during large file uploads. This is new behaviour, as I've been able to upload to my hearts content for months. So what's changed? My hunch was it was network related, so I transferred the same file onto my desktop to see if the upload still failed via a browser, it did.

Digging deeper into the issue, the first thought was that it was buffer related, as these things usually are. But as I'm using Traefik, which by default doesn't impose/define any buffer or upload file size limitations, as it tries to behave as much like a transparent reverse proxy as possible.

Doing the file transfer on a desktop was also handy for one main reason, the network inspector. Popping that open revealed that requests were being terminated prematurely with a 502, but didn't reveal anything else of value, even the request timings (which will be important later), as they weren't that accurate and included the transport and execution times from the browser as well. So nothing stood out as being the problem.

Next port of call was enabling DEBUG level logging and access logging on the Traefik instance to try and debug what on earth was going on. After doing that, all became clear:

# Only access logging enabled at first...
10.10.0.10 - - [17/Apr/2024:10:34:32 +0000] "POST /api/asset/upload HTTP/2.0" 502 11 "-" "-" 638 "photos@docker" "http://172.18.111.1:3001/" 60000ms
10.10.0.10 - - [17/Apr/2024:10:36:28 +0000] "POST /api/asset/upload HTTP/2.0" 502 11 "-" "-" 855 "photos@docker" "http://172.18.111.1:3001/" 60000ms

# Hello! That 60000ms is an immediate red flag and smells like Traefik doing something to the connection as its bang on 60000ms each time. If it was anything else in the network chain that would be +/- a few ms as overhead is added in.
# Let's enable the DEBUG log level to see why it is that Traefik may be terminating the connection...

# Now enabling both a DEBUG log level and access logging, it revealed the important bits...
time="2024-04-17T11:47:54+01:00" level=debug msg="'502 Bad Gateway' caused by: readfrom tcp 172.18.1.100:47366->172.18.111.1:3001: i/o timeout"
10.10.0.10 - - [17/Apr/2024:10:46:54 +0000] "POST /api/asset/upload HTTP/2.0" 502 11 "-" "-" 201 "photos@docker" "http://172.18.111.1:3001/" 60000ms

The important thing here is it revealed that it was indeed Traefik killing the connection, and crucially why Traefik killed the connection, the readfrom config value was set to 60s and therefor triggering the i/o timeout. But this is new, no? It's never done this before. Why is it doing it now?

Burned by Watchtower. It automatically upgraded to Traefik v2.11.2 that has a breaking change in a minor semver patch version, a big no-no in the software game. There is an ongoing discussion as to why exactly this change came in, and its lead to a lot of heated discussion on their repo. It also revealed why the issue was so hard to track down information for online, as the issue was only opened less than a week ago.

Finding the fix!

Initial bug thread: traefik/traefik#10592
The culprit: traefik/traefik#10594 (comment)
The wider discussion thread: traefik/traefik#10598

Tucked away right at the bottom of the changelog, https://doc.traefik.io/traefik/migration/v2/#v2112, is this little doozy:

EntryPoint.Transport.RespondingTimeouts.ReadTimeout

Starting with v2.11.2 the entryPoints readTimeout option default value changed to 60 seconds.

For HTTP, this option defines the maximum duration for reading the entire request, including the body. For TCP, this option defines the maximum duration for the first bytes to be read on the connection.

The default value was previously set to zero, which means no timeout.

This change has been done to avoid Traefik instances with the default configuration to be hanging forever while waiting for bytes to be read on the connection.

Increasing the readTimeout value could be the solution notably if you are dealing with the following errors:

• TCP: Error while handling TCP connection: readfrom tcp X.X.X.X:X->X.X.X.X:X: read tcp X.X.X.X:X->X.X.X.X:X: i/o timeout
• HTTP: '499 Client Closed Request' caused by: context canceled
• HTTP: ReverseProxy read error during body copy: read tcp X.X.X.X:X->X.X.X.X:X: use of closed network connection

And there we go, the i/o timeout error being linked back to it being a change to the default EntryPoint.Transport.RespondingTimeouts.ReadTimeout value.

Checking the docs for this section, we can now apply a fix (notice that the docs now say it defaults to 60 seconds instead of being disabled, further confirming this as the culprit): https://doc.traefik.io/traefik/routing/entrypoints/#respondingtimeouts

entryPoints:
  ...
  name:
    ...
    # This is the fix:
    transport:
      respondingTimeouts:
        readTimeout: "0s"

This will revert readTimeout to its previous default value of being disabled, instead of 60 seconds, and solve the issue.

Happy uploading!

[gartsolutions]

Great job! We had the same issue! Rollback traefik version to 2.11.1 fixed issue.

[othyn]

Glad it helped! Yep, also a valid fix too

[truncsphere]

@othyn
Thanks for this, I'm just starting with Immich and have configured with Traefik already.
Did you add a port to your config for the timeout per the docs? I have it setup as such currently:

 [entyPoints.immich]
   [entryPoints.immich.transport]
     [entryPoints.immich.transport.respondingTimeouts]
       readTimeout = 0```

[lakemike]

Thanks! I'm using traefik 3.0.0 and I also had to change the timeout. This is how it looks like in docker-compose.yml if your endpoint is called websecure (command section):
- --entrypoints.websecure.transport.respondingTimeouts.readTimeout=0s

[AlexeyMatskevich]

If you use truenas scale and install immich from truecharts (with their traefik) - then you can add "Extra Args" in gui for traefik chart with value

--entryPoints.websecure.transport.respondingTimeouts.readTimeout=0

[Alfonce76]

do you need to add under http and https entry points if you have them? or is http enough?

[lakemike]

Not sure I fully understand your question. http would be immich itself and, hence, not required.
My endpoint is called websecure, other setups may call it https. This is where you need it.

[danni-storm]

I am on traefik 3.1 and it was driving me crazy that large uploads kept receiving gateway errors. Just pasted this into my traefik.yml file for websecure and web, restarted the traefik container, and it uploaded right away. Perfect! Thank you so much for figuring this out!

    transport:
      respondingTimeouts:
        readTimeout: 0s

[voltage6082]

Thank you all, I was desperately trying to solve this. I did not notice it with immich but with any other service (Synology DSM, Nextcloud, etc.). When uploading large files >2GB, the upload timed out and I got a standard error inside the application that the file could not be uplaoded. I rolled back to version v2.10.6 and it started working again.
I made the following change to my static configuration (traefik.yaml):

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
    transport:
      respondingTimeouts:
        readTimeout: 0s
  websecure:
    address: ":443"
    transport:
      respondingTimeouts:
        readTimeout: 0s

and now it works with the :latest release of traefik, currently 3.1.1
Thanks for your help!

[martins250]

Thanks for the above. I also came across the above problem. It needed both web and websecure timeout lines to fix (works with Traefik 3.1), as follows:

- --entrypoints.web.transport.respondingTimeouts.readTimeout=0s
- --entrypoints.websecure.transport.respondingTimeouts.readTimeout=0s

[alexrec-alt]

Traefik docs says default value for 'readTimeout' is '0s'. Anyways setting this value fixes upload 502 Bad Gateway issue. I decided to set this in config instead of labels, as there is number of proxied endpoints with download/upload options.