Upstream time out - upstream: "uwsgi://127.0.0.1:9000"

[bigbeka]

Describe the issue
After uploading 3.25GB .mp4 file, the upload screen had Retry 2/2 and then failed.

Environment (please complete the following information):

• OS: Ubuntu 22.04
• Installation method: Docker install - no modifications - upgraded to latest release
• Browser: MacOS - Safari

Additional context

web-1            | 10.0.0.33 - - [18/Mar/2025:23:35:15 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "http://10.0.0.125/upload" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15"
web-1            | 2025/03/18 23:35:15 [warn] 71#71: *483 a client request body is buffered to a temporary file /var/lib/nginx/body/0000001609, client: 10.0.0.33, server: , request: "POST /fu/upload/ HTTP/1.1", host: "10.0.0.125", referrer: "http://10.0.0.125/upload"
web-1            | [pid: 77|app: 0|req: 952/1784] 10.0.0.33 () {54 vars in 1067 bytes} [Tue Mar 18 23:35:15 2025] POST /fu/upload/ => generated 17 bytes in 54 msecs (HTTP/1.1 200) 8 headers in 255 bytes (12 switches on core 1)
web-1            | 10.0.0.33 - - [18/Mar/2025:23:35:15 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "http://10.0.0.125/upload" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15"
web-1            | 2025/03/18 23:35:15 [warn] 71#71: *488 a client request body is buffered to a temporary file /var/lib/nginx/body/0000001610, client: 10.0.0.33, server: , request: "POST /fu/upload/ HTTP/1.1", host: "10.0.0.125", referrer: "http://10.0.0.125/upload"
web-1            | [pid: 77|app: 0|req: 952/1785] 10.0.0.33 () {54 vars in 1067 bytes} [Tue Mar 18 23:35:15 2025] POST /fu/upload/ => generated 17 bytes in 59 msecs (HTTP/1.1 200) 8 headers in 255 bytes (1 switches on core 0)
web-1            | 10.0.0.33 - - [18/Mar/2025:23:35:15 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "http://10.0.0.125/upload" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15"
web-1            | 2025/03/18 23:35:15 [warn] 71#71: *487 a client request body is buffered to a temporary file /var/lib/nginx/body/0000001611, client: 10.0.0.33, server: , request: "POST /fu/upload/ HTTP/1.1", host: "10.0.0.125", referrer: "http://10.0.0.125/upload"
web-1            | 10.0.0.33 - - [18/Mar/2025:23:35:15 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "http://10.0.0.125/upload" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15"
web-1            | [pid: 75|app: 0|req: 835/1786] 10.0.0.33 () {54 vars in 1067 bytes} [Tue Mar 18 23:35:15 2025] POST /fu/upload/ => generated 17 bytes in 53 msecs (HTTP/1.1 200) 8 headers in 255 bytes (1 switches on core 0)
web-1            | 10.0.0.33 - - [18/Mar/2025:23:35:15 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "http://10.0.0.125/upload" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15"
web-1            | [pid: 75|app: 0|req: 835/1787] 10.0.0.33 () {54 vars in 1067 bytes} [Tue Mar 18 23:35:15 2025] POST /fu/upload/ => generated 17 bytes in 50 msecs (HTTP/1.1 200) 8 headers in 255 bytes (1 switches on core 1)
web-1            | 10.0.0.33 - - [18/Mar/2025:23:35:15 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "http://10.0.0.125/upload" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15"
web-1            | [pid: 77|app: 0|req: 953/1788] 10.0.0.33 () {54 vars in 1065 bytes} [Tue Mar 18 23:35:15 2025] POST /fu/upload/ => generated 17 bytes in 43 msecs (HTTP/1.1 200) 8 headers in 255 bytes (1 switches on core 1)
migrations-1     | Running migrations service
migrations-1     | Operations to perform:
migrations-1     |   Apply all migrations: account, actions, admin, auth, authtoken, contenttypes, files, sessions, sites, socialaccount, users
migrations-1     | Running migrations:
migrations-1     |   No migrations to apply.
migrations-1     |   Your models in app(s): 'files' have changes that are not yet reflected in a migration, and so won't be applied.
migrations-1     |   Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
migrations-1     | Loaddata has already run
migrations-1     | RUNNING COLLECTSTATIC
migrations-1     | 
migrations-1     | 0 static files copied to '/home/mediacms.io/mediacms/static', 212 unmodified.
migrations-1 exited with code 0
migrations-1     | Running migrations service
migrations-1     | Operations to perform:
migrations-1     |   Apply all migrations: account, actions, admin, auth, authtoken, contenttypes, files, sessions, sites, socialaccount, users
migrations-1     | Running migrations:
migrations-1     |   No migrations to apply.
migrations-1     |   Your models in app(s): 'files' have changes that are not yet reflected in a migration, and so won't be applied.
migrations-1     |   Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
migrations-1     | Loaddata has already run
migrations-1     | RUNNING COLLECTSTATIC
migrations-1     | 
migrations-1     | 0 static files copied to '/home/mediacms.io/mediacms/static', 212 unmodified.
migrations-1 exited with code 0
migrations-1     | Running migrations service
web-1            | 2025/03/18 23:36:15 [error] 71#71: *483 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.0.33, server: , request: "POST /fu/upload/?done HTTP/1.1", upstream: "uwsgi://127.0.0.1:9000", host: "10.0.0.125", referrer: "http://10.0.0.125/upload"
web-1            | 10.0.0.33 - - [18/Mar/2025:23:36:15 +0000] "POST /fu/upload/?done HTTP/1.1" 504 167 "http://10.0.0.125/upload" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15"
web-1            | 10.0.0.33 - - [18/Mar/2025:23:36:20 +0000] "POST /fu/upload/?done HTTP/1.1" 400 56 "http://10.0.0.125/upload" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15"
web-1            | [pid: 77|app: 0|req: 955/1789] 10.0.0.33 () {50 vars in 929 bytes} [Tue Mar 18 23:36:20 2025] POST /fu/upload/?done => generated 56 bytes in 50 msecs (HTTP/1.1 400) 8 headers in 264 bytes (1 switches on core 1)
migrations-1     | Operations to perform:
migrations-1     |   Apply all migrations: account, actions, admin, auth, authtoken, contenttypes, files, sessions, sites, socialaccount, users
migrations-1     | Running migrations:
migrations-1     |   No migrations to apply.
migrations-1     |   Your models in app(s): 'files' have changes that are not yet reflected in a migration, and so won't be applied.
migrations-1     |   Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
web-1            | [pid: 75|app: 0|req: 836/1790] 10.0.0.33 () {50 vars in 929 bytes} [Tue Mar 18 23:36:25 2025] POST /fu/upload/?done => generated 56 bytes in 60 msecs (HTTP/1.1 400) 8 headers in 264 bytes (1 switches on core 0)
web-1            | 10.0.0.33 - - [18/Mar/2025:23:36:25 +0000] "POST /fu/upload/?done HTTP/1.1" 400 56 "http://10.0.0.125/upload" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15"
migrations-1     | Loaddata has already run
migrations-1     | RUNNING COLLECTSTATIC
migrations-1     | 
migrations-1     | 0 static files copied to '/home/mediacms.io/mediacms/static', 212 unmodified.
migrations-1 exited with code 0
migrations-1     | Running migrations service
migrations-1     | Operations to perform:
migrations-1     |   Apply all migrations: account, actions, admin, auth, authtoken, contenttypes, files, sessions, sites, socialaccount, users
migrations-1     | Running migrations:
migrations-1     |   No migrations to apply.
migrations-1     |   Your models in app(s): 'files' have changes that are not yet reflected in a migration, and so won't be applied.
migrations-1     |   Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
migrations-1     | Loaddata has already run
migrations-1     | RUNNING COLLECTSTATIC
migrations-1     | 
migrations-1     | 0 static files copied to '/home/mediacms.io/mediacms/static', 212 unmodified.
migrations-1 exited with code 0
migrations-1     | Running migrations service
migrations-1     | Operations to perform:
migrations-1     |   Apply all migrations: account, actions, admin, auth, authtoken, contenttypes, files, sessions, sites, socialaccount, users
migrations-1     | Running migrations:
migrations-1     |   No migrations to apply.
migrations-1     |   Your models in app(s): 'files' have changes that are not yet reflected in a migration, and so won't be applied.
migrations-1     |   Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
migrations-1     | Loaddata has already run
migrations-1     | RUNNING COLLECTSTATIC
migrations-1     | 
migrations-1     | 0 static files copied to '/home/mediacms.io/mediacms/static', 212 unmodified.
migrations-1 exited with code 0
web-1            | [pid: 77|app: 0|req: 955/1791] 10.0.0.33 () {50 vars in 929 bytes} [Tue Mar 18 23:35:15 2025] POST /fu/upload/?done => generated 51 bytes in 93800 msecs (HTTP/1.1 200) 8 headers in 255 bytes (1 switches on core 0)
migrations-1     | Running migrations service
migrations-1     | Operations to perform:
migrations-1     |   Apply all migrations: account, actions, admin, auth, authtoken, contenttypes, files, sessions, sites, socialaccount, users
migrations-1     | Running migrations:
migrations-1     |   No migrations to apply.
migrations-1     |   Your models in app(s): 'files' have changes that are not yet reflected in a migration, and so won't be applied.
migrations-1     |   Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
migrations-1     | Loaddata has already run
migrations-1     | RUNNING COLLECTSTATIC
migrations-1     | 
migrations-1     | 0 static files copied to '/home/mediacms.io/mediacms/static', 212 unmodified.

[yatesdr]

This may be related to a recent discussion #1283, where timeouts and bad gateways were observed under heavier loads. Can you try increasing the number of uwsgi workers + processes to see if it improves? I found that long-running processes were not being spawned once the default worker queue size was full in uwsgi. By default it's set to workers=2, processes=2, threads=2, and I recommend increasing this to set workers and processes equal to the number of CPU's available on your system.

Edit these in:
mediacms/deploy/docker/uwsgi.ini

Then redeploy and check to make sure the workers have spawned.