[BUG] all services logging errors in v1.84 and can not get web or mobile to load

[ghost]

The bug

getting lots of errors after upgrading to 1.84

worked fine to start with, but updated the iOS app on the phone and now just getting server 500 errors in the web and mobile app.. most errors seem to come from typesets

         Name                        Command               State                  Ports                
-------------------------------------------------------------------------------------------------------
immich_machine_learning   tini -- ./start.sh               Up                                          
immich_microservices      tini -- /bin/sh start.sh m ...   Up      3001/tcp                            
immich_postgres           docker-entrypoint.sh postgres    Up      5432/tcp                            
immich_proxy              /docker-entrypoint.sh ngin ...   Up      0.0.0.0:80->8080/tcp,:::80->8080/tcp
immich_redis              docker-entrypoint.sh redis ...   Up      6379/tcp                            
immich_server             tini -- /bin/sh start.sh i ...   Up      3001/tcp                            
immich_typesense          /opt/typesense-server            Up      8108/tcp                            
immich_web                tini -- /bin/sh entrypoint.sh    Up      3000/tcp                            
root@immich:~/immich-app# docker ps
CONTAINER ID   IMAGE                                                COMMAND                  CREATED         STATUS          PORTS                                   NAMES
f16fdfa39499   ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   8 minutes ago   Up 41 seconds   0.0.0.0:80->8080/tcp, :::80->8080/tcp   immich_proxy
183368e2782e   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   8 minutes ago   Up 42 seconds   3001/tcp                                immich_server
2b9ad892b652   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   8 minutes ago   Up 42 seconds   3001/tcp                                immich_microservices
f291c0c12a3a   redis:6.2-alpine                                     "docker-entrypoint.s…"   8 minutes ago   Up 42 seconds   6379/tcp                                immich_redis
090b09d021ff   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     8 minutes ago   Up 42 seconds                                           immich_machine_learning
0eae301a664c   typesense/typesense:0.24.1                           "/opt/typesense-serv…"   8 minutes ago   Up 42 seconds   8108/tcp                                immich_typesense
50138c9bd5e4   ghcr.io/immich-app/immich-web:release                "tini -- /bin/sh ent…"   8 minutes ago   Up 42 seconds   3000/tcp                                immich_web
1076f28b948e   postgres:14-alpine                                   "docker-entrypoint.s…"   8 minutes ago   Up 42 seconds   5432/tcp                                immich_postgres

after some time

         Name                        Command                State                    Ports                
----------------------------------------------------------------------------------------------------------
immich_machine_learning   tini -- ./start.sh               Up                                             
immich_microservices      tini -- /bin/sh start.sh m ...   Exit 134                                       
immich_postgres           docker-entrypoint.sh postgres    Exit 1                                         
immich_proxy              /docker-entrypoint.sh ngin ...   Up         0.0.0.0:80->8080/tcp,:::80->8080/tcp
immich_redis              docker-entrypoint.sh redis ...   Up         6379/tcp                            
immich_server             tini -- /bin/sh start.sh i ...   Exit 1                                         
immich_typesense          /opt/typesense-server            Up         8108/tcp                            
immich_web                tini -- /bin/sh entrypoint.sh    Up         3000/tcp                            
root@immich:~/immich-app# docker ps
CONTAINER ID   IMAGE                                                COMMAND                  CREATED          STATUS                         PORTS                                   NAMES
f16fdfa39499   ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   20 minutes ago   Up 12 minutes                  0.0.0.0:80->8080/tcp, :::80->8080/tcp   immich_proxy
183368e2782e   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   20 minutes ago   Up 9 minutes                   3001/tcp                                immich_server
2b9ad892b652   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   20 minutes ago   Up 12 minutes                  3001/tcp                                immich_microservices
f291c0c12a3a   redis:6.2-alpine                                     "docker-entrypoint.s…"   20 minutes ago   Up 12 minutes                  6379/tcp                                immich_redis
090b09d021ff   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     20 minutes ago   Up 12 minutes                                                          immich_machine_learning
0eae301a664c   typesense/typesense:0.24.1                           "/opt/typesense-serv…"   20 minutes ago   Up 12 minutes                  8108/tcp                                immich_typesense
50138c9bd5e4   ghcr.io/immich-app/immich-web:release                "tini -- /bin/sh ent…"   20 minutes ago   Up 12 minutes                  3000/tcp                                immich_web
1076f28b948e   postgres:14-alpine                                   "docker-entrypoint.s…"   20 minutes ago   Restarting (1) 7 minutes ago                                           immich_postgres

The OS that Immich Server is running on

ubuntu 22

Version of Immich Server

1.84

Version of Immich Mobile App

latest today

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

default

Your .env content

default

Reproduction steps

1.
2.
3.
...

Additional information

immich_typesense           | E20231102 09:13:12.495561   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.495658   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.495747   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.495829   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.495914   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.495994   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496078   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496163   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496245   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496345   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496434   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496521   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496608   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496691   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496780   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496865   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.496949   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.497061   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.497184   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.497293   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.497406   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.497514   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.497628   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:13:12.497740   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_server              | Request #1698916334394: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server              | Request #1698916334394: Sleeping for 4s and then retrying request...
immich_typesense           | E20231102 09:13:14.599668   358 raft_server.cpp:624] 1988 queued writes > healthy read lag of 1000
immich_typesense           | E20231102 09:13:14.599679   358 raft_server.cpp:636] 1988 queued writes > healthy write lag of 500
immich_microservices       | Request #1698916347097: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices       | Request #1698916347097: Sleeping for 4s and then retrying request...
immich_typesense           | W20231102 09:13:15.233565   425 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
immich_typesense           | W20231102 09:13:16.940913   425 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
immich_server              | /usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23
immich_server              |         var _this = _super.call(this, message) || this;
immich_server              |                            ^
immich_server              | 
immich_server              | ServerError: Request failed with HTTP code 503 | Server said: Not Ready or Lagging
immich_server              |     at ServerError.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
immich_server              |     at new ServerError (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ServerError.js:25:42)
immich_server              |     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:376:21)
immich_server              |     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:225:58
immich_server              |     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
immich_server              |     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
immich_server              |     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
immich_server              |     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
immich_server              |     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
immich_server              |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
immich_server              |   httpStatus: 503
immich_server              | }
immich_server              | 
immich_server              | Node.js v20.8.1
immich_typesense           | W20231102 09:13:18.530943   425 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
immich_typesense           | W20231102 09:13:18.838958   425 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
immich_typesense           | W20231102 09:13:19.053009   425 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
immich_microservices       | Request #1698916347097: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices       | Request #1698916347097: Sleeping for 4s and then retrying request...
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [NestFactory] Starting Nest application...
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +32ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +4ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +157ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] AppModule dependencies initialized +5ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [InstanceLoader] DomainModule dependencies initialized +13ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] ActivityController {/activity}: +106ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/activity, GET} route +3ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/activity/statistics, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/activity, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/activity/:id, DELETE} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] AssetController {/asset}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/map-marker, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/memory-lane, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/random, GET} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/download/info, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/download/archive, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/download/:id, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/statistics, GET} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/time-buckets, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/time-bucket, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/jobs, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset, PUT} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/restore, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/trash/empty, POST} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/trash/restore, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/stack/parent, PUT} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/:id, PUT} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] AssetController {/asset}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/upload, POST} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/import, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/file/:id, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/thumbnail/:id, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/curated-objects, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/curated-locations, GET} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/search-terms, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/search, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/:deviceId, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/assetById/:id, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/exist, POST} route +2ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/asset/bulk-upload-check, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] AppController {/}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/refresh-config, POST} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] AlbumController {/album}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album/count, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album/:id, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album/:id, PATCH} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album/:id/assets, PUT} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album/:id/assets, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album/:id/users, PUT} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/album/:id/user/:userId, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] APIKeyController {/api-key}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/api-key, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/api-key, GET} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/api-key/:id, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/api-key/:id, PUT} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/api-key/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] AuditController {/audit}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/audit/deletes, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/audit/file-report, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/audit/file-report/checksum, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/audit/file-report/fix, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] AuthController {/auth}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/auth/login, POST} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/auth/admin-sign-up, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/auth/devices, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/auth/devices, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/auth/devices/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/auth/validateToken, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/auth/change-password, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/auth/logout, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] JobController {/jobs}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/jobs, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/jobs/:id, PUT} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] LibraryController {/library}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/library, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/library, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/library/:id, PUT} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/library/:id, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/library/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/library/:id/statistics, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/library/:id/scan, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/library/:id/removeOffline, POST} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] OAuthController {/oauth}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/oauth/mobile-redirect, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/oauth/config, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/oauth/authorize, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/oauth/callback, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/oauth/link, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/oauth/unlink, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] PartnerController {/partner}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/partner, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/partner/:id, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/partner/:id, DELETE} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] SearchController {/search}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/search/person, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/search, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/search/explore, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] ServerInfoController {/server-info}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/server-info, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/server-info/ping, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/server-info/version, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/server-info/features, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/server-info/theme, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/server-info/config, GET} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/server-info/stats, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/server-info/media-types, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] SharedLinkController {/shared-link}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/shared-link, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/shared-link/me, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/shared-link/:id, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/shared-link, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/shared-link/:id, PATCH} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/shared-link/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/shared-link/:id/assets, PUT} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/shared-link/:id/assets, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] SystemConfigController {/system-config}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/system-config, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/system-config/defaults, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/system-config, PUT} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/system-config/storage-template-options, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RoutesResolver] TagController {/tag}: +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/tag, POST} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/tag, GET} route +1ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/tag/:id, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/tag/:id, PATCH} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/tag/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/tag/:id/assets, GET} route +0ms
immich_server              | [Nest] 7  - 11/02/2023, 7:13:20 PM     LOG [RouterExplorer] Mapped {/tag/:id/assets, PUT} route +0ms
immich_typesense           | E20231102 09:17:27.830929   425 collection.cpp:340] Update to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:27.833989   425 collection.cpp:340] Update to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:27.845978   425 collection.cpp:340] Update to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:27.852667   425 collection.cpp:340] Update to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:27.856185   425 collection.cpp:340] Update to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:27.864369   425 collection.cpp:340] Update to disk failed. Will restore old document
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:29 PM     LOG [NestFactory] Starting Nest application...
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:29 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +34ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:29 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:29 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:29 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:29 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:29 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +5ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:29 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:29 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | /usr/src/app/node_modules/redis-parser/lib/parser.js:179
immich_microservices       |     return new ReplyError(string)
immich_microservices       |            ^
immich_microservices       | 
immich_microservices       | ReplyError: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.
immich_microservices       |     at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:179:12)
immich_microservices       |     at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:302:14) {
immich_microservices       |   command: {
immich_microservices       |     name: 'hset',
immich_microservices       |     args: [
immich_microservices       |       'immich_bull:thumbnailGeneration:meta',
immich_microservices       |       'opts.maxLenEvents',
immich_microservices       |       '10000'
immich_microservices       |     ]
immich_microservices       |   }
immich_microservices       | }
immich_microservices       | 
immich_microservices       | Node.js v20.8.1
immich_microservices exited with code 1
immich_redis               | 1:M 02 Nov 2023 09:17:30.015 * 100 changes in 300 seconds. Saving...
immich_redis               | 1:M 02 Nov 2023 09:17:30.015 * Background saving started by pid 18
immich_redis               | 18:C 02 Nov 2023 09:17:30.019 # Write error saving DB on disk: No space left on device
immich_redis               | 1:M 02 Nov 2023 09:17:30.116 # Background saving error
immich_server              | Request #1698916602115: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server              | Request #1698916602115: Sleeping for 4s and then retrying request...
immich_typesense           | W20231102 09:17:31.800505   425 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:31 PM     LOG [NestFactory] Starting Nest application...
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:32 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +35ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:32 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:32 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:32 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:32 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:32 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +5ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:32 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:32 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | /usr/src/app/node_modules/redis-parser/lib/parser.js:179
immich_microservices       |     return new ReplyError(string)
immich_microservices       |            ^
immich_microservices       | 
immich_microservices       | ReplyError: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.
immich_microservices       |     at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:179:12)
immich_microservices       |     at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:302:14) {
immich_microservices       |   command: {
immich_microservices       |     name: 'hset',
immich_microservices       |     args: [
immich_microservices       |       'immich_bull:thumbnailGeneration:meta',
immich_microservices       |       'opts.maxLenEvents',
immich_microservices       |       '10000'
immich_microservices       |     ]
immich_microservices       |   }
immich_microservices       | }
immich_microservices       | 
immich_microservices       | Node.js v20.8.1
immich_microservices exited with code 1
immich_typesense           | W20231102 09:17:32.621006   425 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
immich_server              | Request #1698916602115: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server              | Request #1698916602115: Sleeping for 4s and then retrying request...
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:34 PM     LOG [NestFactory] Starting Nest application...
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:34 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +34ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:34 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:34 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:34 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:34 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:34 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +5ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:34 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 11/02/2023, 7:17:34 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | /usr/src/app/node_modules/redis-parser/lib/parser.js:179
immich_microservices       |     return new ReplyError(string)
immich_microservices       |            ^
immich_microservices       | 
immich_microservices       | ReplyError: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.
immich_microservices       |     at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:179:12)
immich_microservices       |     at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:302:14) {
immich_microservices       |   command: {
immich_microservices       |     name: 'hset',
immich_microservices       |     args: [
immich_microservices       |       'immich_bull:thumbnailGeneration:meta',
immich_microservices       |       'opts.maxLenEvents',
immich_microservices       |       '10000'
immich_microservices       |     ]
immich_microservices       |   }
immich_microservices       | }
immich_microservices       | 
immich_microservices       | Node.js v20.8.1
immich_microservices exited with code 1
immich_typesense           | E20231102 09:17:35.622918   358 raft_server.cpp:624] 1328 queued writes > healthy read lag of 1000
immich_typesense           | E20231102 09:17:35.622938   358 raft_server.cpp:636] 1328 queued writes > healthy write lag of 500
immich_redis               | 1:M 02 Nov 2023 09:17:36.029 * 100 changes in 300 seconds. Saving...
immich_redis               | 1:M 02 Nov 2023 09:17:36.030 * Background saving started by pid 19
immich_redis               | 19:C 02 Nov 2023 09:17:36.030 # Write error saving DB on disk: No space left on device
immich_redis               | 1:M 02 Nov 2023 09:17:36.130 # Background saving error
immich_typesense           | E20231102 09:17:36.212462   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.212574   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.212651   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.212726   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.212800   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.212888   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.212983   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213068   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213156   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213254   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213352   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213446   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213533   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213629   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213722   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213806   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213900   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.213986   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.214071   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.214159   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.214268   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.214377   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.214483   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.214592   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.214679   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.214782   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.214882   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.215003   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.215093   425 collection.cpp:364] Write to disk failed. Will restore old document
immich_typesense           | E20231102 09:17:36.215206   425 collection.cpp:364] Write to disk failed. Will restore old document

[bo0tzz]

Converted to a discussion as this is not an Immich bug.

Write error saving DB on disk: No space left on device

Your disk is full

[ghost]

well what you know.. yes it is.. not the upload dir, that is on a NAS.. but the container drive is.. was only 50% yesterday.. and full now.. I will expand it and see what happens :)

[ghost]

ok more space.. things starting better.. but lots of log info and still contact connect

E20231102 09:43:50.595288   358 raft_server.cpp:636] 1927 queued writes > healthy write lag of 500
W20231102 09:43:50.612579   433 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
W20231102 09:43:50.814769   433 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
W20231102 09:43:53.335028   433 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
E20231102 09:43:59.596086   358 raft_server.cpp:624] 1902 queued writes > healthy read lag of 1000
E20231102 09:43:59.596101   358 raft_server.cpp:636] 1902 queued writes > healthy write lag of 500
W20231102 09:44:06.903726   433 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
W20231102 09:44:07.579548   433 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
E20231102 09:44:08.596884   358 raft_server.cpp:624] 1881 queued writes > healthy read lag of 1000
E20231102 09:44:08.596899   358 raft_server.cpp:636] 1881 queued writes > healthy write lag of 500

CONTAINER ID   IMAGE                                                COMMAND                  CREATED         STATUS          PORTS                                   NAMES
ced2236b2ad1   ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   8 minutes ago   Up 4 minutes    0.0.0.0:80->8080/tcp, :::80->8080/tcp   immich_proxy
7db097f8a270   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   8 minutes ago   Up 4 minutes    3001/tcp                                immich_microservices
0f1f5627fbf2   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   8 minutes ago   Up 18 seconds   3001/tcp                                immich_server
f74de1c6b84f   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     8 minutes ago   Up 4 minutes                                            immich_machine_learning
eef8dbf9273a   redis:6.2-alpine                                     "docker-entrypoint.s…"   8 minutes ago   Up 4 minutes    6379/tcp                                immich_redis
a76e7222a163   ghcr.io/immich-app/immich-web:release                "tini -- /bin/sh ent…"   8 minutes ago   Up 4 minutes    3000/tcp                                immich_web
526c0bef57e3   typesense/typesense:0.24.1                           "/opt/typesense-serv…"   8 minutes ago   Up 4 minutes    8108/tcp                                immich_typesense
e61fadb27a7e   postgres:14-alpine                                   "docker-entrypoint.s…"   8 minutes ago   Up 4 minutes    5432/tcp                                immich_postgres

[hooks.server.ts]:handleError connect ECONNREFUSED 172.18.0.7:3001
[hooks.server.ts] Unable to get my user {
  message: 'connect ECONNREFUSED 172.18.0.7:3001',
  code: 'ECONNREFUSED',
  stack: 'Error: connect ECONNREFUSED 172.18.0.7:3001\n' +
[hooks.server.ts]:handle connect ECONNREFUSED 172.18.0.7:3001
    '    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)'
}
[hooks.server.ts]:handleError connect ECONNREFUSED 172.18.0.7:3001
[hooks.server.ts] Unable to get my user {
[hooks.server.ts]:handle connect ECONNREFUSED 172.18.0.7:3001
  message: 'connect ECONNREFUSED 172.18.0.7:3001',
  code: 'ECONNREFUSED',
  stack: 'Error: connect ECONNREFUSED 172.18.0.7:3001\n' +
    '    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)'
}
[hooks.server.ts]:handleError connect ECONNREFUSED 172.18.0.7:3001

[ghost]

after 1000's of lines in the logs of

Request #1698918787875: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698918787875: Sleeping for 4s and then retrying request...

and

W20231102 10:00:38.413360   436 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
W20231102 10:00:40.327246   436 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
W20231102 10:00:40.364270   436 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
W20231102 10:00:40.554965   436 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found
W20231102 10:00:41.295492   436 index.cpp:5406] Error while removing field `embedding` from document, message: Label not found

after 10mins of it doing something (but not logging what) I can connect via mobile and web now

CONTAINER ID   IMAGE                                                COMMAND                  CREATED          STATUS          PORTS                                   NAMES
a240042307c6   ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   19 minutes ago   Up 19 minutes   0.0.0.0:80->8080/tcp, :::80->8080/tcp   immich_proxy
6d5a4f0ae6de   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   19 minutes ago   Up 13 minutes   3001/tcp                                immich_server
ad694bf6c6fb   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   19 minutes ago   Up 9 minutes    3001/tcp                                immich_microservices
ddd46ee9259d   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     20 minutes ago   Up 19 minutes                                           immich_machine_learning
8e51a343f2ed   redis:6.2-alpine                                     "docker-entrypoint.s…"   20 minutes ago   Up 19 minutes   6379/tcp                                immich_redis
a1ba82af0a31   typesense/typesense:0.24.1                           "/opt/typesense-serv…"   20 minutes ago   Up 19 minutes   8108/tcp                                immich_typesense
32e8928b39b8   ghcr.io/immich-app/immich-web:release                "tini -- /bin/sh ent…"   20 minutes ago   Up 19 minutes   3000/tcp                                immich_web
a3c74f7c4ab0   postgres:14-alpine                                   "docker-entrypoint.s…"   20 minutes ago   Up 19 minutes   5432/tcp                                immich_postgres