celery-scheduler restarting in loop

[benoitjpnet]

Hi,

I just deployed the project from scratch. Everything seems to work. But if I look at the logs or docker ps I can see that celery-scheduler container restarts in loop.

celery-scheduler-1  | 1000
celery-scheduler-1  | ERROR: Pidfile (celerybeat.pid) already exists.
celery-scheduler-1  | Seems we're already running? (pid: 13)
celery-scheduler-1  | celery beat v4.4.7 (cliffs) is starting.
celery-scheduler-1 exited with code 73

# docker ps | grep Restart
31f2e0bebc86   lardbit/nefarious          "/app/entrypoint-cel…"   About an hour ago   Restarting (73) 10 seconds ago                                                                                                         nefarious-celery-scheduler-1

I wonder if I missed something?

[lardbit]

It sounds like it's already running because the new container you're starting is seeing an existing pid file in the docker volume path, so it's refusing to start. Can you verify with : docker ps | grep celery

…

On Wed, Feb 19, 2025, 6:00 AM Benoit ***@***.***> wrote: Hi, I just deployed the project from scratch. Everything seems to work. But if I look at the logs or docker ps I can see that celery-scheduler container restarts in loop. celery-scheduler-1 | 1000 celery-scheduler-1 | ERROR: Pidfile (celerybeat.pid) already exists. celery-scheduler-1 | Seems we're already running? (pid: 13) celery-scheduler-1 | celery beat v4.4.7 (cliffs) is starting. celery-scheduler-1 exited with code 73 # docker ps | grep Restart 31f2e0bebc86 lardbit/nefarious "/app/entrypoint-cel…" About an hour ago Restarting (73) 10 seconds ago nefarious-celery-scheduler-1 I wonder if I missed something? — Reply to this email directly, view it on GitHub <#324>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKYIKNEFMBG3XZ4VOSKJCJD2QRW6DAVCNFSM6AAAAABXOAAY5OVHI2DSMVQWIX3LMV43ASLTON2WKOZSHA3DGMJSGQZTQNQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [image: benoitjpnet]*benoitjpnet* created an issue (lardbit/nefarious#324) <#324> Hi, I just deployed the project from scratch. Everything seems to work. But if I look at the logs or docker ps I can see that celery-scheduler container restarts in loop. celery-scheduler-1 | 1000 celery-scheduler-1 | ERROR: Pidfile (celerybeat.pid) already exists. celery-scheduler-1 | Seems we're already running? (pid: 13) celery-scheduler-1 | celery beat v4.4.7 (cliffs) is starting. celery-scheduler-1 exited with code 73 # docker ps | grep Restart 31f2e0bebc86 lardbit/nefarious "/app/entrypoint-cel…" About an hour ago Restarting (73) 10 seconds ago nefarious-celery-scheduler-1 I wonder if I missed something? — Reply to this email directly, view it on GitHub <#324>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKYIKNEFMBG3XZ4VOSKJCJD2QRW6DAVCNFSM6AAAAABXOAAY5OVHI2DSMVQWIX3LMV43ASLTON2WKOZSHA3DGMJSGQZTQNQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[benoitjpnet]

 docker ps | grep celery
566b2ad1dc5c   lardbit/nefarious          "/app/entrypoint-cel…"   2 hours ago      Up 23 minutes                    80/tcp                                                                                                nefarious-celery-1
31f2e0bebc86   lardbit/nefarious          "/app/entrypoint-cel…"   2 hours ago      Restarting (73) 41 seconds ago                                                                                                         nefarious-celery-scheduler-1

[lardbit]

Yeah looks like there's already one running which is why the other one can't start. Stop all celery containers with: docker ps --filter "name=celery" --format "{{.ID}}" | xargs docker rm -f Make sure they're all gone: docker ps | grep celery Then start them back up: docker-compose up -d

…

On Wed, Feb 19, 2025 at 6:16 AM Benoit ***@***.***> wrote: # docker ps | grep celery 566b2ad1dc5c lardbit/nefarious "/app/entrypoint-cel…" 2 hours ago Up 23 minutes 80/tcp nefarious-celery-1 31f2e0bebc86 lardbit/nefarious "/app/entrypoint-cel…" 2 hours ago Restarting (73) 41 seconds ago — Reply to this email directly, view it on GitHub <#324 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKYIKNGDPP66J4WZ7MHCTNL2QRY2FAVCNFSM6AAAAABXOAAY5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRYGQ4DKNBUHA> . You are receiving this because you commented.Message ID: ***@***.***> [image: benoitjpnet]*benoitjpnet* left a comment (lardbit/nefarious#324) <#324 (comment)> # docker ps | grep celery 566b2ad1dc5c lardbit/nefarious "/app/entrypoint-cel…" 2 hours ago Up 23 minutes 80/tcp nefarious-celery-1 31f2e0bebc86 lardbit/nefarious "/app/entrypoint-cel…" 2 hours ago Restarting (73) 41 seconds ago — Reply to this email directly, view it on GitHub <#324 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKYIKNGDPP66J4WZ7MHCTNL2QRY2FAVCNFSM6AAAAABXOAAY5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRYGQ4DKNBUHA> . You are receiving this because you commented.Message ID: ***@***.***>

[benoitjpnet]

Ok thank you that fixed it. Not sure what happened.

[benoitjpnet]

It's back again after a reboot, so I definitively think something is wrong. Could it be some kind of race condition when containers start?

[lardbit]

I'm trying to wrap my head around this. I haven't personally seen it so I can't reproduce it.

It's weird how the celery container that initially starts with docker-compose up -d isn't seen later when it's in the restarting phase, and docker keeps trying to start a new one even though the other one is still present & unhealthy.

Has the device experienced any power disruptions or hard reboots?

[lardbit]

I'm going to update how celery starts and tell it to not use a PID file (why yours is tripping up). I'll let you know when it's ready to test.

[lardbit]

@benoitjpnet I implemented a fix (#329) that should resolve this issue. Celery Scheduler now starts without specifying a PID file which should allow a new container to start when necessary.

This update will automatically rollout to you so you won't have to do anything.

[lardbit]

But reopen this ticket if you see this problem in the future.