Unscheduable garbage collection job and how I solved it

[SISheogorath]

The other day I had a garbage collection job that was never picked up by a worker from the jobservice. All other queues worked fine, even scheduling one worked, but it was never picked up.

Since I never found the answer despite seeing some issues showing similar problems e.g. #20029, #19494 (comment), #18900

I dug down and found the problem: A stale lock for the job in redis.

Caution

This is low level tinkering with your setup and might break it, follow at your own risk. Please use your brain, thanks! 🧠

So If anyone experiences that a job is stuck or not picked up, like in my case the GARBAGE_COLLECTION, here is how I solved it:

1. Stop all jobs
2. Restart the job service to make sure there are no more jobs running
3. Connect redis in the namespace of your jobservice (see your harbor config for details, it was redis-cli -n 1 for me)
4. Find all garbage collection keys using keys *GARBAGE_COLLECTION*, if it's broken the same way as mine, it looks like this:

   127.0.0.1:6379[1]> keys *GARBAGE_COLLECTION*
   1) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION"
   2) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:lock_info"
   3) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:19b507bb4308eac2e7711356:inprogress"
   4) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:max_concurrency"
   5) "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:lock"
   

5. Check if the job is locked: GET "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:lock" it should return "1" if it's locked (which is the origin of the problem)
6. Delete the unnecessary inprogress entry (remember, we stopped/restarted the job service to make sure nothing is running): DEL "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:19b507bb4308eac2e7711356:inprogress"
7. Unlock the job using: SET "{harbor_job_service_namespace}:jobs:GARBAGE_COLLECTION:lock" "0"
8. Schedule a new garbage collection job and see it run 🏃🏻

Since this topic seems to come up similarly ever now and then, I couldn't find the answer in any of the threads, I hope this is a good place to provide this info.

[MalteMagnussen]

Thank you @SISheogorath

[stonezdj]

Because there is a risk of another background GC job still running, you should restart all jobservice pod before you initiate a new GC job.

[wy65701436]

GC is intentionally designed to run as a unique job in Harbor. Harbor uses a Redis key to ensure that only one GC task runs at any given time — this MUST remain a unique task. If you need to stop a running GC job and start a new one, the steps you've taken are correct. However, please make sure to restart all JobService pods to ensure the underlying Go routine is properly terminated.

We’re aware of several known issues with GC, such as performance limitations and jobs becoming invisible after 24 hours. We're actively working to address these problems — thank you for your contribution.

@stonezdj — should we consider adding this to the FAQ?