-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Open
Labels
area/databaseGalaxy's database or data access layerGalaxy's database or data access layer
Description
Describe the bug
Looks like a problem with celery -- but no idea why sqlalchemy is involved (I previously used postgres as backend but now redis).
galaxy.queue_worker ERROR 2026-02-02 12:52:21,842 [pN:main.4,p:1544777,tN:Thread-4 (check)] Error waiting for task: '{'task': 'reload_toolbox', 'kwargs': {}}' sent with routing key 'control.main.4@bioinf4'
Traceback (most recent call last):
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 997, in drain_events
get(self._deliver, timeout=timeout)
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/utils/scheduling.py", line 55, in get
return self.fun(resource, callback, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 1035, in _drain_channel
return channel.drain_events(callback=callback, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 754, in drain_events
return self._poll(self.cycle, callback, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 414, in _poll
return cycle.get(callback)
^^^^^^^^^^^^^^^^^^^
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/utils/scheduling.py", line 55, in get
return self.fun(resource, callback, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 417, in _get_and_deliver
message = self._get(queue)
^^^^^^^^^^^^^^^^
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/sqlalchemy/__init__.py", line 194, in _get
raise Empty()
_queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/gpfs1/data/galaxy_server/galaxy/lib/galaxy/queue_worker.py", line 136, in send_task
self.connection.drain_events(timeout=timeout)
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/connection.py", line 341, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 1000, in drain_events
raise socket.timeout()
TimeoutError
galaxy.queue_worker INFO 2026-02-02 12:52:21,851 [pN:main.4,p:1544777,tN:Thread-4 (check)] Sending reload_toolbox control task.
celery and redis logs look good.
Galaxy Version and/or server at which you observed the bug
Galaxy Version: 25.1
Metadata
Metadata
Assignees
Labels
area/databaseGalaxy's database or data access layerGalaxy's database or data access layer