Skip to content

Kombu error #21713

@bernt-matthias

Description

@bernt-matthias

Describe the bug

Looks like a problem with celery -- but no idea why sqlalchemy is involved (I previously used postgres as backend but now redis).

galaxy.queue_worker ERROR 2026-02-02 12:52:21,842 [pN:main.4,p:1544777,tN:Thread-4 (check)] Error waiting for task: '{'task': 'reload_toolbox', 'kwargs': {}}' sent with routing key 'control.main.4@bioinf4'
Traceback (most recent call last):
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 997, in drain_events
    get(self._deliver, timeout=timeout)
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/utils/scheduling.py", line 55, in get
    return self.fun(resource, callback, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 1035, in _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 754, in drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 414, in _poll
    return cycle.get(callback)
           ^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/utils/scheduling.py", line 55, in get
    return self.fun(resource, callback, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 417, in _get_and_deliver
    message = self._get(queue)
              ^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/sqlalchemy/__init__.py", line 194, in _get
    raise Empty()
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfs1/data/galaxy_server/galaxy/lib/galaxy/queue_worker.py", line 136, in send_task
    self.connection.drain_events(timeout=timeout)
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/connection.py", line 341, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 1000, in drain_events
    raise socket.timeout()
TimeoutError
galaxy.queue_worker INFO 2026-02-02 12:52:21,851 [pN:main.4,p:1544777,tN:Thread-4 (check)] Sending reload_toolbox control task.

celery and redis logs look good.

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 25.1

Metadata

Metadata

Assignees

Labels

area/databaseGalaxy's database or data access layer

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions