psycopg2.errors.UniqueViolation

[bernt-matthias]

Describe the bug

I have the following in my logs:

galaxy.queue_worker ERROR 2025-09-04 12:15:09,118 [pN:main.1,p:4089112,tN:Thread-3 (run_postfork)] Error queueing async task: '{'task': 'rebuild_toolbox_search_index', 'kwargs': {}}'. for control.main.1@bioinf4
Traceback (most recent call last):
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "kombu_message_pkey"
DETAIL:  Key (id)=(113954) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/queue_worker.py", line 112, in send_task
    producer.publish(
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/kombu/messaging.py", line 186, in publish
    return _publish(
           ^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/kombu/connection.py", line 556, in _ensured
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/kombu/messaging.py", line 208, in _publish
    return channel.basic_publish(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 614, in basic_publish
    return self._put(routing_key, message, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/kombu/transport/sqlalchemy/__init__.py", line 166, in _put
    self.session.commit()
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2028, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1313, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1288, in _prepare_impl
    self.session.flush()
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4352, in flush
    self._flush(objects)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4487, in _flush
    with util.safe_reraise():
         ^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4448, in _flush
    flush_context.execute()
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
    return meth(
           ^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "kombu_message_pkey"
DETAIL:  Key (id)=(113954) already exists.

[SQL: INSERT INTO kombu_message (id, visible, timestamp, payload, version, queue_id) VALUES (nextval('message_id_sequence'), %(visible)s, %(timestamp)s, %(payload)s, %(version)s, %(queue_id)s) RETURNING kombu_message.id]
[parameters: {'visible': True, 'timestamp': None, 'payload': '{"body": "eyJ0YXNrIjogInJlYnVpbGRfdG9vbGJveF9zZWFyY2hfaW5kZXgiLCAia3dhcmdzIjoge319", "content-encoding": "utf-8", "content-type": "application/json", ... (143 characters truncated) ... ge": "", "routing_key": "control.main.1@bioinf4"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "2e20a4fa-00a6-4e49-8425-8da433e491b6"}}', 'version': 1, 'queue_id': 657}]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

I'm guessing that this is related to celery. Until today I was using postgres as backend. Now I configured redis, but the problem persisted. Likely that I misconfigured, but celery.log looks good:

 -------------- celery@bioinf4 v5.4.0 (opalescent)
--- ***** ----- 
-- ******* ---- Linux-5.14.0-503.26.1.el9_5.x86_64-x86_64-with-glibc2.34 2025-09-04 12:13:59
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         galaxy:0x7fa2042edb80
- ** ---------- .> transport:   redis://127.0.0.1:6379/11
- ** ---------- .> results:     redis://127.0.0.1:6379/10
- *** --- * --- .> concurrency: 2 (thread)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                .> galaxy.external  exchange=galaxy.external(direct) key=galaxy.external
                .> galaxy.internal  exchange=galaxy.internal(direct) key=galaxy.internal

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 24.2

To Reproduce

Only local: I startup my dev instance.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[mvdbeek]

Do you maybe have two processes running with the same name ?

[bernt-matthias]

Could be. My production and test instance run on the same machine. Where would I specify a name (and for what). In the end I'm just starting via run.sh (systemd service configured by our Hpc admins.

[mvdbeek]

forgot to mention, this is not celery but the internal queue worker / ipc task that uses amqp_internal_connection which falls back to the default database connection if it is not set.

https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/config/__init__.py#L1063-L1079 is how the server_name is determined, but if this is separate instances they should be using separate databases and there shouldn't be any issues.