Skip to content

Kratos operator targets wrong StatefulSet when pulled in as a neighbor app #567

@PeishuoCai

Description

@PeishuoCai

Bug Description

During charm-integration-testing runs, when kratos is pull in as the neighbor charm and attempts to patch, it targets a Kubernetes StatefulSet named kratos, instead of the correct name (neighbor). This mismatch causes repeated hook failures and test skips across multiple runs.

Additionally, the charm attempts to read /etc/config/kratos/kratos.yaml before it exists, causing a separate ops.pebble.PathError.

These two issues cause neighbor/0 to enter an error state during integration tests.

Example failing run:
https://test-observer.canonical.com/#/charms/289345?testExecutionId=289026&testResultId=7975559

Batch of skipped runs showing identical failures:
https://test-observer.canonical.com/#/test-results?families=charm&artefacts=loki-k8s&execution_metadata=charm_qa%3Acharm:kratos&issues=none

To Reproduce

refer to run:
https://test-observer.canonical.com/#/charms/289345?testExecutionId=289026&testResultId=7975559

Environment

ran on latest/stable

Relevant log output

2025-11-24T18:58:55.276Z [container-agent] 2025-11-24 18:58:55 ERROR juju-log Uncaught exception while in charm code:
2025-11-24T18:58:55.276Z [container-agent] Traceback (most recent call last):
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
2025-11-24T18:58:55.276Z [container-agent]     resp.raise_for_status()
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/httpx/_models.py", line 829, in raise_for_status
2025-11-24T18:58:55.276Z [container-agent]     raise HTTPStatusError(message, request=request, response=self)
2025-11-24T18:58:55.276Z [container-agent] httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/apis/apps/v1/namespaces/model-19645599047-251124185117/statefulsets/kratos?fieldManager=neighbor'
2025-11-24T18:58:55.276Z [container-agent] For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
2025-11-24T18:58:55.276Z [container-agent] 
2025-11-24T18:58:55.276Z [container-agent] During handling of the above exception, another exception occurred:
2025-11-24T18:58:55.276Z [container-agent] 
2025-11-24T18:58:55.276Z [container-agent] Traceback (most recent call last):
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/./src/charm.py", line 1350, in <module>
2025-11-24T18:58:55.276Z [container-agent]     main(KratosCharm)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/__init__.py", line 343, in __call__
2025-11-24T18:58:55.276Z [container-agent]     return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/_main.py", line 543, in main
2025-11-24T18:58:55.276Z [container-agent]     manager.run()
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/_main.py", line 529, in run
2025-11-24T18:58:55.276Z [container-agent]     self._emit()
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/_main.py", line 518, in _emit
2025-11-24T18:58:55.276Z [container-agent]     _emit_charm_event(self.charm, self.dispatcher.event_name, self._juju_context)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/_main.py", line 134, in _emit_charm_event
2025-11-24T18:58:55.276Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/framework.py", line 347, in emit
2025-11-24T18:58:55.276Z [container-agent]     framework._emit(event)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/framework.py", line 857, in _emit
2025-11-24T18:58:55.276Z [container-agent]     self._reemit(event_path)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/framework.py", line 947, in _reemit
2025-11-24T18:58:55.276Z [container-agent]     custom_handler(event)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/./src/charm.py", line 900, in _on_pebble_ready
2025-11-24T18:58:55.276Z [container-agent]     self._patch_statefulset()
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/./src/charm.py", line 978, in _patch_statefulset
2025-11-24T18:58:55.276Z [container-agent]     self.client.patch(StatefulSet, name=self.meta.name, namespace=self.model.name, obj=patch)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/lightkube/core/client.py", line 325, in patch
2025-11-24T18:58:55.276Z [container-agent]     return self._client.request("patch", res=res, name=name, namespace=namespace, obj=obj,
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
2025-11-24T18:58:55.276Z [container-agent]     return self.handle_response(method, resp, br)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
2025-11-24T18:58:55.276Z [container-agent]     self.raise_for_status(resp)
2025-11-24T18:58:55.276Z [container-agent]   File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
2025-11-24T18:58:55.276Z [container-agent]     raise transform_exception(e)
2025-11-24T18:58:55.276Z [container-agent] lightkube.core.exceptions.ApiError: statefulsets.apps "kratos" not found
2025-11-24T18:58:55.615Z [container-agent] 2025-11-24 18:58:55 ERROR juju.worker.uniter.operation runhook.go:180 hook "kratos-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1



unit-neighbor-0: 2025-11-24 19:05:21 ERROR unit.neighbor/0.juju-log pg-database:21: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-neighbor-0/charm/./src/charm.py", line 1350, in <module>
    main(KratosCharm)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/__init__.py", line 343, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/_main.py", line 543, in main
    manager.run()
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/_main.py", line 529, in run
    self._emit()
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/_main.py", line 518, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name, self._juju_context)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/_main.py", line 134, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/framework.py", line 347, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/framework.py", line 857, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/framework.py", line 947, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2213, in _on_relation_changed_event
    getattr(self.on, "database_created").emit(
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/framework.py", line 347, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/framework.py", line 857, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/framework.py", line 947, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/./src/charm.py", line 1014, in _on_database_created
    self._handle_status_update_config(event)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/./src/charm.py", line 867, in _handle_status_update_config
    self._restart_service()
  File "/var/lib/juju/agents/unit-neighbor-0/charm/src/utils.py", line 81, in wrapper
    current_config = charm._container.pull(CONFIG_FILE_PATH).read()
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/model.py", line 2524, in pull
    return self._pebble.pull(str(path), encoding=encoding)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/pebble.py", line 2432, in pull
    self._raise_on_path_error(resp, path)
  File "/var/lib/juju/agents/unit-neighbor-0/charm/venv/ops/pebble.py", line 2457, in _raise_on_path_error
    raise PathError(error['kind'], error['message'])
ops.pebble.PathError: not-found - stat /etc/config/kratos/kratos.yaml: no such file or directory

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions