Cannot create cross-controller relation sysbench/lxd->postgresql/microk8s: ERROR creating remote secret client: cannot resolve "controller-service.controller-m2k8s.svc.cluster.local"... i/o timeout

[taurus-forever]

Steps to reproduce

1. bootstrap lxd and microk8s controllers on the same client
2. deploy posrtgresql-k8s on microk8s (14/edge)
3. deploy sysbench on lxd
4. juju offer posrtgresql-k8s:database # on microk8s
5. juju consume microk8s:admin/mymodel.postgresql-k8s
6. juju relate sysbench postgresql-k8s

Expected behavior

Succesful relation to workaround #37

Actual behavior

ubuntu@juju340:~$ controller-0: 08:30:47 INFO juju.worker.firewaller start "sysbench:postgresql postgresql-k8s:database"
unit-sysbench-0: 08:30:48 INFO juju.worker.uniter.operation ran "postgresql-relation-created" hook (via hook dispatching script: dispatch)
unit-sysbench-0: 08:30:48 INFO juju.worker.uniter.operation ran "postgresql-relation-joined" hook (via hook dispatching script: dispatch)
unit-sysbench-0: 08:30:49 INFO juju.worker.uniter.operation ran "postgresql-relation-changed" hook (via hook dispatching script: dispatch)
controller-0: 08:30:52 INFO juju.worker.firewaller stopped "sysbench:postgresql postgresql-k8s:database", err: cannot open facade to remote model to publish network change: cannot resolve "controller-service.controller-mk8s.svc.cluster.local": lookup controller-service.controller-mk8s.svc.cluster.local: i/o timeout
controller-0: 08:30:52 INFO juju.worker.firewaller non-fatal error "sysbench:postgresql postgresql-k8s:database": cannot open facade to remote model to publish network change: cannot resolve "controller-service.controller-mk8s.svc.cluster.local": lookup controller-service.controller-mk8s.svc.cluster.local: i/o timeout
controller-0: 08:30:52 ERROR juju.worker.firewaller exited "sysbench:postgresql postgresql-k8s:database": cannot open facade to remote model to publish network change: cannot resolve "controller-service.controller-mk8s.svc.cluster.local": lookup controller-service.controller-mk8s.svc.cluster.local: i/o timeout
controller-0: 08:30:52 INFO juju.worker.firewaller restarting "sysbench:postgresql postgresql-k8s:database" in 1m0s
unit-sysbench-0: 08:30:52 ERROR unit.sysbench/0.juju-log postgresql:1: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3022, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-sysbench-0/secret-get', 'secret://77fe3914-0206-4d12-8735-32ebe4c82a49/corif6fmp25c760hv220', '--label', 'postgresql.1.user.secret', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-sysbench-0/charm/./src/charm.py", line 325, in <module>
    main(SysbenchOperator)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2846, in _on_relation_changed_event
    self.relation_data._register_secrets_to_relation(event.relation, diff.added)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1604, in _register_secrets_to_relation
    self._register_secret_to_relation(
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1590, in _register_secret_to_relation
    CachedSecret(self._model, self.component, label, secret_id).meta
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 615, in meta
    self._secret_meta = self._model.get_secret(id=self._secret_uri, label=self.label)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 280, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3370, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3024, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR creating remote secret client: cannot resolve "controller-service.controller-mk8s.svc.cluster.local": lookup controller-service.controller-mk8s.svc.cluster.local: i/o timeout
...

unit-sysbench-0: 08:31:06 ERROR juju.worker.uniter.operation hook "postgresql-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-sysbench-0: 08:31:06 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-sysbench-0: 08:31:16 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-sysbench-0: 08:31:16 ERROR unit.sysbench/0.juju-log postgresql:1: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3022, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-sysbench-0/secret-get', 'secret://77fe3914-0206-4d12-8735-32ebe4c82a49/corif6fmp25c760hv220', '--label', 'postgresql.1.user.secret', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-sysbench-0/charm/./src/charm.py", line 325, in <module>
    main(SysbenchOperator)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2846, in _on_relation_changed_event
    self.relation_data._register_secrets_to_relation(event.relation, diff.added)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1604, in _register_secrets_to_relation
    self._register_secret_to_relation(
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1590, in _register_secret_to_relation
    CachedSecret(self._model, self.component, label, secret_id).meta
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 615, in meta
    self._secret_meta = self._model.get_secret(id=self._secret_uri, label=self.label)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 280, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3370, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3024, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR getting cluster client: unable to determine legacy status for namespace "test59": Get "https://127.0.0.1:16443/api/v1/namespaces/test59": dial tcp 127.0.0.1:16443: connect: connection refused

unit-sysbench-0: 08:31:16 ERROR juju.worker.uniter.operation hook "postgresql-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-sysbench-0: 08:31:16 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook

Versions

Operating system: Ubuntu 23.10
Juju CLI: 3.4.2-genericlinux-arm64
Juju agent: see above
Charm revision: see above
LXD: 5.21.1-2d13beb rev28463
MicroK8s: MicroK8s v1.30.0 revision 6788 (strict)

Juju status:

ubuntu@juju340:~$ juju status
Model   Controller  Cloud/Region         Version  SLA          Timestamp
test61  lxd         localhost/localhost  3.4.2    unsupported  08:32:25+02:00

SAAS            Status  Store  URL
postgresql-k8s  active  mk8s   admin/test59.postgresql-k8s

App       Version  Status  Scale  Charm     Channel        Rev  Exposed  Message
sysbench           error       1  sysbench  latest/stable   26  no       hook failed: "postgresql-relation-changed"

Unit         Workload  Agent  Machine  Public address  Ports  Message
sysbench/0*  error     idle   0        10.184.219.31          hook failed: "postgresql-relation-changed"

Machine  State    Address        Inst id        Base          AZ  Message
0        started  10.184.219.31  juju-9e6ab4-0  ubuntu@22.04      Running
ubuntu@juju340:~$ juju status -m mk8s:test59
Model   Controller  Cloud/Region        Version  SLA          Timestamp
test59  mk8s        microk8s/localhost  3.4.2    unsupported  08:32:38+02:00

App                  Version  Status  Scale  Charm                Channel        Rev  Address         Exposed  Message
postgresql-k8s       14.11    active      1  postgresql-k8s       14/edge        247  10.152.183.162  no       

Unit                    Workload  Agent  Address       Ports  Message
postgresql-k8s/0*       active    idle   10.1.204.230         Primary

Offer           Application     Charm           Rev  Connected  Endpoint  Interface          Role
postgresql-k8s  postgresql-k8s  postgresql-k8s  247  1/1        database  postgresql_client  provider
ubuntu@juju340:~$ 

[github-actions]

https://warthogs.atlassian.net/browse/DPE-4288

[phvalguima]

@taurus-forever hey, that seems to be an issue one step before the sysbench charm itself. You are trying to reach out to the internal cluster record of the controller. That is not resolvable from the outside.

I think at least one of the following options must be set at bootstrap:

controller-external-ips:
  type: list
  description: Specifies a comma separated list of external IPs for a k8s controller
    of type external
controller-external-name:
  type: string
  description: Sets the external name for a k8s controller of type external
controller-service-type:
  type: string
  description: |-
    Controls the kubernetes service type for Juju controllers, see
    https://kubernetes.io/docs/reference/kubernetes-api/service-resources/service-v1/#ServiceSpec
    valid values are one of cluster, loadbalancer, external