etcd error: 100: Key not found (/service) #222

prensgold · 2022-11-24T13:42:19Z

prensgold
Nov 24, 2022

We want to install TYPE A .
end of the deployment script is giving error as below.
I couldn't solve it anyway..

RUNNING HANDLER [keepalived : Restart keepalived service] ***********************************************************************
fatal: [192.168.154.80]: FAILED! => {"changed": false, "msg": "Unable to start service keepalived: Job for keepalived.service failed because the control process exited with error code.\nSee \"systemctl status keepalived.service\" and \"journalctl -xe\" for details.\n"}
fatal: [192.168.154.79]: FAILED! => {"changed": false, "msg": "Unable to start service keepalived: Job for keepalived.service failed because the control process exited with error code.\nSee \"systemctl status keepalived.service\" and \"journalctl -xe\" for details.\n"}
fatal: [192.168.154.77]: FAILED! => {"changed": false, "msg": "Unable to start service keepalived: Job for keepalived.service failed because the control process exited with error code.\nSee \"systemctl status keepalived.service\" and \"journalctl -xe\" for details.\n"}

NO MORE HOSTS LEFT ***************

and keepalived.service log .

Nov 24 16:19:36 pgnode01 vip-manager[24344]: 2022/11/24 16:19:36 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:37 pgnode01 vip-manager[24344]: 2022/11/24 16:19:37 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:38 pgnode01 vip-manager[24344]: 2022/11/24 16:19:38 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:39 pgnode01 vip-manager[24344]: 2022/11/24 16:19:39 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:40 pgnode01 vip-manager[24344]: 2022/11/24 16:19:40 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:41 pgnode01 vip-manager[24344]: 2022/11/24 16:19:41 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:41 pgnode01 vip-manager[24344]: 2022/11/24 16:19:41 IP address 192.168.32.21/24 state is false, desired false
Nov 24 16:19:42 pgnode01 vip-manager[24344]: 2022/11/24 16:19:42 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:43 pgnode01 vip-manager[24344]: 2022/11/24 16:19:43 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:43 pgnode01 systemd[1]: Starting LVS and VRRP High Availability Monitor...
Nov 24 16:19:43 pgnode01 keepalived[236863]: /usr/sbin/keepalived: symbol lookup error: /lib64/libnetsnmpagent.so.35: undefined symbol: snmp_free_securityStateRef
Nov 24 16:19:43 pgnode01 systemd[1]: keepalived.service: Control process exited, code=exited status=127
Nov 24 16:19:43 pgnode01 systemd[1]: keepalived.service: Failed with result 'exit-code'.
Nov 24 16:19:43 pgnode01 systemd[1]: Failed to start LVS and VRRP High Availability Monitor.
Nov 24 16:19:44 pgnode01 vip-manager[24344]: 2022/11/24 16:19:44 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:45 pgnode01 vip-manager[24344]: 2022/11/24 16:19:45 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:46 pgnode01 vip-manager[24344]: 2022/11/24 16:19:46 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:47 pgnode01 vip-manager[24344]: 2022/11/24 16:19:47 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:48 pgnode01 vip-manager[24344]: 2022/11/24 16:19:48 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:49 pgnode01 vip-manager[24344]: 2022/11/24 16:19:49 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:50 pgnode01 vip-manager[24344]: 2022/11/24 16:19:50 etcd error: 100: Key not found (/service) [7]
Nov 24 16:19:51 pgnode01 vip-manager[24344]: 2022/11/24 16:19:51 etcd error: 100: Key not found (/service) [7]

Answered by vitabaks

Nov 24, 2022

Hi @prensgold

Why do I see "vip-manager" in the log?

Did you deploy a TYPE B cluster first and then try to deploy a TYPE A schema?

View full answer

vitabaks · 2022-11-24T13:58:41Z

vitabaks
Nov 24, 2022
Maintainer

Hi @prensgold

Why do I see "vip-manager" in the log?

Did you deploy a TYPE B cluster first and then try to deploy a TYPE A schema?

2 replies

vitabaks Nov 24, 2022
Maintainer

In order to convert the scheme B to A, please set the variable vip_manager_disable: true in the playbook balancers.yml

prensgold Nov 25, 2022
Author

I read the whole your article, but I missed it.should I write a different error under this title?

vitabaks · 2022-11-25T09:36:31Z

vitabaks
Nov 25, 2022
Maintainer

etcd error: 100: Key not found (/service)

check the status of the etc cluster, and make sure that the namespace "service" exists

example:

etcdctl cluster-health

member 12115c1f095f1a2e is healthy: got healthy result from http://172.30.58.34:2379
member 24a7d80ac60e30d2 is healthy: got healthy result from http://172.30.58.33:2379
member 2e19309d9a04f569 is healthy: got healthy result from http://172.30.58.31:2379
member 314767b36738c4d8 is healthy: got healthy result from http://172.30.58.35:2379
member 605058917e877cef is healthy: got healthy result from http://172.30.58.30:2379
cluster is healthy

cluster is healthy

etcdctl ls

etcdctl ls
/service

etcdctl ls /service
/service/postgres-cluster

namespace "service" is exists

2 replies

prensgold Nov 25, 2022
Author

we've pass etcd error:100 with your advice.Setup's finish with succes.But etcdctl health status giving as below error.
you think although the installation does not give an error, is there a part that we skipped?

[root@pgnode01 ~]# ETCDCTL_API=3 etcdctl endpoint health --cluster --write-out=table --endpoints=http://127.0.0.1:2379 
{"level":"warn","ts":"2022-11-25T15:50:06.651+0300","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00035ea80/192.168.154.79:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.1 403 Tunnel or SSL Forbidden\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 666\\\\r\\\\nCache-Control: no-store\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html\\\\r\\\\nDate: Fri, 25 Nov 2022 12:50:03 GMT\\\\r\\\\nVia: 1.1 servercluster\\\\r\\\\n\\\\r\\\\n<HEAD><TITLE>Tunnel or SSL Forbidden</TITLE></HEAD>\\\\n<BODY BGCOLOR=\\\\\\\"white\\\\\\\" FGCOLOR=\\\\\\\"black\\\\\\\"><H1>Tunnel or SSL Forbidden</H1><HR>\\\\n<FONT FACE=\\\\\\\"Helvetica,Arial\\\\\\\"><B>\\\\nDescription: 2379 is not an allowed port for Tunnel or SSL connections</B></FONT>\\\\n<HR>\\\\n<!-- default \\\\\\\"Tunnel or SSL Forbidden\\\\\\\" response (403) -->\\\\n</BODY>\\\\n                                                                                                                                                                                                                                                                                                                                                                \\\\x00\\\"\""}
{"level":"warn","ts":"2022-11-25T15:50:06.651+0300","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0003e41c0/192.168.154.77:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.1 403 Tunnel or SSL Forbidden\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 666\\\\r\\\\nCache-Control: no-store\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html\\\\r\\\\nDate: Fri, 25 Nov 2022 12:50:04 GMT\\\\r\\\\nVia: 1.1 servercluster\\\\r\\\\n\\\\r\\\\n<HEAD><TITLE>Tunnel or SSL Forbidden</TITLE></HEAD>\\\\n<BODY BGCOLOR=\\\\\\\"white\\\\\\\" FGCOLOR=\\\\\\\"black\\\\\\\"><H1>Tunnel or SSL Forbidden</H1><HR>\\\\n<FONT FACE=\\\\\\\"Helvetica,Arial\\\\\\\"><B>\\\\nDescription: 2379 is not an allowed port for Tunnel or SSL connections</B></FONT>\\\\n<HR>\\\\n<!-- default \\\\\\\"Tunnel or SSL Forbidden\\\\\\\" response (403) -->\\\\n</BODY>\\\\n                                                                                                                                                                                                                                                                                                                                                                \\\\x00\\\"\""}
{"level":"warn","ts":"2022-11-25T15:50:06.651+0300","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001aa000/192.168.154.80:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.1 403 Tunnel or SSL Forbidden\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 666\\\\r\\\\nCache-Control: no-store\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html\\\\r\\\\nDate: Fri, 25 Nov 2022 12:50:04 GMT\\\\r\\\\nVia: 1.1 servercluster\\\\r\\\\n\\\\r\\\\n<HEAD><TITLE>Tunnel or SSL Forbidden</TITLE></HEAD>\\\\n<BODY BGCOLOR=\\\\\\\"white\\\\\\\" FGCOLOR=\\\\\\\"black\\\\\\\"><H1>Tunnel or SSL Forbidden</H1><HR>\\\\n<FONT FACE=\\\\\\\"Helvetica,Arial\\\\\\\"><B>\\\\nDescription: 2379 is not an allowed port for Tunnel or SSL connections</B></FONT>\\\\n<HR>\\\\n<!-- default \\\\\\\"Tunnel or SSL Forbidden\\\\\\\" response (403) -->\\\\n</BODY>\\\\n                                                                                                                                                                                                                                                                                                                                                                \\\\x00\\\"\""}
+----------------------------+--------+--------------+---------------------------+
|          ENDPOINT          | HEALTH |     TOOK     |           ERROR           |
+----------------------------+--------+--------------+---------------------------+
| http://192.168.154.79:2379 |  false | 5.001876717s | context deadline exceeded |
| http://192.168.154.77:2379 |  false | 5.001889428s | context deadline exceeded |
| http://192.168.154.80:2379 |  false | 5.001857987s | context deadline exceeded |
+----------------------------+--------+--------------+---------------------------+
Error: unhealthy cluster
[root@pgnode01 ~]#

vitabaks Nov 25, 2022
Maintainer

try command

etcdctl cluster-health

prensgold · 2022-11-25T15:38:30Z

prensgold
Nov 25, 2022
Author

actually i've tried , but it's not working on my server , it's giving as "unknown command "cluster-health""

[root@pgnode01 ~]# etcdctl member list -w table
+------------------+---------+----------+----------------------------+----------------------------+------------+
|        ID        | STATUS  |   NAME   |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+----------+----------------------------+----------------------------+------------+
| 6906eb46f54c1a0e | started | pgnode01 | http://192.168.154.77:2380 | http://192.168.154.77:2379 |      false |
| 85b2bda90fff69e8 | started | pgnode03 | http://192.168.154.80:2380 | http://192.168.154.80:2379 |      false |
| f6c4958c5984cb9d | started | pgnode02 | http://192.168.154.79:2380 | http://192.168.154.79:2379 |      false |
+------------------+---------+----------+----------------------------+----------------------------+------------+
[root@pgnode01 ~]# etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.599511ms
[root@pgnode01 ~]# etcdctl version
etcdctl version: 3.5.4
API version: 3.5
[root@pgnode01 ~]# etcdctl cluster-health
Error: unknown command "cluster-health" for "etcdctl"
Run 'etcdctl --help' for usage.
Error: unknown command "cluster-health" for "etcdctl"
[root@pgnode01 ~]# ETCDCTL_ENDPOINTS=$( /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','")  etcdctl endpoint health
{"level":"warn","ts":"2022-11-25T18:37:01.471+0300","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001aa000/192.168.154.77:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.1 403 Tunnel or SSL Forbidden\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 666\\\\r\\\\nCache-Control: no-store\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html\\\\r\\\\nDate: Fri, 25 Nov 2022 15:36:58 GMT\\\\r\\\\nVia: 1.1 servercluster\\\\r\\\\n\\\\r\\\\n<HEAD><TITLE>Tunnel or SSL Forbidden</TITLE></HEAD>\\\\n<BODY BGCOLOR=\\\\\\\"white\\\\\\\" FGCOLOR=\\\\\\\"black\\\\\\\"><H1>Tunnel or SSL Forbidden</H1><HR>\\\\n<FONT FACE=\\\\\\\"Helvetica,Arial\\\\\\\"><B>\\\\nDescription: 2379 is not an allowed port for Tunnel or SSL connections</B></FONT>\\\\n<HR>\\\\n<!-- default \\\\\\\"Tunnel or SSL Forbidden\\\\\\\" response (403) -->\\\\n</BODY>\\\\n                                                                                                                                                                                                                                                                                                                                                                \\\\x00\\\"\""}
{"level":"warn","ts":"2022-11-25T18:37:01.471+0300","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004dc000/192.168.154.80:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.1 403 Tunnel or SSL Forbidden\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 666\\\\r\\\\nCache-Control: no-store\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html\\\\r\\\\nDate: Fri, 25 Nov 2022 15:36:58 GMT\\\\r\\\\nVia: 1.1 servercluster\\\\r\\\\n\\\\r\\\\n<HEAD><TITLE>Tunnel or SSL Forbidden</TITLE></HEAD>\\\\n<BODY BGCOLOR=\\\\\\\"white\\\\\\\" FGCOLOR=\\\\\\\"black\\\\\\\"><H1>Tunnel or SSL Forbidden</H1><HR>\\\\n<FONT FACE=\\\\\\\"Helvetica,Arial\\\\\\\"><B>\\\\nDescription: 2379 is not an allowed port for Tunnel or SSL connections</B></FONT>\\\\n<HR>\\\\n<!-- default \\\\\\\"Tunnel or SSL Forbidden\\\\\\\" response (403) -->\\\\n</BODY>\\\\n                                                                                                                                                                                                                                                                                                                                                                \\\\x00\\\"\""}
{"level":"warn","ts":"2022-11-25T18:37:01.471+0300","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000364a80/192.168.154.79:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.1 403 Tunnel or SSL Forbidden\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 666\\\\r\\\\nCache-Control: no-store\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html\\\\r\\\\nDate: Fri, 25 Nov 2022 15:36:59 GMT\\\\r\\\\nVia: 1.1 servercluster\\\\r\\\\n\\\\r\\\\n<HEAD><TITLE>Tunnel or SSL Forbidden</TITLE></HEAD>\\\\n<BODY BGCOLOR=\\\\\\\"white\\\\\\\" FGCOLOR=\\\\\\\"black\\\\\\\"><H1>Tunnel or SSL Forbidden</H1><HR>\\\\n<FONT FACE=\\\\\\\"Helvetica,Arial\\\\\\\"><B>\\\\nDescription: 2379 is not an allowed port for Tunnel or SSL connections</B></FONT>\\\\n<HR>\\\\n<!-- default \\\\\\\"Tunnel or SSL Forbidden\\\\\\\" response (403) -->\\\\n</BODY>\\\\n                                                                                                                                                                                                                                                                                                                                                                \\\\x00\\\"\""}
http://192.168.154.77:2379 is unhealthy: failed to commit proposal: context deadline exceeded
http://192.168.154.80:2379 is unhealthy: failed to commit proposal: context deadline exceeded
http://192.168.154.79:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
[root@pgnode01 ~]#

6 replies

vitabaks Nov 27, 2022
Maintainer

I will work on compatibility with etcd v3.4 and 3.5 soon.

prensgold Nov 28, 2022
Author

i've tried with 3.3.27 version that your tested version.
It's work , but i couldn't exec cluster-health command.i didn't understand why don't it work.
i've learned cluster status with below command.

[root@pgnode03 ~]# export ETCDCTL_API=3
[root@pgnode03 ~]# HOST_1=192.168.154.77
[root@pgnode03 ~]# HOST_2=192.168.154.79
[root@pgnode03 ~]# HOST_3=192.168.154.80
[root@pgnode03 ~]# ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
[root@pgnode03 ~]# etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 member list
6906eb46f54c1a0e, started, pgnode01, http://192.168.154.77:2380, http://192.168.154.77:2379
85b2bda90fff69e8, started, pgnode03, http://192.168.154.80:2380, http://192.168.154.80:2379
f6c4958c5984cb9d, started, pgnode02, http://192.168.154.79:2380, http://192.168.154.79:2379
[root@pgnode03 ~]# etcdctl --endpoints=$ENDPOINTS endpoint health
192.168.154.79:2379 is healthy: successfully committed proposal: took = 2.303161ms
192.168.154.77:2379 is healthy: successfully committed proposal: took = 2.320798ms
192.168.154.80:2379 is healthy: successfully committed proposal: took = 2.344208ms
[root@pgnode03 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status
+---------------------+------------------+---------+---------+-----------+-----------+------------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+---------------------+------------------+---------+---------+-----------+-----------+------------+
| 192.168.154.77:2379 | 6906eb46f54c1a0e |  3.3.27 |   20 kB |     false |         2 |       2124 |
| 192.168.154.79:2379 | f6c4958c5984cb9d |  3.3.27 |   20 kB |      true |         2 |       2124 |
| 192.168.154.80:2379 | 85b2bda90fff69e8 |  3.3.27 |   20 kB |     false |         2 |       2124 |
+---------------------+------------------+---------+---------+-----------+-----------+------------+
[root@pgnode03 ~]# etcdctl version
etcdctl version: 3.3.27
API version: 3.3
[root@pgnode03 ~]# etcdctl cluster-health
Error: unknown command "cluster-health" for "etcdctl"
Run 'etcdctl --help' for usage.
Error: unknown command "cluster-health" for "etcdctl"
[root@pgnode03 ~]#

vitabaks Nov 28, 2022
Maintainer

at least now we can see that the etcd cluster nodes are healthy.

vitabaks Nov 28, 2022
Maintainer

but i couldn't exec cluster-health command.i didn't understand why don't it work.

perhaps because for some reason you are trying to use ETCDCTL_API=3

prensgold Nov 29, 2022
Author

when if i don't set export ETCDCTL_API=3 , the command giving error as below.
i guess etcd 3 is not stable version for Redhat.

[root@pgnode01 ~]# etcdctl cluster-health
failed to check the health of member 6906eb46f54c1a0e on http://192.168.154.77:2379: invalid character '<' looking for beginning of value
member 6906eb46f54c1a0e is unreachable: [http://192.168.154.77:2379] are all unreachable
failed to check the health of member 85b2bda90fff69e8 on http://192.168.154.80:2379: invalid character '<' looking for beginning of value
member 85b2bda90fff69e8 is unreachable: [http://192.168.154.80:2379] are all unreachable
failed to check the health of member f6c4958c5984cb9d on http://192.168.154.79:2379: invalid character '<' looking for beginning of value
member f6c4958c5984cb9d is unreachable: [http://192.168.154.79:2379] are all unreachable
cluster is unavailable

Uh oh!

etcd error: 100: Key not found (/service) #222

Uh oh!

Uh oh!

prensgold Nov 24, 2022

Replies: 3 comments · 10 replies

Uh oh!

vitabaks Nov 24, 2022 Maintainer

Uh oh!

vitabaks Nov 24, 2022 Maintainer

Uh oh!

prensgold Nov 25, 2022 Author

Uh oh!

vitabaks Nov 25, 2022 Maintainer

Uh oh!

Uh oh!

prensgold Nov 25, 2022 Author

Uh oh!

vitabaks Nov 25, 2022 Maintainer

Uh oh!

Uh oh!

prensgold Nov 25, 2022 Author

Uh oh!

Uh oh!

vitabaks Nov 27, 2022 Maintainer

Uh oh!

Uh oh!

prensgold Nov 28, 2022 Author

Uh oh!

vitabaks Nov 28, 2022 Maintainer

Uh oh!

vitabaks Nov 28, 2022 Maintainer

Uh oh!

prensgold Nov 29, 2022 Author

prensgold
Nov 24, 2022

Replies: 3 comments 10 replies

vitabaks
Nov 24, 2022
Maintainer

vitabaks Nov 24, 2022
Maintainer

prensgold Nov 25, 2022
Author

vitabaks
Nov 25, 2022
Maintainer

prensgold Nov 25, 2022
Author

vitabaks Nov 25, 2022
Maintainer

prensgold
Nov 25, 2022
Author

vitabaks Nov 27, 2022
Maintainer

prensgold Nov 28, 2022
Author

vitabaks Nov 28, 2022
Maintainer

vitabaks Nov 28, 2022
Maintainer

prensgold Nov 29, 2022
Author