etcdserver: http2Server.HandleStreams failed to read frame

[heyitsanthony]

Boot master etcd on windows via ./etcd:

2017-05-13 17:45:31.479400 I | etcdmain: etcd Version: 3.2.0-rc.0+git
2017-05-13 17:45:31.479400 I | etcdmain: Git SHA:
2017-05-13 17:45:31.479400 I | etcdmain: Go Version: go1.8.1
2017-05-13 17:45:31.479400 I | etcdmain: Go OS/Arch: windows/amd64
2017-05-13 17:45:31.479400 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2017-05-13 17:45:31.479400 N | etcdmain: failed to detect default host (default host not supported on windows_amd64)
2017-05-13 17:45:31.480400 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2017-05-13 17:45:31.480400 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-05-13 17:45:31.482400 I | embed: listening for peers on http://localhost:2380
2017-05-13 17:45:31.483400 I | embed: listening for client requests on localhost:2379
2017-05-13 17:45:31.485400 W | snap: skipped unexpected non snapshot file db.lock
2017-05-13 17:45:31.485400 I | etcdserver: name = default
2017-05-13 17:45:31.485400 I | etcdserver: data dir = default.etcd
2017-05-13 17:45:31.485400 I | etcdserver: member dir = default.etcd\member
2017-05-13 17:45:31.485400 I | etcdserver: heartbeat = 100ms
2017-05-13 17:45:31.485400 I | etcdserver: election = 1000ms
2017-05-13 17:45:31.485400 I | etcdserver: snapshot count = 100000
2017-05-13 17:45:31.485400 I | etcdserver: advertise client URLs = http://localhost:2379
2017-05-13 17:45:31.486400 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 19
2017-05-13 17:45:31.486400 I | raft: 8e9e05c52164694d became follower at term 3
2017-05-13 17:45:31.486400 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 3, commit: 19, applied: 0, lastindex: 19, lastterm: 3]
2017-05-13 17:45:31.586400 W | auth: simple token is not cryptographically signed
2017-05-13 17:45:31.653400 I | etcdserver: starting server... [version: 3.2.0-rc.0+git, cluster version: to_be_decided]
2017-05-13 17:45:31.655400 E | etcdserver: cannot monitor file descriptor usage (cannot get FDUsage on windows)
2017-05-13 17:45:31.677400 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2017-05-13 17:45:31.677400 N | etcdserver/membership: set the initial cluster version to 3.2
2017-05-13 17:45:31.677400 I | etcdserver/api: enabled capabilities for version 3.2
2017-05-13 17:45:32.086400 I | raft: 8e9e05c52164694d is starting a new election at term 3
2017-05-13 17:45:32.086400 I | raft: 8e9e05c52164694d became candidate at term 4
2017-05-13 17:45:32.086400 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 4
2017-05-13 17:45:32.086400 I | raft: 8e9e05c52164694d became leader at term 4
2017-05-13 17:45:32.086400 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 4
2017-05-13 17:45:32.118400 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2017-05-13 17:45:32.118400 I | embed: ready to serve client requests
2017-05-13 17:45:32.118400 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

Try ETCDCTL_API=3 etcdctl put /abc def and ETCDCTL_API=3 etcdctl get /abc a few times. etcd logs an info line for each etcdctl call:

2017-05-13 17:45:35.404400 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:49377: wsarecv: An existing connection was forcibly closed by the remote host.
2017-05-13 17:45:35.978400 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:49380: wsarecv: An existing connection was forcibly closed by the remote host.
2017-05-13 17:45:36.432400 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:49383: wsarecv: An existing connection was forcibly closed by the remote host.
2017-05-13 17:45:36.724400 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:49386: wsarecv: An existing connection was forcibly closed by the remote host.
2017-05-13 17:45:37.019400 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:49389: wsarecv: An existing connection was forcibly closed by the remote host.
2017-05-13 17:45:37.227400 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:49392: wsarecv: An existing connection was forcibly closed by the remote host.

[aim4pole]

I am seeing a similar issue when I try to connect to etcd server, values are updated but I get this error.

2017-10-02 15:37:02.179455 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:61736: wsarecv: An existing connection was forcibly closed by the remote host.

[xiang90]

Can someone verify if this is still the case on windows?

[NStanleyZhang]

we also run into this issue on RedHat 7.2 with ETCD V3.2.5

systemctl status etcd

● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2017-12-21 14:02:22 CST; 16min ago
Main PID: 15143 (etcd)
CGroup: /system.slice/etcd.service
└─15143 /usr/bin/etcd --name=bvnf-demo-oame-0 --data-dir=/appdata/etcd --listen-client-urls=http://192.168.3.10:2379

Dec 21 14:15:50 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.11:55906: read: connection reset by peer
Dec 21 14:16:50 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.11:56966: read: connection reset by peer
Dec 21 14:17:10 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.10:34142: read: connection reset by peer
Dec 21 14:17:10 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.11:57348: read: connection reset by peer
Dec 21 14:17:40 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.10:34178: read: connection reset by peer
Dec 21 14:17:40 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.11:57892: read: connection reset by peer
Dec 21 14:18:50 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.10:34256: read: connection reset by peer
Dec 21 14:18:50 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.11:59120: read: connection reset by peer
Dec 21 14:19:00 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.10:34264: read: connection reset by peer
Dec 21 14:19:00 bvnf-demo-oame-0 etcd[15143]: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.10:2379->192.168.3.11:59322: read: connection reset by peer

[hzwyihao]

Anyone fix this? I see the same error when trying to establish a lot of connections to an etcd cluster, on centOS, and can be reproduced stably. etcd 3.2.7

[SaranBalaji90]

@hzwyihao are you seeing this on single node or multi node cluster and whether this happens only during bootstrap? I had similiar issue, documented them here #9165

[hzwyihao]

@SaranBalaji90 only on multi node, three for example, and single doesn't, during connecting. And it seems not to affect the connection.

[SaranBalaji90]

Is it only during bootstrap? Also if you take tcpdump it might help with debugging these issue.

[hzwyihao]

No, bootstrap is done already. Good idea, I'll try later.

[xiaoping378]

I met the same issue, found my etcdctl version is different from etcd.
make the same, issue is disappeared

[xiaoping378]

run on Linux server.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.