Investigate gradually increasing memory utilization w/ etcd 3.5.4 (potential leak?)

[coders1122]

We are running etcd as a pre-requisite to running milvus. While doing so, we see that our etcd (5-node) cluster with 2G ram, 0.5 cpu has memory utilization gradually climbing. We do see go GC kick in from time to time, but are worried about an eventual OOM if this trend continues.

We've profiled one of the etcd nodes and here's the top10 output. Appreciate any help in diagnosing if this is a leak or if it's WAI.

Here's an initial sample:

File: etcd
Type: inuse_space
Time: Jun 5, 2023 at 12:37pm (PDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top10
Showing nodes accounting for 56.62MB, 88.44% of 64.02MB total
Showing top 10 nodes out of 137
      flat  flat%   sum%        cum   cum%
    9.51MB 14.85% 14.85%     9.51MB 14.85%  runtime.allocm
    9.45MB 14.76% 29.62%    17.72MB 27.68%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.startPeer
    9.27MB 14.48% 44.10%     9.27MB 14.48%  bufio.NewReaderSize
    8.51MB 13.29% 57.39%     8.51MB 13.29%  google.golang.org/grpc/internal/transport.newBufWriter
    8.27MB 12.92% 70.31%     8.27MB 12.92%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.startStreamWriter
    4.63MB  7.23% 77.54%     4.63MB  7.23%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.newMsgAppV2Encoder
    2.31MB  3.61% 81.15%     2.31MB  3.61%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.newMsgAppV2Decoder
    2.01MB  3.14% 84.29%     2.01MB  3.14%  go.etcd.io/etcd/raft/v3/raftpb.(*Entry).Unmarshal
    1.50MB  2.34% 86.63%     1.50MB  2.34%  runtime.malg
    1.16MB  1.81% 88.44%     1.73MB  2.70%  go.etcd.io/etcd/server/v3/wal.newEncoder

After a few hours, here's a sample (with most of the incremental lift coming from UnMarshal:

File: etcd
Type: inuse_space
Time: Jun 5, 2023 at 11:36pm (PDT)
(pprof) top10
Showing nodes accounting for 82.27MB, 91.08% of 90.32MB total
Showing top 10 nodes out of 133
      flat  flat%   sum%        cum   cum%
   27.61MB 30.57% 30.57%    27.61MB 30.57%  go.etcd.io/etcd/raft/v3/raftpb.(*Entry).Unmarshal
    9.51MB 10.53% 41.10%     9.51MB 10.53%  runtime.allocm
    9.45MB 10.46% 51.56%    17.72MB 19.62%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.startPeer
    8.77MB  9.71% 61.27%     8.77MB  9.71%  bufio.NewReaderSize
    8.51MB  9.42% 70.69%     8.51MB  9.42%  google.golang.org/grpc/internal/transport.newBufWriter
    8.27MB  9.16% 79.85%     8.27MB  9.16%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.startStreamWriter
    4.63MB  5.12% 84.97%     4.63MB  5.12%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.newMsgAppV2Encoder
    2.31MB  2.56% 87.53%     2.31MB  2.56%  go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.newMsgAppV2Decoder
    1.70MB  1.89% 89.42%     1.70MB  1.89%  go.etcd.io/etcd/raft/v3.(*MemoryStorage).Append
    1.51MB  1.67% 91.08%     1.51MB  1.67%  golang.org/x/net/http2/hpack.(*headerFieldTable).addEntry

I'm also attaching the graphviz output which shows the (indirect) call stack for the Unmarshal call and makes it more obvious to see the incremental difference between samples:

[tjungblu]

Thanks for the report and sorry for the late reply. This is somewhat normal depending on your workload, I have no idea how/what milvus stores in etcd.

I only found this little piece of information: milvus-io/milvus#21995 (comment)
Which seems like a good candidate for such a big entity to marshal/unmarshal. Are you running a version before 2.2.2?