High memory consumption on nodes and possible memory leak

[k-mitevski]

We have an issue with a couple of nodes running on K8s, that look like they have memory leaks. Two nodes with a memory limit of 16Gb, and one with 120Gb eventually hit a limit and get OOMkilled. Increasing the limit just delays the OOM hit and provides more time before the restart happens.

Can this be checked out? We want to know if this may be a misconfiguration on our side and if we need to modify a flag to fix it, or if this is internal.

For example:

System information

Erigon version: erigon-2.60.10-0.8.1

OS & Version: Kubernetes v1.30.6

Erigon Command (with flags/config):

      command:
        - erigon
        - '--datadir=/home/node/data'
        - '--db.pagesize=4096'
        - '--ethash.dagdir=/home/node/data/dag'
        - '--db.size.limit=8TB'
        - '--chain=base-mainnet'
        - '--networkid=8453'
        - '--authrpc.jwtsecret=/home/node/jwt-secret.txt'
        - '--authrpc.vhosts=*'
        - '--http'
        - '--http.addr=0.0.0.0'
        - '--http.corsdomain=null'
        - '--http.vhosts=*'
        - '--http.api=eth,erigon,net,web3,debug,trace'
        - '--ws'
        - '--rollup.sequencerhttp=https://mainnet-sequencer.base.org'
        - '--rollup.historicalrpc='
        - '--rollup.historicalrpctimeout=5s'
        - '--maxpeers=0'
        - '--nodiscover'
        - '--identity=xxx'
        - '--rollup.disabletxpoolgossip=true'
        - '--metrics'
        - '--metrics.addr=0.0.0.0'
        - '--healthcheck'
        - '--private.api.addr=0.0.0.0:9090'
        - '--private.api.ratelimit=31872'
        - '--http.timeouts.read=300s'

[mininny]

Thanks for raising the issue @k-mitevski !

I don't see anything out of the ordinary with the run flags, and we have base nodes running well on our end. There might be something specific here, so to exactly debug this issue, I think we'd need to trace the memory pattern and heap profile when the OOM occurred.

Can you please do the following?

• build op-erigon from this custom branch: feature/mininny/experiment-dbg-heapprofile(which is a solution from upstream erigon Sporadic and seemingly random crashes on different setups but same issue erigontech/erigon#13732) and run make erigon
• run op-erigon with 2 env vars set, e.g.SAVE_HEAP_PROFILE=true && HEAP_PROFILE_FILE_PATH=<yourdir>/erigon-mem.prof && ./build/bin/erigon --datadir ...
  • This will check every 30 sec to see if the node is close to OOM threshold, and save a heap profile at HEAP_PROFILE_FILE_PATH
• If you run op-erigon with the above env vars, you should see logs like [Experiment] heap profile threshold check after about 30 seconds.

Doing the above will give us the heap profile to debug with when the OOM happens. Please attach the resulting heap profile file afterwards.

Greatly appreciate it! 🙏

[k-mitevski]

Hello @mininny, thank you for the info. We tried what you suggested, building an image from the branch and running it along with the specified environment variables. Even though the startup logs showed that the environment variables were set and we got the [Experiment] log lines visible, the .prof file was never created.
The node ran for ~5 hours and was fully synced. We also tried a restart in case the profiling file is created on termination but this also was not the case.

Can you tell us the cause or how to fix/set it up so the heap profile file is generated?

[mininny]

@k-mitevski I've pushed a new commit to the above branch. Please build a new version of erigon and run it like

SAVE_HEAP_PROFILE=true HEAP_PROFILE_FILE_PATH=erigon-mem.prof ./build/bin/erigon --chain=op-mainnet

Also, in your previous runs, did the erigon node get OOM killed? Or did you just want to capture the .prof file while the node is alive
Anyhow, the new commit will allow you to terminate the node and save the memprof on termination.

[k-mitevski]

@mininny I did a brief test to see and now the .prof file is generated, so I'll leave the node running for a couple of hours.

In the previous runs, we didn't encounter the OOM, mostly because this node has a higher memory limit(~130 GB), so it will take longer to reach the OOMkill. I assume even without it you can analyze if there is a memory leak given the steady increase in memory consumption we notice.

[mininny]

mostly because this node has a higher memory limit(~130 GB), so it will take longer to reach the OOMkill. I assume even without it you can analyze if there is a memory leak given the steady increase in memory consumption we notice.

@k-mitevski Yes that makes sense. Please let us know after running it!

[k-mitevski]

@mininny Here is the file with gathered data from the last ~24h along with the memory usage. The drop is when I shutdown the node to grab the file.
I had to upload it to an external host since GH doesn't allow this filetype to be uploaded. Please check if all is good.
https://limewire.com/d/2ae88cb7-8950-41bc-81c0-98f9f1f0d9ec#qb-JI1KS0lLezT5y8w8Gd0wUSEWAcXNL2nsNuti5uLg

[mininny]

@k-mitevski Thank you! I'll take a look at the possible leaks in the code.

[mininny]

@k-mitevski Sorry for the delayed response.
I have looked at the memory profile you provided, and the memory increase led to the txpool part of the code in erigon. However, since op-erigon doesn't need to maintain txpool at all, we can try disabling the txpool from op-erigon by setting this in your run flag: --txpool.disable=true

Also, I have added another commit to the previous branch to get a better view of the whole memory stats.

Can you try re-building op-erigon with above commit, and try running it with txpool.disble=true?
Once you do that, pleaes provide the erigon-mem.prof and erigon-mem.prof.json files

[k-mitevski]

Hello @mininny, I've rerun the test by building the new image from the branch and setting the txpool.disable=true flag.
The memory usage pattern seems the same: slow linear growth and then ramping up. The data gathered is from ~20h.

https://limewire.com/d/4920ef1b-052e-4a06-940b-4f751bb03212#a6N4fl3kqVG-KaK5dIr9PTkGjOAQQL6oio73m77M1Qw

[mininny]

@k-mitevski Hey, I've looked at the memory stats and heap profile that you've sent over, and the active memory usage reported from these are less than a few gigabytes.

I know that this issue has been opened for a long time and I apologize for not getting a definite answer quickly.

Given the low memory usage from the stats you sent over, I was wondering if other processes were using the memory. Are you able to filter out erigon from that memory usage graph in grafana by process?