Replies: 3 comments 13 replies
-
|
Which version of Kvrocks are you using? |
Beta Was this translation helpful? Give feedback.
-
|
v2.10.0 |
Beta Was this translation helpful? Give feedback.
-
|
There's several known bugs in 2.10.0. Could you try to upgrade to at least 2.10.1 (or 2.11.1, if you want) and see if it still exists? |
Beta Was this translation helpful? Give feedback.
-
|
OK，I will try it. |
Beta Was this translation helpful? Give feedback.
-
|
I am using kvrocks version 2.11.1 with the following configurations: rocksdb.block_cache_size 1000, rocksdb.max_write_buffer_number 16, and rocksdb.write_buffer_size 64. After running tests for a period, I found that the memory usage increased to 4.5 GB. According to my understanding, the memory usage should be around block_cache (1 GB) + memtable (1 GB) = 2 GB. Since RocksDB uses jemalloc, the resident memory is 1.4 GB. Why, then, does the memory usage rise to 4.5 GB? Detailed kvrocks Configuration: slowlog-max-len 128 small: 1041185624 2869758745 40679 2868831004 40666 297726959299 4220323 238617686 3382 165571637 2347 active: 1252102144 |
Beta Was this translation helpful? Give feedback.
-
|
Additionally, I observed through top that the RSS (Resident Set Size) is 4.5 GB. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
@twoholes Please refer to https://lists.apache.org/thread/xzn62z7xz3bvxwgsd9toh6wq2g2j8zng |
Beta Was this translation helpful? Give feedback.
-
|
yeah. getconf PAGESIZE 4096 |
Beta Was this translation helpful? Give feedback.
-
It should be |
Beta Was this translation helpful? Give feedback.
-
|
To clarify, is this environment variable |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
yes, I have tried to disable jemalloc, the phenomenon remains the same 。 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
here is my KVROCKS Configuration:
rocksdb.block_cache_size 26000
rocksdb.write_buffer_size 64
rocksdb.max_write_buffer_number 16
rocksdb.max_background_jobs 6
rocksdb.max_bytes_for_level_base 2147483647
rocksdb.use_direct_io_for_flush_and_compaction yes
rocksdb.block_size 4096
rocksdb.compression no
rocksdb.level0_file_num_compaction_trigger 8
rocksdb.min_write_buffer_number_to_merge 4
rocksdb.enable_pipelined_write yes
rocksdb.use_direct_reads no
rocksdb.stats_dump_period_sec 60
rocksdb.level0_slowdown_writes_trigger 40
rocksdb.level0_stop_writes_trigger 60
Problem Phenomenon:
The memory usage of KVROCKS rises to about 45G, triggering an OOM (Out Of Memory) condition, and the process is killed by the system.
the log of jemalloc statics：
2025/02/09-23:30:13.745060 140398374680320 [db/db_impl/db_impl.cc:1179] ------- Malloc STATS -------
2025/02/09-23:30:13.747330 140398374680320 [db/db_impl/db_impl.cc:1181] ___ Begin jemalloc statistics ___
Version: "0.0.0-0-g000000missing_version_try_git_fetch_tags"
Build-time option settings
config.cache_oblivious: true
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.opt_safety_checks: false
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.cache_oblivious: true
opt.confirm_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 192
opt.percpu_arena: "disabled"
opt.oversize_threshold: 8388608
opt.hpa: false
opt.hpa_slab_max_alloc: 65536
opt.hpa_hugification_threshold: 1992294
opt.hpa_hugify_delay_ms: 10000
opt.hpa_min_purge_interval_ms: 5000
opt.hpa_dirty_mult: "0.25"
opt.hpa_sec_nshards: 4
opt.hpa_sec_max_alloc: 32768
opt.hpa_sec_max_bytes: 262144
opt.hpa_sec_bytes_after_flush: 131072
opt.hpa_sec_batch_fill_extra: 0
opt.metadata_thp: "disabled"
opt.mutex_max_spin: 600
opt.background_thread: false (background_thread: false)
opt.dirty_decay_ms: 0 (arenas.dirty_decay_ms: 0)
opt.muzzy_decay_ms: 0 (arenas.muzzy_decay_ms: 0)
opt.lg_extent_max_active_fit: 6
opt.muzzy_decay_ms: 0 (arenas.muzzy_decay_ms: 0)
opt.lg_extent_max_active_fit: 6
opt.junk: "false"
opt.zero: false
opt.experimental_infallible_new: false
opt.tcache: true
opt.tcache_max: 32768
opt.tcache_nslots_small_min: 20
opt.tcache_nslots_small_max: 200
opt.tcache_nslots_large: 20
opt.lg_tcache_nslots_mul: 1
opt.tcache_gc_incr_bytes: 65536
opt.tcache_gc_delay_bytes: 0
opt.lg_tcache_flush_small_div: 1
opt.lg_tcache_flush_large_div: 1
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_interval: -1
opt.stats_interval_opts: ""
opt.zero_realloc: "free"
Arenas: 193
Quantum size: 16
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 36
Number of thread-cache bin size classes: 41
Number of large size classes: 196
Allocated: 27801994432, active: 30404927488, metadata: 2245246960 (n_thp 0), resident: 32649306112, mapped: 32904732672, retained: 77356179456
Count of realloc(non-null-ptr, 0) calls: 0
Background threads: 0, num_runs: 0, run_interval: 0 ns
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
background_thread 35702 0 0 0 0 0 34 0 0 0 0 0
max_per_bg_thd 0 0 0 0 0 0 0 0 0 0 0 0
ctl 2489532827 3485 0 0 0 0 1 0 0 0 0 0
prof 0 0 0 0 0 0 0 0 0 0 0 0
prof_thds_data 0 0 0 0 0 0 0 0 0 0 0 0
prof_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_alloc 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_stats 0 0 0 0 0 0 0 0 0 0 0 0
Merged arenas stats:
assigned threads: 33
uptime: 714329953265331
dss allocation precedence: "N/A"
decaying: time npages sweeps madvises purged
dirty: N/A 0 2996702376 3001657810 18056038556
muzzy: N/A 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 24982279360 17865668758 25010 17839439738 24973 1541062626296 2157356 5114564180 7159 3336905767 4671
large: 2819715072 544703684 762 544623769 762 3145314414 4403 544703684 762 89169400 124
total: 27801994432 18410372442 25772 18384063507 25736 1544207940710 2161760 5659267864 7922 3426075167 4796
active: 30404927488
mapped: 32904732672
retained: 77356179456
base: 2244235248
internal: 1011712
metadata_thp: 0
tcache_bytes: 3408448
tcache_stashed_bytes: 0
resident: 32649306112
abandoned_vm: 0
extent_avail: 8445586
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 403596 0 0 0 0 0 34 0 0 0 0 0
extent_avail 3156399582 4418 7 0 3440 0 25849510 36 999997 1 999997 1
extents_dirty 9012355310 12616 564 0 412531 0 68854529 96 9999975 13 999999 2
extents_muzzy 403596 0 0 0 0 0 34 0 0 0 0 0
extents_retained 6009323314 8412 3021 0 843217 1 65468945 91 64999813 90 999999 1
decay_dirty 6024835308 8434 75 0 56304 0 61215713 85 999997 1 999997 1
decay_muzzy 403596 0 0 0 0 0 34 0 0 0 0 0
base 15094958 21 0 0 0 0 2712 0 0 0 0 0
tcache_list 403655 0 0 0 0 0 87 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0
My Questions is:
From the jemalloc statistics, I see that resident: 32G and cache size = 26G. Why does the memory usage of the process show RSS = 45G? This process is a slave and only handles replicate incrementBatchLoop traffic. If it is RocksDB that is occupying the memory, why do the jemalloc statistics only show 32G?
Beta Was this translation helpful? Give feedback.
All reactions