Possible memory leak - 8GB of RAM is not enough for gemini-3h-with-nemesis

[enaydanov]

gemini process was oom-killed in https://argus.scylladb.com/test/2873b203-404e-492c-957e-8cf49830f0f5/runs?additionalRuns[]=b54477a2-75ef-4244-91bb-21e2f3cd7556 :

2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: node_exporter invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_
adj=0
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: CPU: 0 PID: 4210 Comm: node_exporter Not tainted 6.5.0-1018-aws scylladb/scylla-cluster-tests#18~22.04.1-Ubuntu
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Hardware name: Amazon EC2 c6i.xlarge/, BIOS 1.0 10/16/2017
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Call Trace:
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  <TASK>
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  dump_stack_lvl+0x48/0x70
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  dump_stack+0x10/0x20
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  dump_header+0x50/0x270
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  oom_kill_process+0x10c/0x1b0
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  out_of_memory+0x109/0x350
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  __alloc_pages_may_oom+0x114/0x1e0
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  __alloc_pages_slowpath.constprop.0+0x4b1/0x9a0
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  __alloc_pages+0x313/0x340
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  alloc_pages+0x95/0x1a0
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  folio_alloc+0x1b/0x60
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  filemap_alloc_folio+0x31/0x40
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  __filemap_get_folio+0xf8/0x260
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  filemap_fault+0x4d1/0x800
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  __do_fault+0x36/0x150
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  do_read_fault+0xfc/0x170
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  do_fault+0xf3/0x170
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  handle_pte_fault+0x7b/0x190
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  __handle_mm_fault+0x3c8/0x730
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  handle_mm_fault+0xe9/0x360
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  do_user_addr_fault+0x227/0x680
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  exc_page_fault+0x84/0x190
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  asm_exc_page_fault+0x27/0x30
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: RIP: 0033:0x459890
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Code: Unable to access opcode bytes at 0x459866.
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: RSP: 002b:000000c00000f218 EFLAGS: 00010202
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: RAX: 000000000001726a RBX: 0000000000000000 RCX: 00000000010bf360
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: RDX: 00000000000001c4 RSI: 0000000000003d2d RDI: 00000000000001c5
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: RBP: 000000c00000f228 R08: 0000000000f4e320 R09: 0000000000f4f148
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: R10: 000000c00034a820 R11: 0000000000000000 R12: 000000c000108408
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: R13: 0000000001110fe0 R14: 000000c000006000 R15: 000000000000000a
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel:  </TASK>
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Mem-Info:
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: active_anon:1889266 inactive_anon:25485 isolated_anon:0
 active_file:4 inactive_file:0 isolated_file:0
 unevictable:8683 dirty:0 writeback:0
 slab_reclaimable:6565 slab_unreclaimable:13927
 mapped:2241 shmem:247 pagetables:4619
 sec_pagetables:0 bounce:0
 kernel_misc_reclaimable:0
 free:25041 free_pcp:52 free_cma:0
B isolated(anon):0kB isolated(file):0kB mapped:8964kB dirty:0kB writeback:0kB shmem:988kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 550912kB writeback_tmp:0kB kernel_stack:3360kB pagetables:18476kB sec_pageta
bles:0kB all_unreclaimable? no
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Node 0 DMA free:14336kB boost:0kB min:128kB low:160kB high:192kB reserved_highatomic:0KB active_anon:0kB 
inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: lowmem_reserve[]: 0 2984 7760 7760 7760
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Node 0 DMA32 free:44876kB boost:0kB min:25940kB low:32424kB high:38908kB reserved_highatomic:0KB active_a
non:3012160kB inactive_anon:16kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129252kB managed:3063716kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: lowmem_reserve[]: 0 0 4775 4775 4775
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Node 0 Normal free:40952kB boost:0kB min:41508kB low:51884kB high:62260kB reserved_highatomic:0KB active_
anon:4544904kB inactive_anon:101924kB active_file:800kB inactive_file:0kB unevictable:34732kB writepending:0kB present:5095424kB managed:4898636kB mlocked:27616kB bounce:0kB free_pcp:208kB local_pcp:144kB free_cm
a:0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: lowmem_reserve[]: 0 0 0 0 0
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) =
 14336kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Node 0 DMA32: 754*4kB (UM) 649*8kB (UM) 529*16kB (UM) 357*32kB (UM) 186*64kB (UM) 42*128kB (UM) 0*256kB 0
*512kB 0*1024kB 0*2048kB 0*4096kB = 45376kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Node 0 Normal: 662*4kB (UME) 314*8kB (ME) 352*16kB (ME) 217*32kB (ME) 136*64kB (ME) 64*128kB (UME) 17*256
kB (E) 5*512kB (UME) 0*1024kB 0*2048kB 0*4096kB = 41544kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: 4332 total pagecache pages
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: 0 pages in swap cache
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Free swap  = 0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: Total swap = 0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: 2060167 pages RAM
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: 0 pages HighMem/MovableOnly
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: 65739 pages reserved
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1  !WARNING | kernel: 0 pages hwpoisoned
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: Tasks state (memory values in pages):
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    128]     0   128    16178     1047   106496        0          -250 systemd-journal
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    168]     0   168    72328     6848   110592        0         -1000 multipathd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    172]     0   172     2973     1111    65536        0         -1000 systemd-udevd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    351]   100   351     4065      992    69632        0             0 systemd-network
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    353]   101   353     6418     1848    86016        0             0 systemd-resolve
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    444]     0   444      704      480    49152        0             0 acpid
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    448]     0   448     1822      576    53248        0             0 cron
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    449]   102   449     2175      992    57344        0          -900 dbus-daemon
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    456]     0   456    20692      800    65536        0             0 irqbalance
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    457]     0   457     8271     3008   106496        0             0 networkd-dispat
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    459]     0   459   328886     2131   204800        0             0 amazon-ssm-agen
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    470]     0   470     3879      960    77824        0             0 systemd-logind
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    475]     0   475     1555      512    49152        0             0 agetty
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    478]     0   478     1544      480    49152        0             0 agetty
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    482]   114   482     4723      743    65536        0             0 chronyd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    516]     0   516    27525     2944   122880        0             0 unattended-upgr
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    517]     0   517    58868     1148    94208        0             0 polkitd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [    520]   114   520     2641      433    65536        0             0 chronyd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   1496]     0  1496    74000     1632   167936        0             0 packagekitd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   1542]     0  1542   385428     3511   274432        0          -900 snapd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   2800]     0  2800     3860     1248    73728        0         -1000 sshd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   3291]  1000  3291     4235     1120    73728        0             0 systemd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   3292]  1000  3292    42604     1496    98304        0             0 (sd-pam)
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   3397]     0  3397   105873     1781   163840        0             0 syslog-ng
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   4210]   998  4210   310475     1839   126976        0             0 node_exporter
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   5530]     0  5530    24364     2894    94208        0             0 dirmngr
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   5533]     0  5533    20454      524    57344        0             0 gpg-agent
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   7685]     0  7685   468247     3441   294912        0          -999 containerd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   7778]     0  7778   533762     8394   438272        0          -500 dockerd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   8264]     0  8264     4266     1344    69632        0             0 sshd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   8309]  1000  8309     4301     1152    69632        0             0 sshd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   8393]     0  8393     4267     1344    77824        0             0 sshd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   8438]  1000  8438     4302     1146    77824        0             0 sshd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   8457]     0  8457   309118      718   114688        0          -998 containerd-shim
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   8478]     0  8478     1100       32    53248        0             0 tail
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   8492]  1000  8492     2975      832    69632        0             0 sudo
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   8493]     0  8493   424983     2290   221184        0             0 docker
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: [   8510]     0  8510  2049685  1877425 15196160        0             0 gemini
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1     !INFO | kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=node_exporter.service,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-9211e23c8eb758d31b282cf584797f768dabe6b2825eee667953454456f12fee.scope,task=gemini,pid=8510,uid=0
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1      !ERR | kernel: Out of memory: Killed process 8510 (gemini) total-vm:8198740kB, anon-rss:7509700kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:14840kB oom_score_adj:0
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1   !NOTICE | systemd[1]: docker-9211e23c8eb758d31b282cf584797f768dabe6b2825eee667953454456f12fee.scope: A process of this unit has been killed by the OOM killer.

c6i.xlarge instance type (default for a loader in SCT) has 4 CPUs and 8GB of RAM. We need to switch to an instance type with more RAM or add a swap for gemini-3h-with-nemesis test.

[enaydanov]

Gemini Version: 1.8.6
Gemini command: ./gemini -d --duration 3h --warmup 30m -c 50 -m mixed -f --non-interactive --cql-features normal --max-mutation-retries 10 --max-mutation-retries-backoff 500ms --async-objects-stabilization-attempts 5 --async-objects-stabilization-backoff 500ms --replication-strategy "{'class': 'NetworkTopologyStrategy', 'replication_factor': '3'}" --oracle-replication-strategy "{'class': 'NetworkTopologyStrategy', 'replication_factor': '1'}" --test-cluster=10.12.2.67,10.12.2.40,10.12.3.247 --outfile /gemini_result_d4ade4d7-17ab-45d7-af8a-47de6de07458.log --seed 80 --request-timeout 180s --connect-timeout 120s --oracle-cluster=10.12.2.243

[fruch]

@enaydanov please open an issue for gemini that it's leaking memory....

[roydahan]

Moved the issue to Gemini, need to investigate a memory leakage.

[fruch]

• - taking out the partition logic (lower the partitions sizes)
• - proving it the root cause of the leakage

[fruch]

@CodeLieutenant

we can extract the re-architect partitions into its own issue, with the details of the problem, and what are the goals we want achieve by re-architecting it ?

[fruch]

• - think of a way to calculate upfront the memory usage for the partition, so we won't run out of memory during the test