After first pass, ram usage increases unexpectedly. Starts swapping and eventually crashes #1001
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
I'm using madmax 1.1.8-ecec17d.
I have a supermicro X9DRI motherboard, with 512 GB of ddr3.
Prior to starting plotting, less than 1% of the system's memory is in use, even before I stop superfluous services:
I am attempting to run madmax in parallel, per numa node.
I create tmpfs partitions:
and I begin to plot:
I also set swappiness to zero now:
sysctl vm.swappiness=0The first pass everything seems to go fine. No swap storage gets used:
The parallel job is similarly unremarkable. However, I start to notice there seems to be memory in use that is unaccounted for. After the first pass, the free ram seems to not all be freed. Some is buffered, and eventually released as the 2nd pass starts to use more RAM. However, very quickly it starts swapping.
As it's copying the completed plots to the disk, and the tmpfs volumes are being emptied, the RAM usage does not decrease. Instead, after one of the copy jobs finishes, the RAM usage hits a low of about 100 GB, and then actually increases until about half my RAM is in use when the 2nd transfer completes:
Eventually, if I allow it to keep going, one of the jobs will be killed for lack of memory when the swap becomes exhausted. Whether or not the process is killed or terminates normally, a large part of my RAM is no longer available, as per above.
The only way I have found to free it is to reboot.
I use zfs on the destination disk, but I limit arc cache to 4 GB. Also, I have 24 GB of swap. Once again, none is used the first pass, but increasingly it is exhausted until one of the processes is killed, and the other will continue normally.
In addition, the overhead incurred by swapping actually makes parallel jobs slower.
UPDATE:
I tried running with no swap, and it nearly worked.
I could have probably freed up another 3 GB by using a lighter distro, but I estimate, there was still ~7.6 GB of tmpfs to be written. I calculated that based on 8.7 GB shown free on tmpfs1, and the past observation that it bottoms out at ~1.1 GB. Therefore 8.7-1.1=7.6:
Shortly thereafter, the second job crashed at the same point. Both file transfers had completed, so something else is causing the additional memory usage.
Obviously, "MemAvailable", which I got from /prox/meminfo, was a bit optimistic. That pane and the one with /tmpfs/ space to the left are updated every second. Only a few seconds elapsed before I was able to get the screengrab.
I don't have a problem with it swapping some at ~P2/T2. That happens when I run one job with 256 GB of RAM, and it only adds a minute or so onto plot times. Unfortunately, if I let it run, it will use all 24 GB of my swap, and I'm sure it would grow much larger.
tl;dr
When plotting parallel jobs in tmpfs with 512 GB of RAM, every pass needs increasing amounts of RAM, until one job crashes.
The text was updated successfully, but these errors were encountered: