Blow away ramdisks

[Hardcore-fs]

Can we look at totally removing all the nonsense with 120G of ram to build a ramdisk, then trying to jimmy the files thru the os File system API.

start program..
take a look at available ram.
allocate a FBRB...
write as many file blocks as possible to this buffer, using a mult as index thru the data.

Any file data that WONT fit write out to secondary storage. (this then scales to the users ram without special situations.)
each file varies from 600mb ->1.3gb depending on the bucket size.

on read back, process the stuff in ram, then the files, DO NOT copy files into this ram buffer. from secondary storage for "faster" processing...

as the ram is freed up as each file is destroyed, write a new set of data to this buffer, when full write balance out to secondary storage.

In effect build your own FS in ram, without the need to store directory & file information, worry about mutex locking and all the other bit allocations for block storage calculations that go with a FS.

I think you will see a significant increase once you take the OS API out of it.

Plus it scales well for everyone who is NOT running servers with massive amounts of ram.

[madMAx43v3r]

That's a pretty good idea indeed, except you will have to specify RAM size as a parameter.

[Hardcore-fs]

You can auto size, there are calls that allow a buffer to be size tested before allocation, but even if you did have to specify.... it's not like it would be rocket science.....

Things to avoid is assumption mistakes.. over on the chia threads someone thought they could decrease the sort times, by using pointers... instead of moving data.... sadly the pointer size is almost the same size as the data it points to , so it's actually slower!!!

but most FS even ram based , have significant overheads.... I was looking at a server that was suffering a few MS on context switching (under a vm)

[alexhu98]

And please don't limit this to just temp2. There are plenty of time in the whole plotting process where temp2 is half used while temp1's bandwidth is maxed out.
Better to look at what you can do given 30 / 60 / 90 GB memory. For example, you may be able to process [P1] Table 5, 6, 7 and [P2] Table 7, 6, 5 together all in memory. Then process [P2] Table 3, 2 and [P3-1/2] Table 2, 3 in memory as well.
Since you already have the data structure in memory before writing it out, you may be able to simply skip writing out the table to disk.
I do suggest allowing RAM size as a parameter in case if people want to run multiple plotters in parallel. For example, I can run 18 plots a day with a single plotter, but 30 plots a day with 3 plotters in parallel, as I seems to have a bottleneck in the disk IO.

[Hardcore-fs]

it's like any Major battle, you fight it in stages....
but yes.. there is a lot of unneeded "writing out" and reading back, becasue of the thought processes used in the original python code.
Lets remember it was all done for "proof of time" & space...
that and shilling 500 million, whilst getting other fools to pay for infrastructure power & disk storage...

[vladaepro]

I completely agree with idea - especially as RAM speed is sometimes 10x faster than accessing files on RAM disk due to FS overhead. (It seems that some NVME-s are even faster than RAM disk with tmpfs)

[Hardcore-fs]

they only "appear" faster......
there was the oracle fiasco back in the 90's
Oracle was looking for a way to improve their database write times... so.. they came up with the idea of building their own FS
this was before there was so much ram available.
they wasted a couple of years to build a FS that was actually slower.. than other FS
when they looked into it, they found out there were all sorts of "clever" optimizations with ram buffers & caching, even they were small by todays standards, it was a complete fiasco, becasue their sales team had been selling databases on "assumptions of new performance"
but that situation does not apply in this case,