Bad Memory

[ajit788]

I'm doing a pretty simple operations shown below but on a large dataset

1. Convert the csv to hdf5 (df is 1.1 million rows by 6 columns) (df)
2. Left join df with another smaller dataset (df1) but with allow duplicates - (df2)
3. Left join df2 with another smaller dataset (df3) but with allow duplicates - (df4 becomes pretty big close to 7billion rows and 12 columns)
4. Group by on df4 (simple sum on one column on unique two columns) - Runs for over 10 hours but fails with memory error. (I've a reasonably good machine with 24 cores and 256 GB memory)

I reckon the dataset is pretty big - is there anyway I can accomplish this task? Thanks in advance!