Data selection in DataFrame

[Kendidi]

Hi,

I have a dataset with 10,000 entries or X and Y.

1. Can I export a range of entries (e.g. entry # 1000 to # 5000) to a new dataset?
2. Can I plot only entries # 1000 to #5000 in a 2-D heatmap?

In both cases, we only care about the entry#, not the their values.

Thanks a lot in advance!

[JovanVeljanoski]

Hi,

Yeah that is easily done, that is just row slicing like you would do in numpy or pandas:

# This creates a shallow copy (does not use extra memory)
df2 = df[1000:5000]

# Or you can do things directly like
df[1000:5000].viz.heatmap(...)
df[1000:5000].export(...)

I hope this helps!

[Kendidi]

With Vaex, I wonder is it possible to plot x and y of df[0:1000], then df[1001:2000] incrementally onto the same heatmap figure? Thanks.

[JovanVeljanoski]

Can you give more information? Are you looking to make an animation? or a widget with a slider that updates the map? Or a single heatmap figure with subplots for each slide?

Everything is possible, the simplest is to follow the approach one would take when using any other library.. there is nothing special about vaex in this regard.

Vaex does support some advanced plotting routines.. maybe you can check out this example

[Kendidi]

The datasets I would like to plot heatmaps with are quite big. So it normally will take a few seconds. My understanding is it will do all the processing (includes binning + aggregation), when all the data is ready, then do the plotting at the end.

Since the plotting process will take sometime, I want the plot to be smooth, like we are adding dots to the figure until all data is processed. That's why I am trying to plot data from #0 to #9999, then #10000 to #19999, etc. I think on 'ipython', I can do it manually and see the graph get updated, or reprinted with additional data. But I prefer the have the whole process be done in a script instead. I do not know what options do I have. Would you please suggest a good way to approach this? Thanks.

[JovanVeljanoski]

I don't understand what you mean by "when all the data is ready". Also there is no adding dots when you are making a heatmap, I guess one can continuously update that but..

Vaex can make a histogram or a heatmap from a billion samples in 1-2 seconds, so unless you have much bigger data than that.. just make a heatmap right away. No need to do it incrementally.

Maybe go over the tutorial and the rest of the examples in the documentation, if you are unsure how things work.

Or maybe I just dont understand what you are after.. maybe somebody else can help.

[Kendidi]

The datasets I have is quite large and are on external disks. Any hint on dynamic heatmap plot would be great. Thanks.

[Kendidi]

By the way, for the x-axis and y-axis of heatmap, how do we set display range. e.g -100 to +100? Thanks a lot in advance.