How To Use vaex at Cloud Data Storage

[medyasun]

Hi,
Vaex is incredible and i loved it.

You did a great job congrats.

I just need to ask a question.

I have data on google Bigquery . How i directly read this data from google Bigquery?

The problem is i could found a code that includes to_dataframe and that code converts data to pandas. This is loosing a lot of time and ram on big data.

It is same other online cloud storages (like AWS)

On tutorials always reading data from csv files.

I need help to use vaex on my cloud projects.

thanks a lot.

[JovanVeljanoski]

Hi,

This is not documented but is implemented. You need to install vaex-contrib, which you get by default if you install the whole of vaex (via pip install vaex or conda/mamba install -c conda-forge vaex).

You can find the relevant methods here:

from vaex.contrib.io.gbq import from_query, from_table, to_table

The docstrings give an example of how these are used. I hope this helps a bit

[medyasun]

it is awesome thanks a lot.
What about AWS ?

[JovanVeljanoski]

What do you mean by AWS?

If you mean S3, vaex can read data directly via df = vaex.open(s3_uri)

If you mean Redshift, then no, we have nothing similar yet. But you can also use pandas or any other tool to get the data and then pass it to vaex.

[medyasun]

Actually both :)

thanx for your quick answers