Support for Polars

Polars is an up and coming alternative for Pandas which already has more then 17.000 GitHub Stars: https://pola-rs.github.io/polars-book/

I wonder if it would be possible to provide an export_to_polars and import_from_polars function without too much effort?

As a workaround Polars could already be used with polars.from_pandas, but this involves copying the whole data.

[tkilias]

Hi xyxz-web,

export_to_pandas is basically a special implementation of export_to_callback, so the quickest thing would implementing a call back which reads the raw data into polaris.

Here is an example for export_to_callback https://github.com/exasol/pyexasol/blob/master/examples/b02_import_export.py.

Here you can see how the export_to_pandas callback is called

pyexasol/pyexasol/connection.py

Line 273 in 63e97d8

def export_to_pandas(self, query_or_table, query_params=None, callback_params=None, export_params=None):

Here you can see how the callback is implemented.

pyexasol/pyexasol/callback.py

Line 36 in 63e97d8

def export_to_pandas(pipe, dst, **kwargs):

It basically reads the CSV coming from the pipe with pandas.read_csv. I think, polars provides almost the same API. https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.read_csv.html

Hope that already helps.

[tkilias]

Hi @xyxz-web,

We discussed this internally, and Polars is currently not our roadmap for this project. I hope, you can write your own callback with the of my previous explanation, should you need it. For that reason, I will close this ticket for the moment.

Hi @tkilias ,

thanks for your answer!
In case I would implement this, could it be merged to the master branch?

[littleK0i]

Ok, here is the CSV-parsing call in Polars: https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.read_csv.html

Implementation is trivial, but we have a few interesting questions to answer:

Decimal type?
Polars supports Decimal type natively: https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.Decimal.html

It might be nice to use it automatically for DECIMAL columns.

try_parse_dates?
I think it should be enabled by default, as long as it can fit full range of Exasol TIMESTAMP values.

rechunk?
It is enabled by default. Not sure if it should stay enabled.

[tkilias]

Hi,

If you want to implement it and have it merged, I would suggest making polars an optional python dependency, such that user can decide if he would like to install it, because I am not sure about the size of polars and compatibility with different platforms.

[littleK0i]

This seems to be working: master...polars

The only problem is related to warnings, which I do not know how to mute:

Polars found a filename. Ensure you pass a path to the file instead of a python file object when possible for best performance.
Polars found a filename. Ensure you pass a path to the file instead of a python file object when possible for best performance.

Pyexasol writes into pipe, but polars thinks it's a file.

Output:

shape: (10_000, 8)
┌─────────┬────────────┬───────────┬────────────┬───────────┬─────────────┬────────────┬───────────┐
│ USER_ID ┆ USER_NAME  ┆ REGISTER_ ┆ LAST_VISIT ┆ IS_FEMALE ┆ USER_RATING ┆ USER_SCORE ┆ STATUS    │
│ ---     ┆ ---        ┆ DT        ┆ _TS        ┆ ---       ┆ ---         ┆ ---        ┆ ---       │
│ i64     ┆ str        ┆ ---       ┆ ---        ┆ i64       ┆ f64         ┆ f64        ┆ str       │
│         ┆            ┆ date      ┆ datetime[μ ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ s]         ┆           ┆             ┆            ┆           │
╞═════════╪════════════╪═══════════╪════════════╪═══════════╪═════════════╪════════════╪═══════════╡
│ 0       ┆ KTSU80DIWH ┆ 2018-08-1 ┆ 2018-05-20 ┆ 0         ┆ 0.04        ┆ 8.07       ┆ PENDING   │
│         ┆            ┆ 9         ┆ 19:19:04.3 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 71         ┆           ┆             ┆            ┆           │
│ 1       ┆ O0PH2A7OM5 ┆ 2018-03-0 ┆ 2018-10-03 ┆ 1         ┆ 0.26        ┆ 46.13      ┆ DISABLED  │
│         ┆            ┆ 2         ┆ 07:59:06.0 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 91         ┆           ┆             ┆            ┆           │
│ 2       ┆ E19GGXPHZE ┆ 2018-04-2 ┆ 2018-05-14 ┆ 0         ┆ 0.28        ┆ null       ┆ SUSPENDED │
│         ┆            ┆ 3         ┆ 15:58:59.7 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 27         ┆           ┆             ┆            ┆           │
│ 3       ┆ NDNSL7VWKL ┆ 2018-05-2 ┆ 2018-03-08 ┆ 0         ┆ 0.84        ┆ 10.75      ┆ ACTIVE    │
│         ┆            ┆ 0         ┆ 15:39:04.0 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 26         ┆           ┆             ┆            ┆           │
│ …       ┆ …          ┆ …         ┆ …          ┆ …         ┆ …           ┆ …          ┆ …         │
│ 9996    ┆ 0WY4RNSIYV ┆ 2018-01-2 ┆ 2018-01-24 ┆ 0         ┆ 0.61        ┆ 44.9       ┆ ACTIVE    │
│         ┆            ┆ 0         ┆ 00:33:56.4 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 00         ┆           ┆             ┆            ┆           │
│ 9997    ┆ CEVORWHH6P ┆ 2018-03-0 ┆ 2018-12-31 ┆ 1         ┆ 0.56        ┆ 31.23      ┆ DISABLED  │
│         ┆            ┆ 7         ┆ 11:19:13.0 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 58         ┆           ┆             ┆            ┆           │
│ 9998    ┆ M8K2YDG7JR ┆ 2018-10-1 ┆ 2018-05-10 ┆ 0         ┆ 0.59        ┆ 21.12      ┆ SUSPENDED │
│         ┆            ┆ 8         ┆ 09:04:13.9 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 55         ┆           ┆             ┆            ┆           │
│ 9999    ┆ CJY3XA21LU ┆ 2018-07-2 ┆ 2018-02-03 ┆ 0         ┆ 0.84        ┆ 60.88      ┆ DISABLED  │
│         ┆            ┆ 9         ┆ 21:06:53.5 ┆           ┆             ┆            ┆           │
│         ┆            ┆           ┆ 95         ┆           ┆             ┆            ┆           │
└─────────┴────────────┴───────────┴────────────┴───────────┴─────────────┴────────────┴───────────┘
EXPORTED 10000 rows in 1.13350248336792s
IMPORTED 0 rows in 0.059839725494384766s
shape: (5_000, 2)
┌─────────┬────────────┐
│ USER_ID ┆ USER_NAME  │
│ ---     ┆ ---        │
│ i64     ┆ str        │
╞═════════╪════════════╡
│ 8192    ┆ 11MPTFV6GF │
│ 8193    ┆ KYVDZY4JJC │
│ 8194    ┆ G6L6A4QRLQ │
│ 8195    ┆ 7CGA3VXMGD │
│ …       ┆ …          │
│ 8188    ┆ IVV5VI3T2Z │
│ 8189    ┆ 5SE7LA5AUR │
│ 8190    ┆ 1YIWZZEPQE │
│ 8191    ┆ ILLJ1XAW78 │
└─────────┴────────────┘
EXPORTED 5000 rows in 1.0486383438110352s

[littleK0i]

Also, I've double-checked DateTime type. It supports the full range of values from 0001-01-01 to 9999-12-31 23:59:59.999.

No issues with that.

[raayu83]

I wonder if there has been any progress on this lately.

[raayu83]

Hello,

since polars passed version 1.0 recently...
Are there any plans to finalize this?
I can help with some tasks, but I can't run the tests locally because I'm on Windows.

[Nicoretti]

Hi @raayu83,

as far as I understand, this was closed because it is not planned to be implemented. However, I will go and bring this up with our PM. In case this issue is not reopened or linked to a new one, you can safely assume that it is still not planned.

best
Nico

[raayu83]

Would really love to see Polars support, since it is probably the next big thing after Pandas.
The 1.0 release was mentioned in many places.