-
|
So I know it's not exactly what ADBC is for, but if I have an iterable of Python objects to insert into a PostgreSQL table, what's the best way of using ADBC to get them into the database? Is there a way to pass the iterables to a COPY ... FROM STDIN statement? (This is what I would do using psycopg2/psycopg) Or is it better to batch up the iterable as arrow tables and use multiple calls of adbc_ingest? Or something else? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Generally I think its best to wrap the iterable(s) in an arrow structure. It doesn't necessarily need to be a Table; RecordBatch and RecordBatchReader objects work just as well (really anything that implements the Arrow PyCapsule Interface) Behind the scenes the ADBC postgres driver issues a COPY command anyway, but if you route it through an Arrow structure it can communicate with the database using the binary format, whereas I (perhaps mistakenly) assume a manual COPY from stdin is going to use the text-based protocol |
Beta Was this translation helpful? Give feedback.
-
|
Ah thank you! (And RecordBatchReader sounds like the sort of thing I'm looking for, so will try that) |
Beta Was this translation helpful? Give feedback.
Generally I think its best to wrap the iterable(s) in an arrow structure. It doesn't necessarily need to be a Table; RecordBatch and RecordBatchReader objects work just as well (really anything that implements the Arrow PyCapsule Interface)
Behind the scenes the ADBC postgres driver issues a COPY command anyway, but if you route it through an Arrow structure it can communicate with the database using the binary format, whereas I (perhaps mistakenly) assume a manual COPY from stdin is going to use the text-based protocol