Skip to content

BigQuery DATETIME objects raise errors when creating a DataFrame instance via arrow #20849

@baggiponte

Description

@baggiponte

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

from google.cloud import bigquery

client = bigquery.Client(...)

query = """
    SELECT CAST('2024-07-26 10:00:00' as DATETIME) as datetime_col
"""

pl.from_arrow(client.query(query).to_arrow())

Log output

File "/Users/luca/work/xtream/a2a/a2a-goodfellow-python/test.py", line 26, in <module>
    pl.from_arrow(client.query(query).to_arrow())
  File "/Users/luca/work/xtream/a2a/a2a-goodfellow-python/.venv/lib/python3.12/site-packages/polars/convert/general.py", line 436, in from_arrow
    arrow_to_pydf(
  File "/Users/luca/work/xtream/a2a/a2a-goodfellow-python/.venv/lib/python3.12/site-packages/polars/_utils/construction/dataframe.py", line 1181, in arrow_to_pydf
    pydf = PyDataFrame.from_arrow_record_batches(batches, data.schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ComputeError: cannot create series from Extension(ExtensionType { name: "google:sqlType:datetime", inner: Timestamp(Microsecond, None), metadata: None })

Issue description

This has only happened to datetime so far.

Expected behavior

Polars should return a datetime col.

Installed versions

Details
--------Version info---------
Polars:              1.20.0
Index type:          UInt32
Platform:            macOS-14.4.1-arm64-arm-64bit
Python:              3.12.8 (main, Dec 19 2024, 14:22:58) [Clang 18.1.8 ]
LTS CPU:             False

----Optional dependencies----
Azure CLI            <not installed>
adbc_driver_manager  <not installed>
altair               5.5.0
azure.identity       <not installed>
boto3                <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
google.auth          2.37.0
great_tables         <not installed>
matplotlib           3.10.0
nest_asyncio         1.6.0
numpy                2.2.2
openpyxl             <not installed>
pandas               2.2.3
pyarrow              19.0.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           2.0.37
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds triageAwaiting prioritization by a maintainerpythonRelated to Python Polars

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions