Dependency version mismatch on pyarrow

version 3.22.0 uses ArrowDtype which is only available in pyarrow>=10.0.1, but dependency requirement specifies pyarrow >= 3.0.0.

Environment details

OS type and version: Darwin mac-mini.lan 20.6.0 Darwin Kernel Version 20.6.0 (also on GCP AppEngine)
Python version: Python 3.10.11
pip version: pip 23.3.1
google-cloud-bigquery version: 3.22.0

Steps to reproduce

note: google-cloud-aiplatform requires pyarrow 7.0.0 for python 3.10

install google-cloud-bigquery and google-cloud-aiplatform db-dtypes
call to_dataframe on RowIterator

Code example

# example
client = bigquery.Client()
df = bqclient.query(sql_statement).result().to_dataframe()

Stack trace

File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/cloud/bigquery/table.py", line 2283, in to_dataframe
    range_date_dtype = pandas.ArrowDtype(
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/pandas/core/dtypes/dtypes.py", line 2096, in __init__
    raise ImportError("pyarrow>=10.0.1 is required for ArrowDtype")
ImportError: pyarrow>=10.0.1 is required for ArrowDtype"

What version of pandas do you have?

I suspect older versions of pandas may be compatible with older pyarrow.

I have pandas 2.2.2. I've reported this issue since pip's dependency resolution didn't prevent this issue from happening. I'd expect google-cloud-bigquery 3.22.0 to require pyarrow>=10.0.1, or older version of pandas which maybe compatible with this change assuming you're right about it.

We test with minimum versions of all of our dependencies to make sure they are compatible and the latest versions of all of our dependencies.

For the in-betweens we have some version detection logic throughout the code, but it sounds like we need to extend it for this case.

Thanks for raising the issue! Indeed I only accounted for when pandas is an older version. I will add a min version check for pyarrow too.

Navigation Menu

Repository navigation

Dependency version mismatch on pyarrow #1912

Environment details

Steps to reproduce

Code example

Stack trace

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Dependency version mismatch on pyarrow #1912

Description

Environment details

Steps to reproduce

Code example

Stack trace

Activity

tswast commented on May 7, 2024

naoki-jarvisml commented on May 8, 2024

tswast commented on May 8, 2024

Linchin commented on May 8, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions