[Python] Arrow to Python list (to_pylist) conversion is slow

@pitrou

It seems that we are 20x slower than Numpy for converting the exact same data to a Python list.

With integers:

>>> arr = np.arange(0,1000, dtype=np.int64)
>>> %timeit arr.tolist()
8.24 µs ± 3.46 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
>>> parr = pa.array(arr)
>>> %timeit parr.to_pylist()
218 µs ± 2.39 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

With floats:

>>> arr = np.arange(0,1000, dtype=np.float64)
>>> %timeit arr.tolist()
10.2 µs ± 25.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
>>> parr = pa.array(arr)
>>> %timeit parr.to_pylist()
199 µs ± 1.04 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Reporter: Antoine Pitrou / @pitrou

Related issues:

[Python] Efficient "row accessor" for a pyarrow RecordBatch / Table (is related to)

_{Note: This issue was originally created as ARROW-12976. Please see the migration documentation for further details.}

@pitrou

Antoine Pitrou / @pitrou:
@jorisvandenbossche @amol-

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
I see a slightly smaller difference, but anyway still big (10µs vs 200µs instead of 800µs, maybe due to release/debug build?).

I suppose the main reason for the slowness here is that to_pylist is currently implemented by accessing each element of the array (getitem), wrapping that into a pyarrow Scalar object, and then converting that scalar to a native python object (as_py()). I can imagine this getitem / wrapping in a scalar object gives quite some overhead.

@pitrou

Antoine Pitrou / @pitrou:
Oops, yes, I had posted my number on debug mode :-o My bad ! I'll edit the issue.

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
Happens to me all the time ;)

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
I was thinking that we could in principle put those different steps we now take in cython in a single specialized to_list function on a concrete array class, eg for Int64Array:

    def to_pylist_primitive(self):
        cdef:
            shared_ptr[CScalar] scalar

        res = []
        for i in range(len(self)):
            scalar = GetResultValue(self.ap.GetScalar(i))
            if scalar.get().is_valid:
                res.append((<CInt64Scalar*> scalar.get()).value)
            else:
                res.append(None)
        return res

which still creates the C scalars, but avoids wrapping it the Python (cython) class, and avoids some function call overhead. Now, this only gives a 2x speed-up (and thus still 10x slower as numpy), and a large part of the remaining time is spent in the C++ Array::GetScalar.

So if we really want to improve this, it might need some more custom code at the C++ level to get vectorized access to raw data (at least for primitive arrays). But not really sure this is worth it for to_pylist.

@pitrou

Antoine Pitrou / @pitrou:
Yes, the only reasonable way to speed this up is to bypass Scalar creation entirely (which is slow).

Since we spent quite some time optimizing the Python-to-Arrow case, it seems worthwhile to optimize the reverse path.

cc @wesm for opinions.

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
Actually, the APIs are already available to avoid the Scalar creation, for example for Int64Array:

    def to_pylist(self):
        cdef:
            CInt64Array* int_arr = (<CInt64Array*> self.ap)
            int64_t val

        res = []
        for i in range(len(self)):
            if int_arr.IsValid(i):
                val = int_arr.Value(i)
                res.append(val)
            else:
                res.append(None)
        return res

This gives me 13µs for the example case, which is now almost the same as for the numpy tolist.

This might certainly be worth including. Are there ways in cython to avoid having to duplicate this in each of Int8/Int16/Int... Array? The problem is that NumericArray::Value\(i) return type is type-dependent.

@pitrou

Antoine Pitrou / @pitrou:
No, Cython doesn't allow generating templated C++ code.
To avoid repeating oneself, it's best to do it in C++. It can also help making it faster.

@emkornfield

Micah Kornfield / @emkornfield:
@pitrou @jorisvandenbossche going to see if I consolidate this logic in C++ (unless you were thinking of taking it up). Any preference for trying to split up into smaller PRs or one large one to migrate all types to C++ code?

@pitrou

Antoine Pitrou / @pitrou:
Migrating all types at once sounds more logical to me. I wonder what the overall approach should be, though.

@emkornfield

Micah Kornfield / @emkornfield:
One thing we discussed on the sync call is if a more explicit API should be provided to control coercion of timestamp[ns] to pd.Timestamp instead of the current behavior that will do the conversion if pandas is installed but fall back to datetime (and check nanoseconds=0) if pandas is not installed. @jorisvandenbossche any thoughts here?

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
@emkornfield this is specifically for the scalar as_py() and array to_pylist() behaviour, right? (and not the Table.to_pandas)

Personally I would be fine with a more explicit API (the idea would be to add a keyword to those functions to explicitly ask for a pandas object?). But some concerns:

changing to use datetime.datetime instead of pd.Timestamp by default for ns resolution would be a backwards incompatible change. How do we see that? Just change, or deprecate first? (it seems a bit annoying to deprecate, although if we add a keyword, that can directly be used to silence the warning, and I suppose those functions are not used that much anyway)
If you have nanoseconds in the timestamp value, that means we would raise an error by default? (the one we raise now if pandas is not installed) That doesn't feel super nice user experience, but I suppose this is the inevitable consequence of a more explicit API.

@emkornfield

Micah Kornfield / @emkornfield:
Yeah, given #1 and #2, I think I'll try to simply replicate existing behavior in C++, even though it can lead to unexpected behavior.

@toddfarmer

Todd Farmer / @toddfarmer:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

Just started using PyArrow. Are there any updates/recommended practices on this issue, this would significantly speed up "iterating through rows" for me.

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

Navigation Menu

Repository navigation

[Python] Arrow to Python list (to_pylist) conversion is slow #28694

Related issues:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python] Arrow to Python list (to_pylist) conversion is slow #28694

Description

Related issues:

Activity

asfimport commented on Jun 4, 2021

asfimport commented on Jun 7, 2021

asfimport commented on Jun 7, 2021

asfimport commented on Jun 7, 2021

asfimport commented on Jun 7, 2021

asfimport commented on Jun 7, 2021

asfimport commented on Jun 7, 2021

asfimport commented on Jun 7, 2021

asfimport commented on Oct 10, 2021

asfimport commented on Oct 13, 2021

asfimport commented on Oct 16, 2021

asfimport commented on Oct 21, 2021

asfimport commented on Oct 24, 2021

asfimport commented on Jul 12, 2022

hitesh-pathak commented on Mar 24, 2024

github-actions commented on Jan 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions