[Python] Efficient "row accessor" for a pyarrow RecordBatch / Table

It would be nice to have a nice row accessor for a Table akin to pandas.DataFrame.itertuples.

I have a lot of code where I am converting a parquet file to pandas just to have access to the rows through iterating with itertuples. Having this ability in pyarrow natively would be a nice feature and would avoid memory copy in the pandas conversion.

Reporter: Luke Higgins

Related issues:

[Python] Arrow-to-Python conversion is slow (relates to)

_{Note: This issue was originally created as ARROW-12970. Please see the migration documentation for further details.}

@pitrou

Antoine Pitrou / @pitrou:
Ironically, an efficient solution might be simply to convert Table slices and iterate each slice. Something like (untested):

def itertuples(table):
    chunk_size = 1024
    for i in range(0, table.num_rows, chunk_size):
        rows = table[i:i + chunk_size].to_pydict().values()
        yield from map(tuple, rows)

cc @jorisvandenbossche

@pitrou

Antoine Pitrou / @pitrou:
Also, it seems it would be a nice addition to be allow to restrict to_pydict to certain columns.

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
~~In your snippet, what's the .values()? (that's not a Table method I think. Did you mean to_numpy or so?)~~

Sorry, ignore my comment, I missed the to_pydict() before it ;)

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
For restricting to certain columns, you can also quite easily do table.select([...]) before the to_pydict call (I think in general selecting columns is a quite cheap operation?)

@pitrou

Antoine Pitrou / @pitrou:
Ah, right. It should be quite cheap indeed.

Luke Higgins:
that snippet is yielding columns of the table and not rows though right?

@pitrou

Antoine Pitrou / @pitrou:
Ah, right. Here is a corrected version:

def itertuples(table):
    chunk_size = 1024
    for i in range(0, table.num_rows, chunk_size):
        rows = table[i:i + chunk_size].to_pydict().values()
        yield from zip(*rows)

@pitrou

Antoine Pitrou / @pitrou:
Related: ARROW-12976 (#28694)

@wesm

Wes McKinney / @wesm:
It would probably be worth the effort to implement the "tuplization" of RecordBatch in the libarrow_python C++ library to make it reasonably efficient. This would also be a good opportunity to move the implementation of the *Scalar.as_py methods into libarrow_python since you would only want to have one canonical implementation of boxing Arrow array values as Python objects. This relates to ARROW-12976 also. I can't find the Jira issue about moving the as_py implementations into C++, but I recall there was one in the past that @kszucs may have been working on at some point.

@kszucs

Krisztian Szucs / @kszucs:
When we refactored/cleaned up the scalar handling it has occurred to me that libarrow_python should be responsible for arrow=>python conversions as well but we deferred that work. This is something we should definitely work on.

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
I think we can discuss those performance related aspects of the arrow->python object conversion (implementing it in C++ vs in python/cython) in ARROW-12976. For bulk conversion like here, we probably want to avoid creating a Scalar at all (so also not going through Scalar.as_py, even when impemented in C++, based on the experiments described in ARROW-12976).

(of course, ARROW-12976 uses "to_pylist" as example, and for this issue it might also be interesting to look into directly creating tuples instead of lists for each columns which gets zipped)

Grant Williams:
I think natively returning tuples (or at least having the option to do so) makes a lot of sense from a user perspective. I would guess the use case for a row iterator is going to be most similar to a database cursor or a Result/Row Proxy in sqlalchemy. Having the data returned in a tuple (or tuple-like) format would make integration really nice in mixed data source code bases.

Personally, I think the row accessor makes a lot of sense on Record Batches since it's already pretty straightforward to slice rows from Tables as tuples and a conversion like:

yield from chain.from_iterable(zip(*map(lambda col: col.to_pylist(), batch.columns)) for batch in record_batches)

feels clunky (and inefficient).

@pitrou

Antoine Pitrou / @pitrou:
While it's clunky, the jury is out on whether it's inefficient or not. The overhead might not be as high as it would seem.

Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

I still think a fast row accessor akin to pandas itertuples would be a good addition, any thoughts?

Navigation Menu

Repository navigation

[Python] Efficient "row accessor" for a pyarrow RecordBatch / Table #28689

Related issues:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python] Efficient "row accessor" for a pyarrow RecordBatch / Table #28689

Description

Related issues:

Activity

asfimport commented on Jun 4, 2021

asfimport commented on Jun 4, 2021

asfimport commented on Jun 4, 2021

asfimport commented on Jun 4, 2021

asfimport commented on Jun 4, 2021

asfimport commented on Jun 4, 2021

asfimport commented on Jun 4, 2021

asfimport commented on Jun 4, 2021

asfimport commented on Jul 6, 2021

asfimport commented on Jul 6, 2021

asfimport commented on Jul 6, 2021

asfimport commented on Jul 7, 2021

asfimport commented on Jul 7, 2021

asfimport commented on Oct 5, 2022

github-actions commented on Jan 28, 2026

virtualluke commented on Mar 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions