Repository navigation
[Python] Efficient "row accessor" for a pyarrow RecordBatch / Table #28689
Description
Activity
Antoine Pitrou / @pitrou:
Ironically, an efficient solution might be simply to convert Table slices and iterate each slice. Something like (untested):def itertuples(table): chunk_size = 1024 for i in range(0, table.num_rows, chunk_size): rows = table[i:i + chunk_size].to_pydict().values() yield from map(tuple, rows)
Antoine Pitrou / @pitrou:
Also, it seems it would be a nice addition to be allow to restrictto_pydictto certain columns.Joris Van den Bossche / @jorisvandenbossche:
In your snippet, what's the.values()? (that's not a Table method I think. Did you mean to_numpy or so?)Sorry, ignore my comment, I missed the
to_pydict()before it ;)Joris Van den Bossche / @jorisvandenbossche:
For restricting to certain columns, you can also quite easily dotable.select([...])before theto_pydictcall (I think in general selecting columns is a quite cheap operation?)Antoine Pitrou / @pitrou:
Ah, right. It should be quite cheap indeed.Luke Higgins:
that snippet is yielding columns of the table and not rows though right?Antoine Pitrou / @pitrou:
Ah, right. Here is a corrected version:def itertuples(table): chunk_size = 1024 for i in range(0, table.num_rows, chunk_size): rows = table[i:i + chunk_size].to_pydict().values() yield from zip(*rows)
Antoine Pitrou / @pitrou:
Related: ARROW-12976 (#28694)Wes McKinney / @wesm:
It would probably be worth the effort to implement the "tuplization" of RecordBatch in the libarrow_python C++ library to make it reasonably efficient. This would also be a good opportunity to move the implementation of the*Scalar.as_pymethods into libarrow_python since you would only want to have one canonical implementation of boxing Arrow array values as Python objects. This relates to ARROW-12976 also. I can't find the Jira issue about moving the as_py implementations into C++, but I recall there was one in the past that @kszucs may have been working on at some point.Krisztian Szucs / @kszucs:
When we refactored/cleaned up the scalar handling it has occurred to me that libarrow_python should be responsible for arrow=>python conversions as well but we deferred that work. This is something we should definitely work on.Joris Van den Bossche / @jorisvandenbossche:
I think we can discuss those performance related aspects of the arrow->python object conversion (implementing it in C++ vs in python/cython) in ARROW-12976. For bulk conversion like here, we probably want to avoid creating a Scalar at all (so also not going through Scalar.as_py, even when impemented in C++, based on the experiments described in ARROW-12976).(of course, ARROW-12976 uses "to_pylist" as example, and for this issue it might also be interesting to look into directly creating tuples instead of lists for each columns which gets zipped)
Grant Williams:
I think natively returning tuples (or at least having the option to do so) makes a lot of sense from a user perspective. I would guess the use case for a row iterator is going to be most similar to a database cursor or a Result/Row Proxy in sqlalchemy. Having the data returned in a tuple (or tuple-like) format would make integration really nice in mixed data source code bases.Personally, I think the row accessor makes a lot of sense on Record Batches since it's already pretty straightforward to slice rows from Tables as tuples and a conversion like:
yield from chain.from_iterable(zip(*map(lambda col: col.to_pylist(), batch.columns)) for batch in record_batches)
feels clunky (and inefficient).
Antoine Pitrou / @pitrou:
While it's clunky, the jury is out on whether it's inefficient or not. The overhead might not be as high as it would seem.Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
I still think a fast row accessor akin to pandas itertuples would be a good addition, any thoughts?
It would be nice to have a nice row accessor for a Table akin to pandas.DataFrame.itertuples.
I have a lot of code where I am converting a parquet file to pandas just to have access to the rows through iterating with itertuples. Having this ability in pyarrow natively would be a nice feature and would avoid memory copy in the pandas conversion.
Reporter: Luke Higgins
Related issues:
Note: This issue was originally created as ARROW-12970. Please see the migration documentation for further details.