Repository navigation
(feat) Additional Arrow support #1785
Description
Activity
Could we not implement via a more lightweight solution like
nanoarrow? https://arrow.apache.org/nanoarrow/latest/index.htmlCould we not implement via a more lightweight solution like
nanoarrow? https://arrow.apache.org/nanoarrow/latest/index.htmlIt looks like
nanoarrowis mainly designed for data streaming. It still relies onarrowforDatasetandarrow_dplyr_query. For our data viewer, we also need efficient forward and backward scrolling, filtering, sorting, and caching, soarrowseems better fitted here. Type conversion might also be an issue withnanoarrow.For those files sit on disc rather than in memory, I think some trade-offs are expected. For example, loading data might be slower compared to in-memory data, especially after sorting.
Additional Arrow support
With the current on demand data loading, I think Data Viewer could extend existing Arrow table support to lazy Arrow-backed data sources:
Datasetand its subclasses, includingFileSystemDatasetarrow_dplyr_queryThis allows file-backed Arrow datasets and dplyr queries on Arrow data to be opened directly with
View()without first collecting the full dataset into memory. Rows are fetched on demand as they are requested by the Data Viewer, allowing large Arrow datasets to be browsed efficiently.This should only be a small change in the data viewer model.