Repository navigation
(feat) Additional Arrow support #1785
Description
Additional Arrow support
With the current on demand data loading, I think Data Viewer could extend existing Arrow table support to lazy Arrow-backed data sources:
Datasetand its subclasses, includingFileSystemDatasetarrow_dplyr_query
This allows file-backed Arrow datasets and dplyr queries on Arrow data to be opened directly with View() without first collecting the full dataset into memory. Rows are fetched on demand as they are requested by the Data Viewer, allowing large Arrow datasets to be browsed efficiently.
This should only be a small change in the data viewer model.
Activity
grantmcdermott commentedon Sep 27, 2026
Could we not implement via a more lightweight solution like
nanoarrow? https://arrow.apache.org/nanoarrow/latest/index.htmlFred-Wu commentedon Sep 28, 2026
It looks like
nanoarrowis mainly designed for data streaming. It still relies onarrowforDatasetandarrow_dplyr_query. For our data viewer, we also need efficient forward and backward scrolling, filtering, sorting, and caching, soarrowseems better fitted here. Type conversion might also be an issue withnanoarrow.For those files sit on disc rather than in memory, I think some trade-offs are expected. For example, loading data might be slower compared to in-memory data, especially after sorting.