Repository navigation
Extend 2 billion row benchmarks e.g. memory usage, sorting, joining, by-reference #2
Description
Activity
- changed the title
[-]Extensive benchmarking on large in-memory dataset[/-][+]Extend 2 billion row benchmarks[/+]on Sep 26, 2014 - changed the title
[-]Extend 2 billion row benchmarks[/-][+]Extend 2 billion row benchmarks e.g. memory usage, sorting, joining, by-reference[/+]on Dec 12, 2014 For memory usage, perhaps: https://github.com/gsauthof/cgmemtime
Figured
cgmemtimeout. Quite straightforward. Just need some time to run through grouping/joins/reshaping benchmarks.- removed this from the milestone
on May 10, 2018 I would like to close this one as it is already epic, and will be epic for a long time, due to broad scope defined here.
Work on benchmarks has been shifted to a dedicated h2o project https://github.com/h2oai/db-benchmark and related issues are:Reacted by Michael Chirico and Matt Dowle- added a commit that references this issue
on Nov 20, 2025
We've currently gone to 2E9 rows (the 32bit index limit) with 9 columns (100GB). See benchmarks page on wiki.
Ideally it would be great to compare all available tools that are either specifically developed for large in-memory data manipulation or are capable of handling data at these sizes much better than base. Of course base-R should also be included, typically as control.
Aspect of benchmarking should be to highlight not just run time (speed), but also memory usage. The sorting/ordering by reference, sub-assignment by reference etc.. features, for example, at this data size should display quite clearly on speed and memory gains attainable.