Repository navigation
fwrite crashes for tables with large columns #2974
Description
Bit of an edge case here, but when trying to fwrite a table with a very large column R crashes every time.
Potentially related to #2612.
Minimal reproducible example:
huge_vector <- paste(rep('x', 30e6), collapse='')
foo <- data.table::data.table(x=huge_vector)
tf <- tempfile()
data.table::tables()
# NAME NROW NCOL MB COLS KEY
# 1: foo 1 1 29 x
# Total: 29MB
# this works fine
write.table(foo, file=tf)
# this reliably crashes my R session
data.table::fwrite(foo, file=tf)
Output of sessionInfo():
R version 3.4.3 (2017-11-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=English_Australia.1252 LC_CTYPE=English_Australia.1252 LC_MONETARY=English_Australia.1252
[4] LC_NUMERIC=C LC_TIME=English_Australia.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.4.3 tools_3.4.3 yaml_2.1.19 data.table_1.11.0
For what it's worth, I have 16GB RAM on the system with ~8GB free before running the above example.
Activity
philippechataignon commentedon Jan 16, 2019
fwritefirst attempt to compute max line length from a sample. By default, buffer is 8MB. Here line is 30M. In this PR #3288, I use strlen and a message error indicates that buffer is too small and give an advice to increase it.mattdowle commentedon Apr 23, 2019
Awesome PR @philippechataignon! I made a few minor changes to PR #3288 and now this example works without needing user to increase
buffMB. The test is not huge really, but to keep resources to a minimum on CRAN I haven't added it to the test suite.