serialized data.tables from before cannot be read out of the box #7488

Activity

1. aitap commented on Dec 19, 2025

   

   aitap

   on Dec 19, 2025

   Last edited by aitap

   Member

   More actions

   Yes, the old (pre-#7451) data.tables don't have it set at all. The new ones unserialize as "growable but TRUELENGTH=0" and work fine. Checking to see if adding a check for the growable bit to selfrefok will suffice.

   Deleting columns used to be safe even if the data.table was not over-allocated, because SETLENGTH worked on all lists. R_resizeVector needs a resizable list now:

   data.table/src/assign.c

   Lines 665 to 666 in be67a4d

   	R_resizeVector(dt, ndt-ndelete);
   	R_resizeVector(names, ndt-ndelete);

   Unserialized data.tables come out with TRUELENGTH=0; tables from the old version don't have the growable bit set; tables from the new version do.

   This setalloccol:

   data.table/R/data.table.R

   Lines 1389 to 1423 in f6bbd5b

   	if(!anyNA(m)) {
   	# updates by reference to existing columns
   	cols = as.integer(m)
   	newnames = NULL
   	} else {
   	# Adding new column(s).
   	newnames = setdiff(lhs, names_x)
   	m[is.na(m)] = ncol(x) + seq_along(newnames)
   	cols = as.integer(m)
   	# ok <- selfrefok above called without verbose -- only activated when
   	# ok=-1 which will trigger setalloccol with verbose in the next
   	# branch, which again calls _selfrefok and returns the message then
   	# !is.data.table for DF |> DT(,:=) tests 2212.16-19 (#5113) where a shallow copy is routine for data.frame
   	if ((ok<1L) || (truelength(x) < ncol(x)+length(newnames))) {
   	DT = x # in case getOption contains "ncol(DT)" as it used to. TODO: warn and then remove
   	n = length(newnames) + eval(getOption("datatable.alloccol")) # TODO: warn about expressions and then drop the eval()
   	# i.e. reallocate at the size as if the new columns were added followed by setalloccol().
   	name = substitute(x)
   	if (is.name(name) && ok && verbose) { # && NAMED(x)>0 (TO DO) # ok here includes -1 (loaded from disk)
   	catf("Growing vector of column pointers from truelength %d to %d. A shallow copy has been taken, see ?setalloccol. Only a potential issue if two variables point to the same data (we can't yet detect that well) and if not you can safely ignore this. To avoid this message you could setalloccol() first, deep copy first using copy(), wrap with suppressWarnings() or increase the 'datatable.alloccol' option.\n", truelength(x), n)
   	# #1729 -- copying to the wrong environment here can cause some confusion
   	if (ok == -1L) catf("Note that the shallow copy will assign to the environment from which := was called. That means for example that if := was called within a function, the original table may be unaffected.\n")
   	# Verbosity should not issue warnings, so cat rather than warning.
   	# TO DO: Add option 'datatable.pedantic' to turn on warnings like this.
   	# TO DO ... comments moved up from C ...
   	# Note that the NAMED(dt)>1 doesn't work because .Call
   	# always sets to 2 (see R-ints), it seems. Work around
   	# may be possible but not yet working. When the NAMED test works, we can drop allocwarn argument too
   	# because that's just passed in as FALSE from [<- where we know `*tmp*` isn't really NAMED=2.
   	# Note also that this growing will happen for missing columns assigned NULL, too. But so rare, we
   	# don't mind.
   	}
   	setalloccol(x, n, verbose=verbose) # always assigns to calling scope; i.e. this scope

   ...probably needs to happen when removing columns as well.

   Test:

   mtcars |> as.data.table() |> serialize(NULL) |> unserialize() |> _[, carb := NULL]
2. ben-schwen

   closed this as completedin #7492on Dec 20, 2025