Repository navigation
Compatibility with the future native pipe #4872
Description
Activity
This is base R limitation not related to data.table
toupper(letters) |> x => x[1:5] #Error: function '[' not supported in RHS call of a pipe
Is there a good reason for not supporting it? For me it almost classify as a bug.
In case it is not going to be fixed in R-devel then this issue #641 should take care of it.This is deliberately so. Special symbols are not allowed in the RHS of a pipe, see here.
Nevertheless, only the name of the symbol is checked, so a workaround is to assign a new symbol to the given primitive function. E.g.:
1:4 |> `+`(3) # error add <- `+` 1:4 |> add(3) # OK extract <- `[` data.table(x = letters) |> extract(j = x) # OK
Of course the
extractfunction above is not safe, e.g., you will get a warning if the input is a data.frame instead of a data.table:extract <- `[` data.frame(x = letters) |> extract(j = x) # Warning message: # In `[.data.frame`(data.frame(x = letters), j = x) : # named arguments other than 'drop' are discouraged
Reacted by Jan Gorecki and Mwavu KennedyYes, #641 should solve it. IMHO,
dtquery()is too long a name, though. Especially with the goal of using the pipe in mind.mtcars |> dtquery(mpg > 10) |> dtquery(, disp2 := disp/wt) |> dtquery(, .(hp = mean(hp)), by = cyl)
To me, that code feel horrible to write and distracting to read, but it's purely a personal matter.
Of course, since we are not talking about cache invalidation, the difficult thing here is naming the thing.
Agree, maybe
sbfrom square bracket?mtcars |> sb(mpg > 10) |> sb(, disp2 := disp/wt) |> sb(, .(hp = mean(hp)), by = cyl)
data.table symbols start with dot, perhaps something like that? It might be consistent.. It would also evoke a bit of the
%>% .[]notation..S(for "Square"?)mtcars |> .S(mpg > 10) |> .S(, disp2 := disp/wt) |> .S(, .(hp = mean(hp)), by = cyl)
Also,
.t? because it kinda looks like.[mtcars |> .t(mpg > 10) |> .t(, disp2 := disp/wt) |> .t(, .(hp = mean(hp)), by = cyl)
Adding my 2 cents:
I really like the idea of a dot prefix, but would go for
.d(). Fast to type and the most common letter assignment ("d") for a data frame-like object.Failing that, I would stick to well-known data.table naming conventions, which makes
.DT()the next obvious candidate IMO.Reacted by Adam AltmejdI personally like the idea of
.S("Square") for square brackets but would prefer.s(lowercase) because it is easier to type. Because this is intended to be used with the pipe, I think that it should not only be as easier to type as possible but also as short as possible (possibly not more than 2 characters). This is just my opinion.Also, I noticed that the following expression does not print anything (even if the last function call is
print());
but, the output gets printed if the row with assignment (.s(, b := -a)) is removed..s <- data.table:::`[.data.table` y <- list(a = 1:5) setDT(y) |> .s(, b := -a) |> print()I also tried the next chunk of code but it did not work either. I thought that it would have behaved like
setDT(y)[, b := -a][]but it did not print anything.setDT(y) |> .s(, b := -a) |> .s()Personally, I'm not so keen on either of
.t()or.s()/.S().For
.t(), I think it goes against the mnemonic that data.table uses for its other special operators..SD,.GRP, etc. are all based on an abbreviation heuristic rather than the "shape" of the operator. I also think it's too close to the base transpose function, which is where my brain automatically goes when I seet... and invites unexpected behaviour in the case of a simple typo. (C.f.DT |> .t()vsDT |> t()).(
dt()would be even worse ofc because that creates a namespace conflict with a base R function.)Similarly,
.s()/.S()is too close to.SDand.SDcolsin my view, without being at all related to the underlying functionality.Taking a step back, IMO the point of this symbol should be to encapsulate a data.table. I know I've already said this, but I genuinely think that
.DT()is the obvious solution here. It's the widely used shorthand for constructed data.tables already, used throughout the docs, and should be intuitive enough for newcomers as well. (Although, I'll readily accept.d()or.Dtoo ;-)Reacted by Vincent Arel-Bundock, Jaap Walhout, Tokhir Dadaev, Adam Altmejd and Rick SaportaPersonnally
.dt()is my preference.Reacted by Mark Fairbanks(FWIW the tidyverse team are planning to work with R core to figure out a placeholder syntax (or equivalent) for the base pipe that there's some hope that 4.2 might allow you do to
dt|> .[i](or similar))Reacted by Michael Chirico, Mark Fairbanks, Grant McDermott, Elio Campitelli, Eyayaw Beze, Vincent Arel-Bundock, Jan Gorecki, Jaap Walhout, Adam Altmejd, Michael Steinbaugh and 10 moreReacted by Marcell Granát and Karthat sounds great @hadley -- is there a bugzilla/r-devel thread we could use to track progress here to help gauge whether short-term investment in a wrapper is worthwhile?
Not that I know of, sorry.
If such a function is implemented it would be nice to have it work with non data.table data frames and return an output of the original class. I suppose we could make a case to always return a data.table but I like personally the idea of not forcing the user into data.table world if they only need one call (that'd be similar to https://github.com/moodymudskipper/withDT).
I also wonder if it makes sense to modify by reference using the pipe, maybe it should always copy?
@moodymudskipper I don't think we're planning on maintaining any more complicated interface than a simple wrapper for convenience:
<foo> = `[.data.table` = function(...) {...}My sense is that iif there is some guarantee that 4.2.0 will support
dt |> .[], then there's not much point in introducing a wrapper that could be easily be created by an user on a per-script basis. In the meantime, data.table's documentation could just suggest users to add the wrapper themselves.Reacted by Vincent Arel-Bundock, Kevin Kainan Li, Jan Gorecki and Jaap WalhoutI don't have 4.1.0 handy, but I'd think that this would work:
mtcars |> `[`(mpg > 10) |> `[`(, disp2 := disp/wt) |> `[`(, .(hp = mean(hp)), by = cyl)
Given
`[`is three characters, for a shortcut to make sense it would need to be either one or two characters. And of course that's easy to make yourself:X <- `[` mtcars |> X(mpg > 10) |> X(, disp2 := disp/wt) |> X(, .(hp = mean(hp)), by = cyl)
(I'm not proposing
Xas a name, just using it as an example.)There's no guarantee that 4.2.0 will have a placeholder syntax, but it's something that I hope we can persuade R core is useful and important. One of the main objections is that
.is a very small symbol and easy to miss.Reacted by Mark Fairbanks and haozhou1988Under 4.1.0 the first example fails:
> mtcars |> + `[`(mpg > 10) |> + `[`(, disp2 := disp/wt) |> + `[`(, .(hp = mean(hp)), by = cyl) Error: function '[' not supported in RHS call of a pipe >
The second one works if and only if
mtcarswas made into adata.tableobject:> X <- `[` > mtcars |> + X(mpg > 10) |> + X(, disp2 := disp/wt) |> + X(, .(hp = mean(hp)), by = cyl) cyl hp 1: 6 122.2857 2: 4 82.6364 3: 8 209.2143 >
It is of course nothing but sugar over existing syntax:
> mtcars[mpg > 10][, .(hp = mean(hp)), by = cyl] cyl hp 1: 6 122.2857 2: 4 82.6364 3: 8 209.2143 >
Right now, r-devel is implementing a native pipe that is incompatible with chaining in data.table. Using the proposed
d =>syntax I get an error using this R versionIf "[" won't be supported as the RHS of a pipe, then no syntax transformation can make data.table compatible with the
|>pipe, if I understand correctly.I see two ways of addressing this.
On the one side, we could talk with r-core to present this issue and see if they can change their implementation to make it work.
Another option would be to create a functional alias to
data.table:::"[.data.table", lets saydt(). This, for example, seems to work.(in fact, it also works with
dt <- base::"[")Personally, I don't mind that notation at all. It is true that if such a simple fix resolves the issue, then each user could do it in their own scripts, but I think it might be preferable if data.table provided a standardised alias so that other people's code stays readable. For the record, I don't like
dtvery much, it's just the first thing that came in mind.