Repository navigation
Option for unique to _also_ delete first instance of any duplicated observation #1163
Description
Activity
Would be nice to have this option as well..(ex:
strict = TRUEorstrictly.unique=TRUE) but how aboutDT[, if (.N == 1L) .SD, by=key(DT)]?Indeed that's a much better workaround; I've edited the FR to reflect that, but also note that there still seems to be room for up to 3x improvement in speed by extending the
uniquefunctionality.Another workaround: Using
uniqueitself and then differencing row numbers may be faster:fr <- function() unique(DT[,ii:=.I])[(!c(diff(ii)-1L,ii[.N]-nrow(DT)))]For example in this case:
n_group <- 1e5 draw_n_member <- 1:5 draw_val <- 1:5 DT0 <- data.table(id=1:n_group) DT <- DT0[, sample(draw_val,sample(if(.GRP%in%c(1,n_group)) 1 else draw_n_member,1) ,replace=TRUE),keyby=id] microbenchmark(times=10,arun(),fr(),unique(DT))Change the simulation parameters, and this way turns out worse, though.
This would match the
keep = Falsebehavior ofpandas.DataFrame.drop_duplicates:https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop_duplicates.html
- changed the title
[-]Feature Request: Option for `unique` to _also_ delete first instance of any duplicated observation[/-][+]Option for `unique` to _also_ delete first instance of any duplicated observation[/+]on Apr 3, 2020
I'm frequently presented with
data.tables which have (a very small percentage of) duplicated keys which causes some trouble when I use them injto merge.In my application, it makes sense to just drop any of those observations because they can't reliably be distinguished and they're so infrequent;
uniqueseems perfectly suited to this end, except that it retains at least one of the observations in any duplicated group, while I prefer to cut them out completely because it's essential not to mess up the merge process, the results of which are crucial for the whole project.There's a workaround which is quite verbose; let's use the sample in
?duplicated.data.table:DT <- data.table(A = rep(1:3, each=4), B = rep(1:4, each=3), C = rep(1:2, 6), key = "A,B")The troublesome observations are 1,3,4,6.
From what I can tell my only recourse at the moment is something rather elaborate like
Something like
unique(DT,only.unique=T)that achieves the same end seems like it would be easy to implement.---EDIT 2015 May 27---
Arun's suggested workaround is much better than my approach, but comparison with
uniquesuggests there's still considerable speed being lost:Performance of the workaround also deteriorates substantially when
.SDis large, while that ofuniqueis barely affected: