Fully indexed cross join - secondary key, Nth key #804

The following answer http://stackoverflow.com/a/18935344/2490497
inspired me to write a generic Nth key function.
I've manage to do it, it is a first working version.
You can read about it/install it from #803.
It is working fine but it scales bad in terms of memory usage. If you are using only secondary key it is still good, but if you will use few more keys it scales worse and worse. Total size (in nrow) for index is (2^length(keys(DT)))*nrow(DT).
Examples from the documentation of CJI and setindex were perfomed on nrow(DT) = 500k and 6 keys, it was OK on my notebook c2d 2.2GHz, 4GB RAM, but it was close to the limit, maybe 2 more cols in keys could be too much already. In my machine I have 16.25x speed gain over using vector scan:

# load DT and IDX from examples in `CJI`
library(microbenchmark)
microbenchmark(DT[id2=="id061" & id4==83 & id6==3267],
               # DT[CJ(unique(id1),"id061",unique(id3),83,unique(id5),3267)], # this crash my R session
               DT[CJI(IDX,TRUE,"id061",TRUE,83,TRUE,3267)],
               times = 100L)

Besides it is not scale well for number of keys to index on, it is now ready for tons use cases:

• where the priority is to minimize time to query data with particular filters (and you don't mind few extra GB of RAM) - I wonder if we could call it low latency queries :)
• and it should scales well for more columns (which are not part of the key), kind of DW use case where you have just a few keys (dimensions) and many values (measures).

Regarding the scalability improvement.
Possibily it can be dramatically improved if we could reuse parent (already calculated) index-sets by reference. Get to know with examples in setindex and see below:

1. calculate index for cols id2, id3, id4, store as "2_3_4"
2. when calculating any of the sub-indices of the above "2_3_4" (e.g. "2_3_4_5", "2_3_4_6") store its result in memory only for additional columns (id5 or id6) and reuse pre-calculated "2_3_4" index by reference.
3. when calculating the sub-index of the above "2_3_4_5" ("2_3_4_5_6") store its result in memory only for additional column (id6) and reuse pre-calculated "2_3_4_5" index by reference.

Of course it would not start from "2_3_4" but from "1".
Yet, I'm not sure if it's possible in R.

Summarizing, this is not a final secondary key or Nth key solution, it is a implementation which solve the problem for many (maybe even most of) cases but is not as slim, powerful and scalable as the data.table functions naturally are.
It might be a good beginning for dealing with the Nth key problem (just want to point that 2th key is not worth to focus on, Nth key should be the target).
Please mark the issue as question and make it open as discussion until we will not find better Nth key solution.