Repository navigation
Read collection of values out of backends #1605
Description
This is the first piece of the "Collections API", which is the ability to work with collections of (key, value) pairs in a similar way to using RDDs, without the overhead of Spark.
This task is to read a collection of tiles from the backends. We can already read in single values (e.g. Tiles), and RDDs of tiles. With the completion of this task, we will read some collection of values (e.g. Vector[(SpatialKey, Tile)]) out of the backend. This should be able to use the Query API to query the backend in order to produce the collection.
A potential interface for this type of reader is follows, which is distinctly similar to FilteringLayerReader
abstract class CollectionLayerReader[ID] {
def read[
K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
V: AvroRecordCodec: ClassTag,
M: JsonFormat: GetComponent[?, Bounds[K]]
](id: ID, rasterQuery: LayerQuery[K, M], numPartitions: Int, indexFilterOnly: Boolean): Vector[(K, V)] with Metadata[M]
def read[
K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
V: AvroRecordCodec: ClassTag,
M: JsonFormat: GetComponent[?, Bounds[K]]
](id: ID, rasterQuery: LayerQuery[K, M], numPartitions: Int): Vector[(K, V)] with Metadata[M] =
read(id, rasterQuery, numPartitions, false)
def read[
K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
V: AvroRecordCodec: ClassTag,
M: JsonFormat: GetComponent[?, Bounds[K]]
](id: ID, rasterQuery: LayerQuery[K, M]): Vector[(K, V)] with Metadata[M] =
read(id, rasterQuery, defaultNumPartitions)
def read[
K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
V: AvroRecordCodec: ClassTag,
M: JsonFormat: GetComponent[?, Bounds[K]]
](id: ID, numPartitions: Int): Vector[(K, V)] with Metadata[M] =
read(id, new LayerQuery[K, M], numPartitions)
def query[
K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
V: AvroRecordCodec: ClassTag,
M: JsonFormat: GetComponent[?, Bounds[K]]
](layerId: ID): BoundLayerQuery[K, M, Vector[(K, V)] with Metadata[M]] =
new BoundLayerQuery(new LayerQuery, read(layerId, _))
def query[
K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
V: AvroRecordCodec: ClassTag,
M: JsonFormat: GetComponent[?, Bounds[K]]
](layerId: ID, numPartitions: Int): BoundLayerQuery[K, M, Vector[(K, V)] with Metadata[M]] =
new BoundLayerQuery(new LayerQuery, read(layerId, _, numPartitions))
}This might require us to have a similar type as ContextRDD for the collection type, e.g. ContextCollection
Activity
lossyrob commentedon Jul 26, 2016
/cc @pomadchin
pomadchin commentedon Jul 26, 2016
@lossyrob great, thanks!
fosskers commentedon Sep 28, 2016
This can be closed @lossyrob @echeipesh @rossbernet.