Rasters FeatureExtraction by pomadchin · Pull Request #3117 · locationtech/geotrellis

pomadchin · 2019-10-06T23:41:29Z

Overview

This PR adds a type class to extract PointFeatures from Rasters by a passed Geometry

Checklist

docs/CHANGELOG.rst updated, if necessary
Unit tests added for bug-fix or new feature

Demo

scala> val ext = Extent(0.0, 0.0, 3.0, 3.0)
ext: geotrellis.vector.Extent = Extent(0.0, 0.0, 3.0, 3.0)

scala>       val data = Array(
     |         1, 2, 3,
     |         4, 5, 6,
     |         7, 8, 9
     |       )
data: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)

scala> val raster = Raster(ArrayTile(data, 3, 3), ext)
raster: geotrellis.raster.Raster[geotrellis.raster.IntConstantNoDataArrayTile] = Raster(IntConstantNoDataArrayTile([I@5e8a55b6,3,3),Extent(0.0, 0.0, 3.0, 3.0))

scala> val features = raster.cellFeaturesAsPoint[Int](ext.toPolygon)
features: Iterator[geotrellis.vector.Feature[geotrellis.vector.Point,Int]] = non-empty iterator

scala> features.toList
res0: List[geotrellis.vector.Feature[geotrellis.vector.Point,Int]] = List(
Feature(POINT (0.5 2.5),1), 
Feature(POINT (1.5 2.5),2), 
Feature(POINT (2.5 2.5),3), 
Feature(POINT (0.5 1.5),4), 
Feature(POINT (1.5 1.5),5), 
Feature(POINT (2.5 1.5),6), 
Feature(POINT (0.5 0.5),7), 
Feature(POINT (1.5 0.5),8), 
Feature(POINT (2.5 0.5),9))

scala> val areaFeatures = raster.cellFeaturesAs
cellFeaturesAsArea   cellFeaturesAsPoint

scala> val areaFeatures = raster.cellFeaturesAsArea[Int](ext.toPolygon)
areaFeatures: Iterator[geotrellis.vector.Feature[geotrellis.vector.Polygon,Int]] = non-empty iterator

scala> areaFeatures.toList
res1: List[geotrellis.vector.Feature[geotrellis.vector.Polygon,Int]] = List(
Feature(POLYGON ((0 2, 0 3, 1 3, 1 2, 0 2)),1), 
Feature(POLYGON ((1 2, 1 3, 2 3, 2 2, 1 2)),2), 
Feature(POLYGON ((2 2, 2 3, 3 3, 3 2, 2 2)),3), 
Feature(POLYGON ((0 1, 0 2, 1 2, 1 1, 0 1)),4), 
Feature(POLYGON ((1 1, 1 2, 2 2, 2 1, 1 1)),5), 
Feature(POLYGON ((2 1, 2 2, 3 2, 3 1, 2 1)),6), 
Feature(POLYGON ((0 0, 0 1, 1 1, 1 0, 0 0)),7), 
Feature(POLYGON ((1 0, 1 1, 2 1, 2 0, 1 0)),8), 
Feature(POLYGON ((2 0, 2 1, 3 1, 3 0, 2 0)),9))

Notes

The user is able to "easily" provide new instances for CellFeatures type class for in-memory rasters by using CellFeatures.make and providing: getGrid: R => GridExtent[Int], cellValue: (R, Int, Int) => D

R is unbounded so any kind of crazy thing that should be treated as Raster will pass mustard here.
Closes #2942

CloudNiner

The core interface looks good to my relatively untrained eye. This gets at the core desired functionality.

Don't forget additions to the module hierarchy document and the CHANGELOG for this feature.

Signed-off-by: Grigory Pomadchin <[email protected]>

The Rasterizer is defined on Geometry, so any type variance on the feature we're using to mask the raster is pretty much wasted

Array of features from a sizable can produce very high load on heap, likely blowing out a spark job if it is used in that context. The expectation is that this feature will be used in context of RDD map/flatMap where Iterator is very benficial or the first operation will be a filter or foldLeft on the generated features.

This is important to decide how to capture or ignore pixels on the border of the geometry. Also geom and raster switch palces. Since operation is conceptually on raster it takes the first parameter place, that of self.

Previous type-class seemed too wide open for interpretation. This implementation narrows down the meaning for greater clarity.

echeipesh · 2019-10-17T05:08:47Z

@pomadchin this PR is ready for re-review. This is the result of our conversation last night.
I tried to keep the interface open enough to be able to implement it for RasterSource where the raster would be fetched and featurized in chunks but I don't want to do that here.

echeipesh · 2019-10-17T05:09:29Z

+
+
+/** Type class to convert a raster into features of cell geometries */
+trait CellFeatures[R, D] {


R is unconstrained because I want to be able to do this to RasterSource which doesn't have much in common with Raster[Tile]

echeipesh · 2019-10-17T05:11:11Z

+trait CellFeatures[R, D] {
+
+  /** Describe cell grid of the source raster */
+  def cellGrid(raster: R): GridExtent[Long]


This very clearly is crying out to be a type class of its own. RasterLike or something like that. That could be a meaningful filter on the Methods as well. If something can't be described with a GridExtent[*] we don't want to be talking about cell features.

IMO we can deal with that later. This method can be removed without impacting subclasses that may spring up in the meantime.

echeipesh · 2019-10-17T05:12:58Z

+  /** Produce a CellFeatures instance given functions to retrieve a values and describe the cell grid.
+    * This function is suitable for use with rasters that fit entirely in memory.
+    */
+  def make[R, D](getGrid: R => GridExtent[Int], cellValue: (R, Int, Int) => D): CellFeatures[R, D] =


cellValue is Function3 and is not specialized, causing boxing for col/row here. Elsewhere in the code we would make a specific trait. However since the main purpose of this interface is to turn a single double cell value into a nested class with a whole geometry ... its fine, we'll let the boxing slide here.

It is itself a slow operation, would like to see someday how bad is boxing here.

pomadchin

LGTM, I can't approve it due to github limitations, but merging it in. 💯

pomadchin · 2019-10-17T18:13:54Z

+  /** Produce a CellFeatures instance given functions to retrieve a values and describe the cell grid.
+    * This function is suitable for use with rasters that fit entirely in memory.
+    */
+  def make[R, D](getGrid: R => GridExtent[Int], cellValue: (R, Int, Int) => D): CellFeatures[R, D] =


It is itself a slow operation, would like to see someday how bad is boxing here.

pomadchin · 2019-10-17T18:14:08Z

+        Rasterizer.foreachCellByGeometry(geom, grid.toRasterExtent) { case (col, row) => mask.set(col, row, 1) }
+        for {
+          row <- Iterator.range(0, grid.rows)
+          col <- Iterator.range(0, grid.cols) if mask.get(col, row) == 1


Nice catch 👍

pomadchin added the in progress label Oct 6, 2019

pomadchin commented Oct 6, 2019

View reviewed changes

Comment thread raster/src/main/scala/geotrellis/raster/FeatureExtraction.scala Outdated

pomadchin force-pushed the feature/feature-extractor branch 2 times, most recently from 56deb7e to aba9c53 Compare October 7, 2019 19:32

CloudNiner reviewed Oct 7, 2019

View reviewed changes

Comment thread raster/src/main/scala/geotrellis/raster/FeatureExtraction.scala Outdated

pomadchin force-pushed the feature/feature-extractor branch from 430debf to 763a10b Compare October 7, 2019 20:23

CloudNiner reviewed Oct 7, 2019

View reviewed changes

Comment thread raster/src/main/scala/geotrellis/raster/FeatureExtraction.scala Outdated

Comment thread raster/src/main/scala/geotrellis/raster/Implicits.scala Outdated

Comment thread raster/src/main/scala/geotrellis/raster/Implicits.scala Outdated

pomadchin force-pushed the feature/feature-extractor branch 2 times, most recently from fe68872 to 44bbfb1 Compare October 7, 2019 20:56

pomadchin removed the in progress label Oct 7, 2019

pomadchin requested a review from echeipesh October 7, 2019 21:01

pomadchin added the needs review label Oct 7, 2019

pomadchin force-pushed the feature/feature-extractor branch from 44bbfb1 to fa54bf4 Compare October 7, 2019 21:15

rossbernet assigned echeipesh Oct 8, 2019

pomadchin and others added 9 commits October 17, 2019 01:07

FeatureExtraction TC prototype

4c1a4b2

Signed-off-by: Grigory Pomadchin <[email protected]>

Fix codestyle, make TypeClass more generic, cover with tests

2b8a8c3

Features should contain Array of values per band in the Data

ef48fdb

Make PointFeatureExctractor more generic

00dd0b1

Remove I parameter from FeatuerExtractor

ad0d1b4

The Rasterizer is defined on Geometry, so any type variance on the feature we're using to mask the raster is pretty much wasted

thread through Rasterizer.Options

0c3555e

This is important to decide how to capture or ignore pixels on the border of the geometry. Also geom and raster switch palces. Since operation is conceptually on raster it takes the first parameter place, that of self.

Reformulate as CellFeatures

a089d64

Previous type-class seemed too wide open for interpretation. This implementation narrows down the meaning for greater clarity.

Update changelog [skip ci]

ba91ade

echeipesh force-pushed the feature/feature-extractor branch from f71733f to ba91ade Compare October 17, 2019 05:07

echeipesh reviewed Oct 17, 2019

View reviewed changes

fix: rename of file [skip ci]

e130ea1

pomadchin commented Oct 17, 2019

View reviewed changes

echeipesh assigned pomadchin and unassigned echeipesh Oct 17, 2019

pomadchin merged commit 3901c7f into locationtech:master Oct 17, 2019

pomadchin removed the needs review label Oct 31, 2019

Navigation Menu

Repository navigation

Rasters FeatureExtraction - #3117

Rasters FeatureExtraction#3117
pomadchin merged 10 commits into
locationtech:masterfrom
pomadchin:feature/feature-extractor

pomadchin commented Oct 6, 2019 •

edited by echeipesh

Loading

Uh oh!

Uh oh!

Uh oh!

CloudNiner left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

echeipesh commented Oct 17, 2019

Uh oh!

echeipesh Oct 17, 2019

Uh oh!

echeipesh Oct 17, 2019

Uh oh!

echeipesh Oct 17, 2019

Uh oh!

pomadchin Oct 17, 2019

Uh oh!

pomadchin left a comment

Uh oh!

pomadchin Oct 17, 2019

Uh oh!

pomadchin Oct 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		/** Type class to convert a raster into features of cell geometries */
		trait CellFeatures[R, D] {

Conversation

pomadchin commented Oct 6, 2019 • edited by echeipesh Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Checklist

Demo

Notes

Uh oh!

Uh oh!

Uh oh!

CloudNiner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

echeipesh commented Oct 17, 2019

Uh oh!

echeipesh Oct 17, 2019

Choose a reason for hiding this comment

Uh oh!

echeipesh Oct 17, 2019

Choose a reason for hiding this comment

Uh oh!

echeipesh Oct 17, 2019

Choose a reason for hiding this comment

Uh oh!

pomadchin Oct 17, 2019

Choose a reason for hiding this comment

Uh oh!

pomadchin left a comment

Choose a reason for hiding this comment

Uh oh!

pomadchin Oct 17, 2019

Choose a reason for hiding this comment

Uh oh!

pomadchin Oct 17, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pomadchin commented Oct 6, 2019 •

edited by echeipesh

Loading